Lodestar key import takes too long

[barnabasbusa]

Currently loading in lodestar validator keys one by one takes too long ~ 0.3 sec/key on an M1 macbook pro.

This means that on an average kurtosis run where the default validator will have 64 validator keys, each validator process will add an additional 20 seconds to start up.

Once the key loaded in once, lodestar stores it in a cache at <dataDir>/cache (default to ~/.local/share/lodestar/<network_name>).

Example local_keystores.cache:

  {
    "version": 4,
    "uuid": "402a9ffa-8c4b-443d-947f-473fefdc1729",
    "path": "/root/.local/share/lodestar/testnet/cache/local_keystores.cache",
    "pubkey": "81ef1d058664e94bcdd4876d049fc40e9a2d55a104a2fbcd33b63774a55bf994d72e6c7dc641af29bfa57362413db70581f9ac9f60825d682d5ab33098117dbdcf3c5245116c8c03a8c0493a5d441ba578a0b3d069d745cfdff70122c65e421a8257c261afa77e79086b503de88dad720443a1d135cbc14f8a6de408a03ac5b9c4263731d6693bf843f0a9657aa3c4e8837d8e7320247799d20afbddb410fda1bb7fcd31e36ccc841012e4dac0d643e8e5e12d2467d8d3219aab33b64280eaea84c1b3ec3752b9d7e3ddfe87c68c59443b5fca9d18d78e5441c80e4fb6c0ecec27f8074811c2e5f823364a71e056739484e8581cf13f7df6a96835ad1330593bf5d1f4e5bd6341f7a54063b0233e921250d0c09a48849155953e293cf635d7b6859155dd5a22f116ae8f61b1516770f8ff41ec0ea24b8b745171b4cf34981bb7d235e7e1a739a0589e7c7ff69ede9b15863fb35bdce0573031210c5a6d3521b5a3e11cc99356de9e8adccf3accb4c4d387f954b202d0db647e65c331b5019226870342ee85d1d3eda564de4126f20880d59164e2f88652d9dcf3dc93d0bf19e22ca3a11305f1cba1cadaf2d11702893688ce4d8fee80abef17438d6e1dee0da0087b3fde540c7cd157e169866c603af0af26fe3c3d2ba517dead10376d60df898afc5f7128e998c2a59b9e2dd1805ccda56220d1bb25ea94c13d4f9abb4ab55a522518aabface4f10d54f45d08dffa8d8bf6583d6de04a89b9ca61c69977de4ca440c4c9b13c7a1c65e205979849376e239d03cb46d8bcdf58afca96a8e0fcfa8c3f73416c86d93ae2dbc2a468b5d6cc39d42c2ea7e20cc215212d3dd3b8efbed324bcd4a28de941fe855c54a92c69738cb3b628e5ec89b1cae8fffb946a1277613deec1f3f0e7d75708fe3f6b15c5efb75a0ec8dce885812cf14f3d589b6f918d19ef6c96e7ea917640ea3ad0c6c6a9ac8320456c3ec046cebaa625d415476f71d854d186a1758c79843925b2d268ac8d1df9e9132058de96486f102ff4ef6e34c988b6dd42a9462954218b8728310bdc25a4251c092eec8128bfdde893049e8de14b70a834e78b74d6bd46bd0f5f92d878a1b19f9081808dae86391aeab2a048368f0ae5b93ab412561db70482dba08f14d19885ec7d1bf7b6c4668d505283d68d1af78588532a15aa80ecf96a2d8bbeb813f63c25c15cd8f04c10fca4c49f8faadcfded5c85beb36f1cdc234f0cfd8ebb0820a18899445accf3e6e35efc0ee34419eeb0cfb097dff5fd6462b9ed908fe444e4b5610d3583a667dcf23af26a9f686db05e7de9ebc03f8ea0a756cc96b4f55611f5788d61f26e5bda1df70f688ff3e5eb9c905e42b88cd8fd8593bde4d44e22cbf296d47917a4b0f144fbbe2ab69a4421b2c0ff3500141dcf88a0b007902b0898a017b3e98d334cdf49d5411e507f6043fca0624e937945c7dfb1829a3a2e1d0bdba654aef7d2e3c14c76b48d914dc0bb6af111bc9021887ca4dfb27cc2063c9ebd2be133754d30ca18fa8698787005538f556bd94867cc1c5f7b817a9188ee0eb50d3a88a27a2ce4334ea8cac1662593526f4c280cb3049bc91afd8381ddbda124b9fab871aacf378ab5380a9218096756d3ccec228caddd27979b9050b58d7304f2be4f7d4aafb6ae19fdab0bb9149ce29517a383f59facaacf6d39933d88e34601df31b7a66bae1d49079548a2ae2d72037c7c3b2fb8925631989c1ae353311c36fe325b2f1fdc1f648194947e056beba42f6d7fec6712de16458c85ae513391dc4d42120e5877ed21a0681820a6150cb45af4f0cbf98c1d25c8cc94ef22c9183e15da2e4ad8e05a75c5b9201d52e9ad7f66cb0061b0c68779ba5a1fd0f11b9c365c1721d00199d287923c9592c95f5c1574b8c510e546759f383779a0364fefa009f84dfcebb2efc4e86b909a5fffb90ed79b5f66980b420db7e495e8e9e5c389be338759c40d2e408b1f0b78ebc0ffbafae360e33b683ada3638414338b83490e9ab1ff067425d25d78598249520dcdc8be36b8afed019336e2eb478d44a0f5ebb6d7c994710856abf32ed25313867e060f83b205ea5ebe9f0fc98eeb5417011d88a2924cbe7ff7ed616a6b7dfe273187360757c667f27349f525ac60665f9f7ba2d07d91d2f94566f1e995c28e8767e8677ef93cf6aa49453a8ad6c279622820e321ff9d352ceafffaa7ec082713e4c85e50cf9f7e11439cc16999664783bb5eb59491a99142392f020fb4e3b607203c3f9bec7ccb3537e8f7856d7ddfbede9dc3332cf40bdd4334a64a03f7aa96fd1dcb385cf9f4f29f9f3ef4c25a47efdb090c18763a51608e92267daa98695f765705a7630269561f9086ca0833e9ca3181e33a9b51d7c31722cd07c8f0a18a34ba083eaeb091529c2a454c3950ec449ab7f6f237995bf1ec0a802a1c646e753dd9d9811cd75f58c371e74ce83f05606bc076d64bdc77adb313b1bfd8ceceae38d8a8b7d4795f10ac68d18a567c07d1f258a7dc4f685b9c45c3217e9e640d8cbb3fde3a875e31b0212df6d48985f8524922205aaf6917a5b577d89a608c06384bc606f723cde2a6f7d64a29de9cd987c0626dabea2414bd5b653647b31953c1156803191484d5ecf4630f1a6094298efdad03170daefba1bf92ee8691aa86fb753afcc252090c74c5f97c7d6cdfbbe0017f57e25d406d4ccebdb0ea62b23a8a25355c20cf4ebf93bc43a7b4076ab247ac7bff133ddaa7cfb9588e598d1022285b0e2f2fdd0ee3fd51a6f39a757657d95e795d3460d5454b0f3987885f0a8138c4de92e08d2843709f808adb191c9d2b22399bb9445dbb94e190382a83c93593c32b08e89f0089e1a0892dcf121a856199b751ec40959433b4e64ee7fea260a16f4929261c5e2ebc148042ca8f3c2f7371294ca58c85e7b1513c155a39e5bd76eedd90f50a4fb50d6ae55b40623d5ba0c5d85c40def89d6c67d5fc3a94d36839c1557727aa55fcca24faaab0d1e5f0ecff1dd709e22f0c55408b8b74d22173bc312d27baaa8688ae1b4be2ca9f291de2f415ab3a4002206769493d82a094e1934b78d98b79c93f70ffff7389d8ea0962b34b6df04ba999442a0fda5aa228e16f801dfbde52d5f57b44e7e9a613fef33f893fad979ce003eeed012c8f45e9050db814e241be87ae94c0c4011aaafd2bc633a130d0798c8f398ebd00f5c1b131c2b4d48cbaddde1c8ed59eae6af8290d27700852df2b3cf23d630d807adc1c39301fa1fe99678a7b7887e895c8df24e15546b13d2237ab2795cc7004e6b68e69724f2c0922f119d5af8819bc9adcb081ad4dd8f1acfdf1a71360c6d5655bdd58d9bb1f09e4de43b7ff8a6da60b61df9ce65f9ffc951740dfc69812667adfdf512e8ac8e3f01ab9a03733c6d0bd0a5403778eebf691d2906380bc01591e93b43cae86c94f657c92c29d5698a06ae2bb170260de9afb23f49d5770474a65b4d380d904aa41cab7c2852074ed8ad9fb94f6d7e5a5a1f6f71c0dce0bddd12b05cafec5912f22dbd6f15677f25f13d93ecd5ec6f957fddd7cf27d73521b34aaaf6a219f77b21128d18321c2c8d679bb06cb2010c1167c72840b3149ba92b326799375b1b05a1c0ca38ace5e8f61cba48c1b350a7220938e9c2c0fe6b6c2881b0862b7f0788739e315de558384d6f95531088016f18330ab572fe89f3600098284f7a08a1ea8e7e33666904b918b17cb0db46ced0115b365df2d2c1e29ef3333b0bd4ed297288f7a09ca9c1de5e702ef8f2cbeb89d0d70a584cfe991cf7bb65b1748deeb17775232e5b53f55bc2d1b08c494a80cb727a2f2361af61464afa85a1afcae978e5d802e6c75e3be60965dab7c37e964efdbe3916e611daa4a1241bad3aeedf6a57a21c87c23bec872452c05816ee9179d021c3cf36843063c687d0b7ec2f481129da715b78d3c6bc1ab2e04bedadb937812497196409c08d7837d133c8bf52aec70689b1b180d8eef2676cb7f216edfaf073f84d71ed41b7376d8fe85c88b40297636a2870278d00b452fd37b41af5e357a2dbe1297b53fb027e9ab83d8622fe3180d6e8fd95b59d5e5bc6b7c0451040ec3b14588b1405410bbb249c15d449dd9afa4d9b42650b26a58fccb869ba1794050a014193ad467452efa3a54bfe6c6d1689bf7de9576eac2d2c2dfd4383a219e2d450b00cd9a70fc5e2a7b8a434ac1fa9c6a2c2c37da065576261d981d870923222a50b225422097a3e59598f548be8c6bc0b4764c02546f11b7cb956f93f164469f3bb6b2e95c7d3db7d979b03457be68c6b227a9bda9b4be639198e89c9a62452162640cf95a00ad339",
    "crypto": {
      "kdf": {
        "function": "pbkdf2",
        "params": {
          "dklen": 32,
          "c": 262144,
          "prf": "hmac-sha256",
          "salt": "cf4cd3d5e53974d54450ab228e4c4afbfb31ec35f169da2b39b7e115fd285e78"
        },
        "message": ""
      },
      "checksum": {
        "function": "sha256",
        "params": {},
        "message": "fe3f157ffcbe91120fd60cde10bf1c6dad27d7693b859fdf15a806baf7fa41d1"
      },
      "cipher": {
        "function": "aes-128-ctr",
        "params": {
          "iv": "9c1043f98617ad44af506887835e3dc2"
        },
        "message": "f65a70134da92d495cf305890b76fd39eb74210c305a9e157f3f4a3f2bc7d4fabd663a8df2503d5afe5d2ac6ff5a242b774ccbd1557255160038c93130a9ad3d4bef60be5b1b88bfa44fc368643252e5627523d4800d959e6c34e7477597b9226051ddd570e96fe1a4347ea0a61cfc8af9a37c536ffc84f6da6ba2541947405397bdd1154a53aec7cef40dbdb9bff2227362e1b5d896d47bbdf48dace9413384f04b828fad0f893d3a941b6cbfc75fff1a860a80d89331103e5402b9ca749464f2b007fd21e9fee134ba915c0d09310ba3b09675e3d7e856f372d6c36826a3798cbfb46e05a240ae79bab6e7151921e1407c5def5a07892a1fc8dcff3f05eb320fcb025f225897972de64e8a5580f92de015b4e3b2bfe4c525ec18c7a58e9b780a674428cc15117d5f824bba43fa642c333062380419b761662d3c1961b2858d85830b97b1a7fb49346108bc6957f5292b7bf92091f7e180cc901383d3ffa6861aa7e6bfabdc9e157a42914c1be049ce896f97210f4baa7934f85a6c7123f53393df2a2a64b05ffe81e9568c616f4433eb0eeb81d18c0882567482bc3308b1c2a051c9dfa986200c4f86e64dabe3c5581a76d93c4a2f3a0dc3ea4d3fcd62df3ec612d45a4d2241949808f1c9a14343a77b560725b619cc6de1afd8175ca7a8c0f829faa02e03995aebb25db91f9c0252227981b0af7d9dbb2220b34aa32d7d6304e32d1d08de3f662e5b723791714370939a181adeb5dcb2490db142942fc165b369b65f44c7247ae4ca0f68d1a552fc15b5f33147f0580b70599d585c3aaeccbc7b6b38386397fe733dc34131a84c3c7d52be0ad8cb7c077c1cde879ef2f98d121572874cdce22e8a7e628a73d7e54ccde2406bfd726d471ef7d7f48931e3acb0875f43bbb59607c4c833d9efdd1ec8a6b5d7c1c7777680535fbfad5c8df872d54622a0781357d210ef1908ddd05719c479e8fed29daa6919e55aefbee6ee8202324c00585622cb1d304fb75c80b07f4f0529b4fddecb2b4876ac3be0a09041427f2bcb14d9ea57f6984028bc7252f7beef3de5d0a6c906896daa64509b02fb1507230d183b1b890e5d2be86dccac10dbc06ab700b26ae3b15507969cbfadea25f41f4f88d60714f329547015f27ceee5b27b3115e93401f3547ca40880aed57ec49f01390188f450b24fff38002db810f4a9b378d23f75d82ff96e96ad8fa428ed0124767a4dae885819148bd2845ee09b97ef0836110af16f95a0cf9058ac779e197877e65cbdce09a7e5798db1b324e4d7ba6367bcf7da4d400ac59ceec71be4683eadfcc434eb06dbd1f5382190067cb30a2f3b4c54181b81d35087c03289b5307685e0fb12d625dee46466e4236a817a2ccc8e522dbcef5929cf1b5d00f20d56aa4e8d2db09d7145c4465adf029c9911efcf1634a41bdcf2284dee6893c9efc15ee97dddb6caf6748bb1ea1404f8f682e319bea327be96fadd0aef1f3d58029593734905faccbdb6093dc92b5398f43c397e69e30e47377cee9fe41bd7432a94c457bef39307cdd7c3b904776267d893f0704e8a878727c9c3e414469c5628771c9773699a9869f0ab4d43a7b2c08a87a20b8a6f082034bb51e638b94f38ef7f70a4151603823c8fb8a4af2237680560c4f518faa1c1e03157fdda3520ae3863ca548ee6cc767b18750eeadf869382e8d231a5b29cd1c6af2e0cb2d1fe154dee71feaee60c4547da431e916956cfc01bac9e4708d3fb72016bb8ba9436094f83ab178f75f9a78df1159f77461aba8c1ace1a84e80f84a8df45fa37e504a17651732636fbd3a37d6fd78be851b8db40ba414823bbd5671d079393d9fe34e8b73b9e472ba81840adf833cf9c9ec5014bc5c40d39e0c970819122e85b4a3d117417e7d1ca510dd79673c84bdcc86d751993019028cdcb6a3b45d7ad71f59805e828bac9c978086ad014f2173369ae40ce6c064139fbbd48e8454781152ef5546035de6a50a58c57e8a7e1105227f15cc0b6148d1d79b15c8aa9fa6be48d5dea040acc238c1adb2c1c3ebe8db8e6a2f13798e75b01bc7cce4f4d53c6c5771a54b07ff9cf5046f3ea920a219449f3c188c6027226751f08c6d0ec0f36ed0329b85bd81cc857f01b32c338706243b4a85ff8b95ed7b5635d0ca91ae16acb2b79eabea62c8a6e8e5814fbcd3ef17bb7f547ae77a60b6a3426495e80acfdb6a2e1da74972a5b53f32972bdff0a17b051336ab0a2e65a654dfb1c79902a5d593f15d75d2dc546274bdaa83d5d041be7e53d973865a207e9193b81886220b342ab113968d6414794435824842c159e5911f44774b0744e2fa88394f50dcd4eb46a3b8899ed50faee7412038264654814613fa50b0727281e8f8df2e107990635312226294c01407a1ff1cfd8f0867e8a3413b5bcda6183c189013d70e45def8ff41e720468ae9c816ccecc9b2df4e0de539597a8da7e55b9f5081471062e7b5716ab2b345d21737fcb5f5b42e72ca44960d12026ea653a7755549b5f413c9b9aafef2dda81e44b6c51f8ab44d064705ffdee94d5298eeb7face9fbb35fb26d40712be73a58cc59584fec17976ce5ec564ecd062a1deee737fa74e904e27cd137a69392abc9e46052fc46f57eb3cf6cedf2ef39cc0f685cd32a9cfaade27375aa6ea563ccf1c28b0c1f842312b6a5219df8a4222aaa3ba86a1445b036178a70874bdb4ab0d6b2a1818f64a103086933d47ebd9abe772c05d364417a67d3de44d4512c2231db6c45ad7d374b2d7446fa1a14aac44f21f7c883c1bbd0db3b66cee1d34754eb49fad04143a636e81b34ed06c49fde453b58087822ffe21e551f5bd3bd2de1664c3a03ddbfb550ad4facefdc452def1c96a16ecd0fde934e86fe14b246e9"
      }
    }
  }

The keyfetching from this cache file takes a constant time of ~4 seconds (even if the number of keys is increased significantly). - Majority of the time still spent on decrypting this file, as this cached file is still enrypted.

Would it be possible to add functionality to generate this cache file during key generation? To derive this format this code bit is used: https://github.com/ChainSafe/lodestar/blob/unstable/packages/cli/src/cmds/validator/keymanager/keystoreCache.ts#L65-L85

[nflaig]

@barnabasbusa I don't think this will be needed, did some investigation and it looks like the high decryption time is mostly related to us spawning cpu count workers + warm up inside worker running crypto operations. While the worker pool makes sense for a lot of keys, I noticed that Kurtosis uses pbkdf2 keystores with a lower c=2 param while most normal users use scrypt which is much slower per decrypt operation, like 1000x slower than that.

With workers disabled, and some optimization we did in our keystore library

Jul-15 09:11:56.107[]                 info: Connecting to LevelDB database path=/root/.local/share/lodestar/testnet/validator-db
Jul-15 09:11:56.614[]                 info: 100% of local keystores imported. current=64 total=64 rate=7664.67keys/m
Jul-15 09:11:57.514[]                 info: 64 local keystores

This is on par with other clients, around 1 sec to decrypt 64 keystores.

My idea is to add a CLI flag to disable the worker pool, you can than just set this in Kurtosis and would benefit from the speed up there, while for normal users, mostly using scrypt, it would still spawn the workers.

[nflaig]

For anyone running Lodestar in a testing environment with insecure keys (pbkdf2 with c=2), we now have a new validator client CLI flag --disableKeystoresThreadPool in our unstable / next release which will speed up key import by more than 10x.