Closed valefar-on-discord closed 4 months ago
The multiprocessing module should be the basis for CPU-bound multithreaded in Python: https://docs.python.org/3/library/multiprocessing.html
Some initial results from this improvement. On my modest 8 cores/16 threads CPU, the full process (creation, deposit and verification) of creating 100 keys for mainnet requires 28 seconds with the multiprocess path instead of 4 minutes and 11 seconds without it.
Another test with 1000 keys took 4 minutes and 29 seconds. It seems to scale linearly.
I did a 1000 key generation and got 27.6 minutes with scrypt and 16 minutes with pbkdf2 for the full process. So pbkdf2 + multithreading should be a beast :grin:
It seems like there is some kind of race issue after adding multiprocess support in our test_exit_transaction_menmonic_multiple
test. This is what the CI outputs in a non-deterministic way (sometime it works, sometime it doesn't):
2024-05-09T16:08:06.3531967Z ##[group]Run pytest tests
2024-05-09T16:08:06.3532415Z [36;1mpytest tests[0m
2024-05-09T16:08:06.3586034Z shell: /usr/bin/bash -e {0}
2024-05-09T16:08:06.3586444Z env:
2024-05-09T16:08:06.3586924Z pythonLocation: /opt/hostedtoolcache/Python/3.10.14/x64
2024-05-09T16:08:06.3587572Z PKG_CONFIG_PATH: /opt/hostedtoolcache/Python/3.10.14/x64/lib/pkgconfig
2024-05-09T16:08:06.3588252Z Python_ROOT_DIR: /opt/hostedtoolcache/Python/3.10.14/x64
2024-05-09T16:08:06.3588899Z Python2_ROOT_DIR: /opt/hostedtoolcache/Python/3.10.14/x64
2024-05-09T16:08:06.3589489Z Python3_ROOT_DIR: /opt/hostedtoolcache/Python/3.10.14/x64
2024-05-09T16:08:06.3590163Z LD_LIBRARY_PATH: /opt/hostedtoolcache/Python/3.10.14/x64/lib
2024-05-09T16:08:06.3590621Z ##[endgroup]
2024-05-09T16:08:08.2956600Z ============================= test session starts ==============================
2024-05-09T16:08:08.2958120Z platform linux -- Python 3.10.14, pytest-8.1.1, pluggy-1.4.0
2024-05-09T16:08:08.2959833Z rootdir: /home/runner/work/ethstaker-deposit-cli/ethstaker-deposit-cli
2024-05-09T16:08:08.2961158Z plugins: asyncio-0.23.6
2024-05-09T16:08:08.2962352Z asyncio: mode=strict
2024-05-09T16:08:08.2962952Z collected 711 items
2024-05-09T16:08:08.2963351Z
2024-05-09T16:08:31.6307244Z tests/test_cli/test_existing_mnemonic.py ..... [ 0%]
2024-05-09T16:08:34.7508379Z tests/test_cli/test_exit_transaction_keystore.py .... [ 1%]
2024-05-09T16:08:36.4061359Z tests/test_cli/test_exit_transaction_mnemonic.py .F..... [ 2%]
2024-05-09T16:08:39.2593198Z tests/test_cli/test_generate_bls_to_execution_change.py ... [ 2%]
2024-05-09T16:09:03.8446919Z tests/test_cli/test_new_mnemonic.py ....... [ 3%]
2024-05-09T16:09:08.1184764Z tests/test_cli/test_regeneration.py . [ 3%]
2024-05-09T16:09:08.1195770Z tests/test_credentials.py . [ 3%]
2024-05-09T16:09:16.3293731Z tests/test_deposit.py ........ [ 5%]
2024-05-09T16:09:16.4031321Z tests/test_intl/test_json_schema.py .................................... [ 10%]
2024-05-09T16:09:16.4752560Z .................................... [ 15%]
2024-05-09T16:09:16.5074869Z tests/test_key_handling/test_key_derivation/test_mnemonic.py ........... [ 16%]
2024-05-09T16:09:16.7078626Z ........................................................................ [ 26%]
2024-05-09T16:09:16.9109373Z ........................................................................ [ 36%]
2024-05-09T16:09:19.6588158Z ........................................................................ [ 47%]
2024-05-09T16:09:23.0039761Z ........................................................................ [ 57%]
2024-05-09T16:09:26.3645323Z ........................................................................ [ 67%]
2024-05-09T16:09:29.7298447Z ........................................................................ [ 77%]
2024-05-09T16:09:32.7004665Z ................................................................ [ 86%]
2024-05-09T16:09:32.7586341Z tests/test_key_handling/test_key_derivation/test_path.py ........ [ 87%]
2024-05-09T16:09:32.7728990Z tests/test_key_handling/test_key_derivation/test_tree.py ............... [ 89%]
2024-05-09T16:09:32.8515167Z ........ [ 90%]
2024-05-09T16:09:37.3657005Z tests/test_key_handling/test_keystore.py ............ [ 92%]
2024-05-09T16:09:37.3675366Z tests/test_utils/test_constants.py .. [ 92%]
2024-05-09T16:09:37.9183379Z tests/test_utils/test_crypto.py .......... [ 94%]
2024-05-09T16:09:37.9292972Z tests/test_utils/test_intl.py .......... [ 95%]
2024-05-09T16:09:37.9383465Z tests/test_utils/test_ssz.py ......... [ 96%]
2024-05-09T16:09:40.6441651Z tests/test_utils/test_validation.py ...................... [100%]
2024-05-09T16:09:40.6443173Z
2024-05-09T16:09:40.6443987Z =================================== FAILURES ===================================
2024-05-09T16:09:40.6445580Z ___________________ test_exit_transaction_menmonic_multiple ____________________
2024-05-09T16:09:40.6446349Z
2024-05-09T16:09:40.6449610Z def test_exit_transaction_menmonic_multiple() -> None:
2024-05-09T16:09:40.6450486Z # Prepare folder
2024-05-09T16:09:40.6451457Z my_folder_path = os.path.join(os.getcwd(), 'TESTING_TEMP_FOLDER')
2024-05-09T16:09:40.6452678Z clean_exit_transaction_folder(my_folder_path)
2024-05-09T16:09:40.6453539Z if not os.path.exists(my_folder_path):
2024-05-09T16:09:40.6454459Z os.mkdir(my_folder_path)
2024-05-09T16:09:40.6455167Z
2024-05-09T16:09:40.6455778Z runner = CliRunner()
2024-05-09T16:09:40.6456389Z inputs = []
2024-05-09T16:09:40.6456918Z data = '\n'.join(inputs)
2024-05-09T16:09:40.6457677Z arguments = [
2024-05-09T16:09:40.6458358Z '--language', 'english',
2024-05-09T16:09:40.6459065Z '--non_interactive',
2024-05-09T16:09:40.6459976Z 'exit-transaction-mnemonic',
2024-05-09T16:09:40.6460858Z '--output_folder', my_folder_path,
2024-05-09T16:09:40.6461742Z '--chain', 'mainnet',
2024-05-09T16:09:40.6463056Z '--mnemonic', 'aban aban aban aban aban aban aban aban aban aban aban abou',
2024-05-09T16:09:40.6464318Z '--validator_start_index', '0',
2024-05-09T16:09:40.6465315Z '--validator_indices', '0 1 2 3',
2024-05-09T16:09:40.6466049Z '--epoch', '1234',
2024-05-09T16:09:40.6466468Z ]
2024-05-09T16:09:40.6466982Z result = runner.invoke(cli, arguments, input=data)
2024-05-09T16:09:40.6467458Z
2024-05-09T16:09:40.6467871Z > assert result.exit_code == 0
2024-05-09T16:09:40.6468302Z E AssertionError: assert 1 == 0
2024-05-09T16:09:40.6469046Z E + where 1 = <Result FileExistsError(17, 'File exists')>.exit_code
2024-05-09T16:09:40.6469661Z
2024-05-09T16:09:40.6470047Z tests/test_cli/test_exit_transaction_mnemonic.py:80: AssertionError
2024-05-09T16:09:40.6470698Z =========================== short test summary info ============================
2024-05-09T16:09:40.6471710Z FAILED tests/test_cli/test_exit_transaction_mnemonic.py::test_exit_transaction_menmonic_multiple - AssertionError: assert 1 == 0
2024-05-09T16:09:40.6472741Z + where 1 = <Result FileExistsError(17, 'File exists')>.exit_code
2024-05-09T16:09:40.6473382Z =================== 1 failed, 710 passed in 94.01s (0:01:34) ===================
2024-05-09T16:09:40.7114730Z ##[error]Process completed with exit code 1.
This needs further test and improvements.
Forwarded from
As suggested by Preston, an investigation into building a multithreaded solution for generating keys along with exit messages could drastically improve performance when working with large quantities.