search

rnpgp / rnp

RNP: high performance C++ OpenPGP library used by Mozilla Thunderbird
https://www.rnpgp.org
Other
192 stars 54 forks source link

Find ways to reduce test times #1584

Open ronaldtse opened 2 years ago

ronaldtse commented 2 years ago

Here are the results from the latest runs after CI speed improvements in #1582 .

Most of the time spent now is in the "Build and Test" stage. I've taken the numbers from the most recent run.

Some of the tests should really not be running so slow, and the point of the test may not even relate to testing time. For some it may be due to reduced entropy of the container.

Test Ubuntu Debian 9
cli_tests-Compression 291.32 sec 383.28 sec
cli_tests-EncryptEcdh 8.41 sec 11.13 sec
cli_tests-EncryptElgamal 123.28 sec 230.49 sec
cli_tests-EncryptSignRSA 13.95 sec 20.05 sec
cli_tests-Encryption 549.43 sec 652.10 sec
cli_tests-Keystore 83.89 sec 194.48 sec
cli_tests-Misc 79.50 sec 290.47 sec
cli_tests-SignDSA 105.29 sec 332.32 sec
cli_tests-SignDefault 73.60 sec 116.43 sec
cli_tests-SignECDSA 8.28 sec 10.70 sec
rnp_tests.cipher_test_success 0.07 sec 0.06 sec
rnp_tests.ecdh_decryptionNegativeCases 0.14 sec 0.12 sec
rnp_tests.ecdh_roundtrip 0.30 sec 0.49 sec
rnp_tests.ecdsa_signverify_success 0.34 sec 0.55 sec
rnp_tests.generatekeyECDSA_explicitlySetBiggerThanNeededDigest_ShouldSuceed 0.13 sec 0.17 sec
rnp_tests.generatekeyECDSA_explicitlySetSmallOutputDigest_DigestAlgAdjusted 0.15 sec 0.16 sec
rnp_tests.generatekeyECDSA_explicitlySetUnknownDigest_ShouldFail 0.05 sec 0.05 sec
rnp_tests.generatekey_multipleUserIds_ShouldFail 0.04 sec 0.04 sec
rnp_tests.hash_test_success 0.05 sec 0.07 sec
rnp_tests.issue_1030_rnpkeys_secret_keys_unprotected 3.14 sec 5.73 sec
rnp_tests.key_grip 0.55 sec 1.11 sec
rnp_tests.pkcs1_rsa_test_success 0.46 sec 0.93 sec
rnp_tests.raw_elgamal_random_key_test_success 24.58 sec 66.39 sec
rnp_tests.rnp_test_eddsa 0.03 sec 0.04 sec
rnp_tests.rnp_test_x25519 0.04 sec 0.08 sec
rnp_tests.rnpkeys_exportkey_verifyUserId 0.83 sec 2.08 sec
rnp_tests.rnpkeys_generatekey_testEncryption 2.49 sec 4.72 sec
rnp_tests.rnpkeys_generatekey_testExpertMode 32.50 sec 126.46 sec
rnp_tests.rnpkeys_generatekey_testSignature 12.88 sec 25.85 sec
rnp_tests.rnpkeys_generatekey_verifySupportedHashAlg 13.41 sec 27.12 sec
rnp_tests.rnpkeys_generatekey_verifyUserIdOption 6.88 sec 13.06 sec
rnp_tests.rnpkeys_generatekey_verifykeyHomeDirNoPermission 0.83 sec 1.79 sec
rnp_tests.rnpkeys_generatekey_verifykeyHomeDirOption 2.17 sec 4.71 sec
rnp_tests.rnpkeys_generatekey_verifykeyKBXHomeDirOption 2.08 sec 4.19 sec
rnp_tests.s2k_iteration_encode_decode 0.04 sec 0.04 sec
rnp_tests.s2k_iteration_tuning 0.53 sec 0.53 sec
rnp_tests.sm2_roundtrip 0.13 sec 0.22 sec
rnp_tests.sm2_sha256_signature_test 0.09 sec 0.15 sec
rnp_tests.sm2_sm3_signature_test 0.09 sec 0.12 sec
rnp_tests.test_cli_dump 0.05 sec 0.06 sec
rnp_tests.test_cli_examples 5.75 sec 12.08 sec
rnp_tests.test_cli_g10_operations 42.98 sec 67.85 sec
rnp_tests.test_cli_logname 0.07 sec 0.05 sec
rnp_tests.test_cli_rnp 2.08 sec 2.72 sec
rnp_tests.test_cli_rnp_keyfile 30.60 sec 96.74 sec
rnp_tests.test_cli_rnpkeys 1.20 sec 2.28 sec
rnp_tests.test_cli_rnpkeys_genkey 24.19 sec 41.37 sec
rnp_tests.test_cli_rnpkeys_unicode 0.04 sec 0.05 sec
rnp_tests.test_dsa_roundtrip 83.98 sec 139.46 sec
rnp_tests.test_dsa_verify_negative 5.92 sec 21.52 sec
rnp_tests.test_ffi_add_userid 0.34 sec 0.45 sec
rnp_tests.test_ffi_aead_params 0.48 sec 0.54 sec
rnp_tests.test_ffi_calculate_iterations 0.04 sec 0.04 sec
rnp_tests.test_ffi_clear_keys 0.17 sec 0.28 sec
rnp_tests.test_ffi_customized_enarmor 0.03 sec 0.04 sec
rnp_tests.test_ffi_dearmor_edge_cases 0.03 sec 0.04 sec
rnp_tests.test_ffi_decrypt_edge_cases 0.86 sec 1.20 sec
rnp_tests.test_ffi_decrypt_wrong_mpi_bits 2.17 sec 2.72 sec
rnp_tests.test_ffi_detached_cleartext_signed_input 0.10 sec 0.16 sec
rnp_tests.test_ffi_detached_verify_input 0.10 sec 0.16 sec
rnp_tests.test_ffi_detect_key_format 0.04 sec 0.04 sec
rnp_tests.test_ffi_elgamal4096 43.81 sec 151.90 sec
rnp_tests.test_ffi_enable_debug 0.08 sec 0.05 sec
rnp_tests.test_ffi_enarmor_dearmor 0.04 sec 0.12 sec
rnp_tests.test_ffi_encrypt_and_sign 2.19 sec 2.68 sec
rnp_tests.test_ffi_encrypt_pass 0.85 sec 0.94 sec
rnp_tests.test_ffi_encrypt_pass_provider 0.59 sec 0.61 sec
rnp_tests.test_ffi_encrypt_pk 0.94 sec 1.17 sec
rnp_tests.test_ffi_encrypt_pk_key_provider 0.54 sec 0.70 sec
rnp_tests.test_ffi_encrypt_pk_subkey_selection 0.97 sec 1.25 sec
rnp_tests.test_ffi_exception 0.03 sec 0.03 sec
rnp_tests.test_ffi_export_revocation 3.40 sec 4.18 sec
rnp_tests.test_ffi_file_output 0.19 sec 0.29 sec
rnp_tests.test_ffi_get_signature_type 0.07 sec 0.10 sec
rnp_tests.test_ffi_homedir 0.11 sec 0.18 sec
rnp_tests.test_ffi_import_keys_check_pktlen 0.04 sec 0.05 sec
rnp_tests.test_ffi_import_signatures 0.08 sec 0.12 sec
rnp_tests.test_ffi_iterated_key_import 10.01 sec 32.08 sec
rnp_tests.test_ffi_key_default_subkey 0.13 sec 0.23 sec
rnp_tests.test_ffi_key_dump 0.10 sec 0.16 sec
rnp_tests.test_ffi_key_dump_edge_cases 0.03 sec 0.04 sec
rnp_tests.test_ffi_key_export 0.19 sec 0.33 sec
rnp_tests.test_ffi_key_export_autocrypt 0.35 sec 0.59 sec
rnp_tests.test_ffi_key_export_customized_enarmor 3.21 sec 6.36 sec
rnp_tests.test_ffi_key_generate_algnamecase 66.38 sec 129.96 sec
rnp_tests.test_ffi_key_generate_dsa 32.16 sec 78.33 sec
rnp_tests.test_ffi_key_generate_ecdsa 0.08 sec 0.15 sec
rnp_tests.test_ffi_key_generate_eddsa 0.05 sec 0.04 sec
rnp_tests.test_ffi_key_generate_ex 10.22 sec 200.70 sec
rnp_tests.test_ffi_key_generate_expiry_32bit 0.70 sec 1.21 sec
rnp_tests.test_ffi_key_generate_misc 4.89 sec 9.10 sec
rnp_tests.test_ffi_key_generate_protection 4.72 sec 8.30 sec
rnp_tests.test_ffi_key_generate_rsa 4.87 sec 10.11 sec
rnp_tests.test_ffi_key_generate_sm2 0.09 sec 0.12 sec
rnp_tests.test_ffi_key_get_protection_info 5.69 sec 6.95 sec
rnp_tests.test_ffi_key_import_edge_cases 0.72 sec 1.82 sec
rnp_tests.test_ffi_key_import_gpg_s2k 2.83 sec 3.68 sec
rnp_tests.test_ffi_key_iter 0.10 sec 0.17 sec
rnp_tests.test_ffi_key_protection_change 12.36 sec 29.16 sec
rnp_tests.test_ffi_key_remove 4.37 sec 12.26 sec
rnp_tests.test_ffi_key_revoke 4.58 sec 5.47 sec
rnp_tests.test_ffi_key_set_expiry 18.63 sec 22.91 sec
rnp_tests.test_ffi_key_set_expiry_multiple_uids 4.58 sec 5.70 sec
rnp_tests.test_ffi_key_signatures 0.12 sec 0.19 sec
rnp_tests.test_ffi_key_to_json 0.10 sec 0.11 sec
rnp_tests.test_ffi_key_userid_dump_has_no_special_chars 0.99 sec 2.09 sec
rnp_tests.test_ffi_keygen_json_pair 0.50 sec 0.95 sec
rnp_tests.test_ffi_keygen_json_pair_dsa_elg 12.00 sec 748.69 sec
rnp_tests.test_ffi_keygen_json_primary 0.11 sec 0.08 sec
rnp_tests.test_ffi_keygen_json_sub 0.46 sec 1.01 sec
rnp_tests.test_ffi_keygen_json_sub_pass_required 1.05 sec 1.40 sec
rnp_tests.test_ffi_keys_import 19.88 sec 70.09 sec
rnp_tests.test_ffi_literal_filename 0.93 sec 1.18 sec
rnp_tests.test_ffi_literal_packet 0.03 sec 0.03 sec
rnp_tests.test_ffi_load_keys 0.50 sec 0.89 sec
rnp_tests.test_ffi_load_save_keys_to_utf8_path 0.21 sec 0.36 sec
rnp_tests.test_ffi_load_userattr 0.04 sec 0.04 sec
rnp_tests.test_ffi_locate_key 0.11 sec 0.10 sec
rnp_tests.test_ffi_malformed_keys_import 10.58 sec 30.43 sec
rnp_tests.test_ffi_mdc_8k_boundary 1.95 sec 2.43 sec
rnp_tests.test_ffi_op_set_compression 0.90 sec 1.18 sec
rnp_tests.test_ffi_op_set_hash 0.93 sec 1.18 sec
rnp_tests.test_ffi_op_verify_get_protection_info 6.94 sec 8.01 sec
rnp_tests.test_ffi_op_verify_recipients_info 9.32 sec 14.90 sec
rnp_tests.test_ffi_op_verify_sig_count 1.81 sec 2.27 sec
rnp_tests.test_ffi_output_to_armor 0.06 sec 0.10 sec
rnp_tests.test_ffi_pkt_dump 0.03 sec 0.04 sec
rnp_tests.test_ffi_remove_signature 0.15 sec 0.23 sec
rnp_tests.test_ffi_remove_signatures 0.55 sec 0.93 sec
rnp_tests.test_ffi_remove_uid 0.06 sec 0.08 sec
rnp_tests.test_ffi_revocations 0.12 sec 0.19 sec
rnp_tests.test_ffi_rnp_guess_contents 0.07 sec 0.04 sec
rnp_tests.test_ffi_rnp_key_get_primary_fprint 0.06 sec 0.10 sec
rnp_tests.test_ffi_rnp_key_get_primary_grip 0.07 sec 0.10 sec
rnp_tests.test_ffi_rnp_request_password 0.03 sec 0.04 sec
rnp_tests.test_ffi_rsa_v3_dump 0.07 sec 0.03 sec
rnp_tests.test_ffi_save_keys 0.28 sec 0.50 sec
rnp_tests.test_ffi_secret_sig_import 1.78 sec 2.16 sec
rnp_tests.test_ffi_sig_validity 0.23 sec 0.38 sec
rnp_tests.test_ffi_signatures 0.94 sec 1.19 sec
rnp_tests.test_ffi_signatures_detached 0.94 sec 1.19 sec
rnp_tests.test_ffi_signatures_detached_memory 0.94 sec 1.18 sec
rnp_tests.test_ffi_signatures_detached_memory_g10 0.79 sec 1.01 sec
rnp_tests.test_ffi_signatures_dump 0.08 sec 0.12 sec
rnp_tests.test_ffi_signatures_memory 0.93 sec 1.19 sec
rnp_tests.test_ffi_stripped_keys_import 0.14 sec 0.18 sec
rnp_tests.test_ffi_supported_features 0.07 sec 0.05 sec
rnp_tests.test_ffi_uid_properties 0.03 sec 0.04 sec
rnp_tests.test_ffi_uid_validity 0.09 sec 0.14 sec
rnp_tests.test_ffi_version 0.05 sec 0.04 sec
rnp_tests.test_forged_key_validate 0.27 sec 0.50 sec
rnp_tests.test_fuzz_dump 1.34 sec 1.35 sec
rnp_tests.test_fuzz_keyimport 0.09 sec 0.15 sec
rnp_tests.test_fuzz_keyring 0.03 sec 0.04 sec
rnp_tests.test_fuzz_keyring_g10 0.03 sec 0.03 sec
rnp_tests.test_fuzz_keyring_kbx 0.03 sec 0.04 sec
rnp_tests.test_fuzz_verify_detached 0.23 sec 5.14 sec
rnp_tests.test_generated_key_sigs 1.00 sec 2.03 sec
rnp_tests.test_issue_1115 0.06 sec 0.10 sec
rnp_tests.test_issue_1171_key_import_and_remove 0.07 sec 0.11 sec
rnp_tests.test_kbx_nsigs 6.23 sec 22.71 sec
rnp_tests.test_key_add_userid 0.55 sec 0.72 sec
rnp_tests.test_key_expiry_direct_sig 1.78 sec 2.16 sec
rnp_tests.test_key_import 7.75 sec 15.87 sec
rnp_tests.test_key_prefs 0.05 sec 0.05 sec
rnp_tests.test_key_protect_load_pgp 1.46 sec 1.60 sec
rnp_tests.test_key_protect_sec_data 2.74 sec 4.51 sec
rnp_tests.test_key_store_search 0.08 sec 0.05 sec
rnp_tests.test_key_store_search_by_name 0.05 sec 0.09 sec
rnp_tests.test_key_unlock_pgp 1.36 sec 1.69 sec
rnp_tests.test_key_validate 0.23 sec 0.25 sec
rnp_tests.test_key_validity 0.14 sec 0.22 sec
rnp_tests.test_large_mpi_rsa_priv 94.81 sec 306.71 sec
rnp_tests.test_large_mpi_rsa_pub 0.91 sec 1.84 sec
rnp_tests.test_large_packet 145.15 sec 191.07 sec
rnp_tests.test_load_armored_pub_sec 0.13 sec 0.24 sec
rnp_tests.test_load_check_bitfields_and_times 0.10 sec 0.12 sec
rnp_tests.test_load_check_bitfields_and_times_v3 0.03 sec 0.05 sec
rnp_tests.test_load_g10 13.62 sec 16.72 sec
rnp_tests.test_load_keyring_and_count_pgp 0.12 sec 0.16 sec
rnp_tests.test_load_merge 2.90 sec 3.74 sec
rnp_tests.test_load_public_from_secret 0.13 sec 0.24 sec
rnp_tests.test_load_subkey 0.09 sec 0.15 sec
rnp_tests.test_load_user_prefs 0.09 sec 0.10 sec
rnp_tests.test_load_v3_keyring_pgp 0.07 sec 0.04 sec
rnp_tests.test_load_v4_keyring_pgp 0.06 sec 0.10 sec
rnp_tests.test_log_switch 0.03 sec 0.04 sec
rnp_tests.test_partial_length_first_packet_256 0.08 sec 0.12 sec
rnp_tests.test_partial_length_first_packet_length 0.52 sec 0.61 sec
rnp_tests.test_partial_length_largest 36.72 sec 48.06 sec
rnp_tests.test_partial_length_public_key 0.07 sec 0.06 sec
rnp_tests.test_partial_length_signature 0.06 sec 0.10 sec
rnp_tests.test_partial_length_zero_last_chunk 0.09 sec 0.15 sec
rnp_tests.test_pipe 0.03 sec 0.03 sec
rnp_tests.test_pipe_dest_error 0.03 sec 0.03 sec
rnp_tests.test_pipe_source_error 0.03 sec 0.03 sec
rnp_tests.test_rnp_access 0.03 sec 0.03 sec
rnp_tests.test_rnp_mkstemp 0.03 sec 0.03 sec
rnp_tests.test_rnpcfg 0.03 sec 0.04 sec
rnp_tests.test_rnpcfg_get_expiration 0.03 sec 0.03 sec
rnp_tests.test_s2k_iterations 9.79 sec 11.75 sec
rnp_tests.test_stream_814_dearmor_double_free 0.12 sec 0.05 sec
rnp_tests.test_stream_825_dearmor_blank_line 0.06 sec 0.09 sec
rnp_tests.test_stream_cache 0.03 sec 0.03 sec
rnp_tests.test_stream_dearmor_edge_cases 0.03 sec 0.05 sec
rnp_tests.test_stream_deep_packet_nesting 2.16 sec 2.26 sec
rnp_tests.test_stream_dumper 0.08 sec 0.06 sec
rnp_tests.test_stream_dumper_y2k38 0.03 sec 0.03 sec
rnp_tests.test_stream_file 0.04 sec 0.05 sec
rnp_tests.test_stream_key_decrypt 15.11 sec 16.13 sec
rnp_tests.test_stream_key_encrypt 13.98 sec 16.66 sec
rnp_tests.test_stream_key_load 0.05 sec 0.05 sec
rnp_tests.test_stream_key_load_errors 0.82 sec 1.26 sec
rnp_tests.test_stream_key_signature_validate 0.76 sec 1.53 sec
rnp_tests.test_stream_key_signatures 0.14 sec 0.22 sec
rnp_tests.test_stream_memory 0.04 sec 0.04 sec
rnp_tests.test_stream_memory_discard 0.04 sec 0.04 sec
rnp_tests.test_stream_signatures 1.07 sec 1.48 sec
rnp_tests.test_stream_signatures_revoked_key 0.07 sec 0.04 sec
rnp_tests.test_stream_verify_no_key 0.25 sec 0.25 sec
rnp_tests.test_stream_z 30.17 sec 26.15 sec
rnp_tests.test_sxp_depth 0.03 sec 0.03 sec
rnp_tests.test_utils_hex2bin 0.03 sec 0.03 sec
rnp_tests.test_utils_list 0.03 sec 0.04 sec
rnp_tests.test_validate_key_material 54.00 sec 162.11 sec
rnp_tests.test_y2k38 0.03 sec 0.03 sec
ruby-rnp 89.96 sec (not run?)
setupTestData 0.03 sec 0.04 sec
Total Test time (real) 1299.03 sec 2857.73 sec
maxirmx commented 2 years ago

The only idea that I have is to rewrite one test from python to C++ and see the impact. It can be an order (10x) but it is a suggestion made on a very shaky grounds. I will try it.

ni4 commented 2 years ago

Most of CLI tests were written to check compatibility with GnuPG on initial steps, processing large amounts of random data/generating a lot of keys and so on. Right now we do not need to generate new RSA keys in each test, compress huge files and so on. I would suggest to add some CLI key, like --run-expensive-tests, and modify tests to do only necessary checks by default.

ni4 commented 2 years ago
andrey-utkin commented 2 years ago

In fact our typical CI runner spends 8+ minutes just setting up the stage for configuration and building the codebase. See e.g. https://github.com/rnpgp/rnp/runs/7343968509?check_suite_focus=true The relevant bits are

Run yum -y install git - 1m 0s Setup environment - 1m 25s Build cache - 6m 12s

Only after that comes

tests 1m 57s

That's for cli_tests; rnp_tests take ~2m 45s.

I think producing Docker images for the few prerequisite configurations we use is worth it. CI will go brrr!

ronaldtse commented 2 years ago

@andrey-utkin we did have a long discussion on Docker images some time ago. Perhaps this is a good time to do it!

ni4 commented 2 years ago

Yeah, we discussed year or so ago that it would be good to have Docker images with installed CI pre-requisites, just didn't get to this yet. Would be helpful with local debugging/testing as well.