search

irods / irods_testing_environment

BSD 3-Clause "New" or "Revised" License
8 stars 15 forks source link

[#96] Use queue.Queue to manage test list #122

Closed alanking closed 2 years ago

alanking commented 2 years ago

irods_testing_environment.test_manager now uses queue.Queue to manage the test list. Instead of evenly dealing out tests at the start of a run to each test_runner, the test_manager now adds the test list to a Queue and passes the Queue to the test_runner.run. The test_runners get tests from the Queue and runs them until there are no more tests in the queue.

alanking commented 2 years ago

Tried this out with unit tests and it worked, still need to make sure core and plugin tests work, but I don't see why they wouldn't given that test_runner implementations do not concern themselves with where the tests come from

trel commented 2 years ago

This is very cool.

korydraughn commented 2 years ago

Can't wait to see what the runtime numbers look like.

alanking commented 2 years ago

Ran core tests against main and it completed in 3 hours and 27 minutes. That's not too bad, but a tad disappointing. If we split up the tests more, I think we might start to see the real benefits start to manifest. I am finally fed up with the way the test success messages are done, so I'm going to push a commit to make those a little better: Now:

2022-10-14 13:00:50,730 ERROR - [ubuntu-2004-postgres-1012_irods-catalog-provider_1]: cmd succeeded [['python3', '/var/lib/irods/scripts/run_tests.py', '--run_specific_test', 'test_ipwd', '--xml_output']]
2022-10-14 13:00:50,876 ERROR - [ubuntu-2004-postgres-1012_irods-catalog-provider_2]: cmd succeeded [['python3', '/var/lib/irods/scripts/run_tests.py', '--run_specific_test', 'test_iqmod', '--xml_output']]
2022-10-14 13:00:58,503 ERROR - [ubuntu-2004-postgres-1012_irods-catalog-provider_1]: cmd succeeded [['python3', '/var/lib/irods/scripts/run_tests.py', '--run_specific_test', 'test_special_collections', '--xml_output']]
2022-10-14 13:01:03,595 ERROR - [ubuntu-2004-postgres-1012_irods-catalog-provider_1]: cmd succeeded [['python3', '/var/lib/irods/scripts/run_tests.py', '--run_specific_test', 'test_irmtrash', '--xml_output']]
2022-10-14 13:01:05,635 ERROR - [ubuntu-2004-postgres-1012_irods-catalog-provider_2]: cmd succeeded [['python3', '/var/lib/irods/scripts/run_tests.py', '--run_specific_test', 'test_icd', '--xml_output']]
2022-10-14 13:01:05,636 ERROR - tests completed successfully [ubuntu-2004-postgres-1012_irods-catalog-provider_2]
2022-10-14 13:01:11,135 ERROR - [ubuntu-2004-postgres-1012_irods-catalog-provider_1]: cmd succeeded [['python3', '/var/lib/irods/scripts/run_tests.py', '--run_specific_test', 'test_izonereport', '--xml_output']]
2022-10-14 13:01:11,136 ERROR - tests completed successfully [ubuntu-2004-postgres-1012_irods-catalog-provider_1]

What I want:

2022-10-14 12:59:14,087 WARNING - [ubuntu-2004-postgres-1012_irods-catalog-provider_1]: running test [test_ipwd]
2022-10-14 12:59:14,087 WARNING - [ubuntu-2004-postgres-1012_irods-catalog-provider_2]: running test [test_iqmod]
2022-10-14 12:59:18,996 ERROR - [ubuntu-2004-postgres-1012_irods-catalog-provider_1]: test passed [[   4.9084]s] [test_ipwd]
2022-10-14 12:59:18,996 WARNING - [ubuntu-2004-postgres-1012_irods-catalog-provider_1]: running test [test_special_collections]
2022-10-14 12:59:19,220 ERROR - [ubuntu-2004-postgres-1012_irods-catalog-provider_2]: test passed [[   5.1326]s] [test_iqmod]
2022-10-14 12:59:19,221 WARNING - [ubuntu-2004-postgres-1012_irods-catalog-provider_2]: running test [test_icd]
2022-10-14 12:59:26,859 ERROR - [ubuntu-2004-postgres-1012_irods-catalog-provider_1]: test passed [[   7.8629]s] [test_special_collections]
2022-10-14 12:59:26,860 WARNING - [ubuntu-2004-postgres-1012_irods-catalog-provider_1]: running test [test_irmtrash]
2022-10-14 12:59:31,983 ERROR - [ubuntu-2004-postgres-1012_irods-catalog-provider_1]: test passed [[   5.1234]s] [test_irmtrash]
2022-10-14 12:59:31,983 WARNING - [ubuntu-2004-postgres-1012_irods-catalog-provider_1]: running test [test_izonereport]
2022-10-14 12:59:33,960 ERROR - [ubuntu-2004-postgres-1012_irods-catalog-provider_2]: test passed [[  14.7393]s] [test_icd]
2022-10-14 12:59:33,961 ERROR - tests completed successfully [ubuntu-2004-postgres-1012_irods-catalog-provider_2]
2022-10-14 12:59:39,519 ERROR - [ubuntu-2004-postgres-1012_irods-catalog-provider_1]: test passed [[   7.5349]s] [test_izonereport]
2022-10-14 12:59:39,519 ERROR - tests completed successfully [ubuntu-2004-postgres-1012_irods-catalog-provider_1]

Thoughts?

trel commented 2 years ago

definitely like the shortening.

can you not just make everything shown here be INFO? or is that another yak?

alanking commented 2 years ago

I think that's for #41 to decide.

We can also just not put the log level in our log messages:

2022-10-14 14:12:58,956 - [ubuntu-2004-postgres-1012_irods-catalog-provider_1]: running test [test_ipwd]
2022-10-14 14:12:58,956 - [ubuntu-2004-postgres-1012_irods-catalog-provider_2]: running test [test_iqmod]
2022-10-14 14:13:04,124 - [ubuntu-2004-postgres-1012_irods-catalog-provider_1]: test passed [[   5.1682]s] [test_ipwd]
2022-10-14 14:13:04,125 - [ubuntu-2004-postgres-1012_irods-catalog-provider_1]: running test [test_special_collections]
2022-10-14 14:13:04,341 - [ubuntu-2004-postgres-1012_irods-catalog-provider_2]: test passed [[   5.3839]s] [test_iqmod]
2022-10-14 14:13:04,341 - [ubuntu-2004-postgres-1012_irods-catalog-provider_2]: running test [test_icd]
2022-10-14 14:13:11,324 - [ubuntu-2004-postgres-1012_irods-catalog-provider_1]: test passed [[   7.1988]s] [test_special_collections]
2022-10-14 14:13:11,324 - [ubuntu-2004-postgres-1012_irods-catalog-provider_1]: running test [test_irmtrash]
2022-10-14 14:13:16,305 - [ubuntu-2004-postgres-1012_irods-catalog-provider_1]: test passed [[   4.9809]s] [test_irmtrash]
2022-10-14 14:13:16,305 - [ubuntu-2004-postgres-1012_irods-catalog-provider_1]: running test [test_izonereport]
2022-10-14 14:13:18,565 - [ubuntu-2004-postgres-1012_irods-catalog-provider_2]: test passed [[  14.2236]s] [test_icd]
2022-10-14 14:13:18,566 - tests completed successfully [ubuntu-2004-postgres-1012_irods-catalog-provider_2]
2022-10-14 14:13:23,756 - [ubuntu-2004-postgres-1012_irods-catalog-provider_1]: test passed [[   7.4507]s] [test_izonereport]
2022-10-14 14:13:23,757 - tests completed successfully [ubuntu-2004-postgres-1012_irods-catalog-provider_1]
alanking commented 2 years ago

Verbal approval given