search

ICRAR / daliuge

The DALiuGE Execution Engine
GNU Lesser General Public License v2.1
24 stars 7 forks source link

Update run-unit-tests.yml to reduce log-spam from stderr #250

Closed myxie closed 3 months ago

myxie commented 4 months ago

Problem

It looks like we have an issue with multiple handlers being created when running the translator, which causes significant log spam for the same lines: 

2024-05-06 08:35:48,666 [ INFO] [     MainThread] dlg.common.reproducibility.reproducibility#init_lgt_repro_data:723 Reproducibility data finished at LGT level
2024-05-06 08:35:48,666 [ INFO] [     MainThread] dlg.common.reproducibility.reproducibility#init_lgt_repro_data:723 Reproducibility data finished at LGT level
2024-05-06 08:35:48,666 [ INFO] [     MainThread] dlg.common.reproducibility.reproducibility#init_lgt_repro_data:723 Reproducibility data finished at LGT level
2024-05-06 08:35:48,666 [ INFO] [     MainThread] dlg.common.reproducibility.reproducibility#init_lgt_repro_data:723 Reproducibility data finished at LGT level
2024-05-06 08:35:48,666 [ INFO] [     MainThread] dlg.common.reproducibility.reproducibility#init_lgt_repro_data:723 Reproducibility data finished at LGT level
2024-05-06 08:35:48,666 [ INFO] [     MainThread] dlg.common.reproducibility.reproducibility#init_lgt_repro_data:723 Reproducibility data finished at LGT level
2024-05-06 08:35:48,666 [ INFO] [     MainThread] dlg.common.reproducibility.reproducibility#init_lgt_repro_data:723 Reproducibility data finished at LGT level
2024-05-06 08:35:48,666 [ INFO] [     MainThread] dlg.common.reproducibility.reproducibility#init_lgt_repro_data:723 Reproducibility data finished at LGT level
2024-05-06 08:35:48,666 [ INFO] [     MainThread] dlg.common.reproducibility.reproducibility#init_lgt_repro_data:723 Reproducibility data finished at LGT level
2024-05-06 08:35:48,666 [ INFO] [     MainThread] dlg.common.reproducibility.reproducibility#init_lgt_repro_data:723 Reproducibility data finished at LGT level
.....

2024-05-06 08:35:48,664 [ INFO] [     MainThread] dlg.dropmake.pg_generator#fill:61 Filling Logical Graph with parameters: {}
2024-05-06 08:35:48,664 [ INFO] [     MainThread] dlg.dropmake.pg_generator#fill:61 Filling Logical Graph with parameters: {}
2024-05-06 08:35:48,664 [ INFO] [     MainThread] dlg.dropmake.pg_generator#fill:61 Filling Logical Graph with parameters: {}
2024-05-06 08:35:48,664 [ INFO] [     MainThread] dlg.dropmake.pg_generator#fill:61 Filling Logical Graph with parameters: {}
2024-05-06 08:35:48,664 [ INFO] [     MainThread] dlg.dropmake.pg_generator#fill:61 Filling Logical Graph with parameters: {}
2024-05-06 08:35:48,664 [ INFO] [     MainThread] dlg.dropmake.pg_generator#fill:61 Filling Logical Graph with parameters: {}
2024-05-06 08:35:48,664 [ INFO] [     MainThread] dlg.dropmake.pg_generator#fill:61 Filling Logical Graph with parameters: {}
....

This makes it quite difficult to see the error failures that are reported in our CI workflows because the logs are polluted by this duplication.

Solution

At some point we should investigate why so many handlers are being created; for the time being, I have added --show-capture=no to the test command, so that pytest ignores stderr output.