search

pytroll / pytroll-collectors

Collector modules for Pytroll
GNU General Public License v3.0
3 stars 18 forks source link

cat.py fails to check if command ran successfully #75

Open gerritholl opened 3 years ago

gerritholl commented 3 years ago

Describe the bug

When cat.py is used to concatenate EPS files, for example using kai, it fails to test if kai ran successfully. When configuring cat.py with:

[kai_cat]
topic=/collection/metop/avhrr
command=kai -i {input_files} -o {output_file}
output_file_pattern=/data/pytroll/TMP/avhrr/avhrr_{platform_name}_{start_time:%Y%m%d%H%M%S}_{end_time:%Y%m%d%H%M%S}
publish_topic=/cat/metop/avhrr
nameservers=localhost
subscriber_nameserver = localhost

then the command run with kai may be, for example:

kai -i /data/pytroll/IN/HRPT/AVHR_HRP_00_M03_20210119081730Z_20210119082030Z_N_O_20210119081730Z /data/pytroll/IN/HRPT/AVHR_HRP_00_M03_20210119082030Z_20210119082330Z_N_O_20210119082030Z /data/pytroll/IN/HRPT/AVHR_HRP_00_M03_20210119082330Z_20210119082630Z_N_O_20210119082331Z /data/pytroll/IN/HRPT/AVHR_HRP_00_M03_20210119082630Z_20210119082930Z_N_O_20210119082630Z /data/pytroll/IN/HRPT/AVHR_HRP_00_M03_20210119082930Z_20210119083123Z_N_O_20210119082930Z -o /data/pytroll/TMP/avhrr/avhrr_Metop-C_20210119081730_20210119083123

If the output directory /data/pytroll/TMP/avhrr does not exist, the actual kai command will finish with a message to stderr:

Cannot open file '/data/pytroll/TMP/avhrr/avhrr_Metop-C_20210119081730_20210119083123` mode w+

and exit with exit code 1. The cat log however only monitors stdout, happily noting:

[INFO: 2021-01-19 08:31:27 : cat] b'Writing 61469828 bytes (4784 records) to /data/pytroll/TMP/avhrr/avhrr_Metop-C_20210119081730_20210119083123'
[INFO: 2021-01-19 08:31:27 : cat] Sending pytroll://file/cat/avhrr file pytroll@oflks333.dwd.de 2021-01-19T08:31:27.797434 v1.01 application/json {"path": "", "platform_name": "Metop-C", "start_time": "2021-01-19T08:17:30", "end_time": "2021-01-19T08:31:23", "processing_time": "2021-01-19T08:17:30", "sensor": ["avhrr/3"], "orig_platform_name": "M03", "collection_area_id": "eurol", "filename": "avhrr_Metop-C_20210119081730_20210119083123", "uri": "/data/pytroll/TMP/avhrr/avhrr_Metop-C_20210119081730_20210119083123"}

Expected behaviour

I expect that cat.py verifies that kai or whatever command was used exits successfully (exit code 0). If it doesn't, it should copy the stderr output for the process to the logfile and it should not send a posttroll message.

Actual behaviour

In reality cat.py monitors only stdout, appears to assume everything is fine, and sends a posttroll message in any case. It's up to the next program in the chain to realise the expected input file is absent.