Delete file if exception error not working

[MilkaRadojevic]

At Meteo France, the following problem makes the OPERA production of max reflectivity composite (Cirrus) to exit with error:

When _raveio.open fails to open a corrupted OPERA file, the exception block (Lib/odc_polarQC.py L130) does not delete that file as it is supposed (see a log message down below). In return, the corrupted files is preserved and as such makes OPERA Cirrus compositing code en C++ to abort.

2023-07-11 22:07:06,038 ERROR    ForkPoolWorker-19: T_PAGZ54_C_LYBM_20230711220007.hdf: Traceback (most recent call last):
    File "/SOPRANO/dso/cirrus_modules/toolbox3/Lib/odc_polarQC.py", line 97, in generate
    rio = _raveio.open(ifstr)
    OSError: Failed to open file

It seems that err_msg = traceback.format_exc() does not return the value delete on our Rocky Linux 8.5 server. In another words, a given value should be more flexible or to provide more options.

Thank you. Milka

[guntherhaase]

Hi @MilkaRadojevic,

Did you apply the option "--delete" when you run the toolbox (https://github.com/baltrad/rave/blob/master/bin/odc_toolbox)? By default, "delete" is set to False (https://github.com/baltrad/rave/blob/master/Lib/odc_polarQC.py).

[MilkaRadojevic]

Hi @guntherhaase, Thank you for this useful feedback. OMG I was not aware of that argument at all - it is not used in Odyssey and since I followed the rules from the existing production line the same is applied in Cirrus. Since no tehnical manual exists for Odyssey except running scripts it was guessing game for me. Furhtermore, no explanation how to run odc_toolbox from this Git repository.

I will test this option in developpment mode and if it works well it will be introduced into the planned update of Cirrus.

[MilkaRadojevic]

Hi @guntherhaase ,

I tested odc_toolbox for corrupted files with --delete but no result. This is because Lib/odc_polarQC.py cannot open a ifstr and given try-except block does not cover OSError case. To solve this issue, I corrected the except Exception part by:

except OSError as err_msg:
    os.remove(ifstr)
    rave_pgf_logger.log(logger, "error", "%s: %s - Deleting file." % (fstr, err_msg))

---

The log from testing (toolbox_lybm54_log-solved.txt) shows 2 files available for odc_toolbox and none for compositing since files being erroneous are deleted.

---

Here is what we can conclude:

1. This bug (or no well covered option for open/read file) is present since the Odyssey meaining it is also part of the Odyssey
2. The script odc_polarQC.py should be eventually updated, otherwise we the toolbox does not perform quality control of incoming files.

[guntherhaase]

Hi @MilkaRadojevic,

Great that you have resolved this issue! I'm not sure if you are eligible to check-in your modification to BALTRAD GitHub. If not can you send me your updated script odc_polarQC? Thank you!

[MilkaRadojevic]

Hi @guntherhaase ,

Here is the code: odc_polarQC.py.txt

[guntherhaase]

Hi @MilkaRadojevic,

Finally, we have resolved this issue.

Anders & Günther