search

HYPERNETS / hypernets_processor

Processor for Hypernets field data to generate the user products
9 stars 3 forks source link

Anomaly handling when anomaly_db is None #123

Closed clemgoyens closed 3 years ago

clemgoyens commented 3 years ago

Currently processing fails when a sequence is incomplete. Should go to next one?

shunt16 commented 3 years ago

How/where does it fail? Is it exiting differently for an incomplete sequence than other errors?

Generally, if an unknown error occurs it should just add the "x" anomaly and go on to the next sequence.

clemgoyens commented 3 years ago

Yes indeed, if I have a sequence with only irradiance data (to check the tilt of the instrument) I have the following error:

Traceback (most recent call last): File "/home/cgoyens/OneDrive/BackUpThinkpadClem/Projects/HYPERNETS/NetworkDesign_D52/DataProcChain/.hypernets/bin/hypernets_sequence_processor", line 11, in load_entry_point('hypernets-processor', 'console_scripts', 'hypernets_sequence_processor')() File "/home/cgoyens/OneDrive/BackUpThinkpadClem/Projects/HYPERNETS/NetworkDesign_D52/DataProcChain/hypernets_processor/hypernets_processor/cli/sequence_processor_cli.py", line 122, in cli main(processor_config_path=PROCESSOR_CONFIG_PATH, job_config_path=job_config_path, to_archive=False) File "/home/cgoyens/OneDrive/BackUpThinkpadClem/Projects/HYPERNETS/NetworkDesign_D52/DataProcChain/hypernets_processor/hypernets_processor/main/sequence_processor_main.py", line 132, in main context.anomaly_db.add_x_anomaly() AttributeError: 'NoneType' object has no attribute 'add_x_anomaly'

But then it stops running cause its looking for the radiance data in the sequence_processor.py

Processing to L1b radiance... Failed: UnboundLocalError("local variable 'L1a_rad' referenced before assignment")

clemgoyens commented 3 years ago

Just check if not none else stop. Not sure if this is a good way to do but OK for now ;)

shunt16 commented 3 years ago

Oh I see, thanks I didn't catch this. Need to keep better in mind what is available in the hypernets_sequence_processor vs hypernets_scheduler, and make sure to test new features with both interfaces.

Issue here, as you figured out, is when you run hypernets_sequence_processor you don't need to have created the network databases so the anomaly can't be recorded in the database. I think if None is a good solution, will think about how to best report the error in the else (runtime warning to keep processing going?)

shunt16 commented 3 years ago

Now I think about it... this will apply everywhere we have context.anomaly_db.add_anomaly(). I will create a more robust solution that solves this problem for all these cases