search

aodn / content

Tracks AODN Portal content and configuration issues
0 stars 0 forks source link

Burst-averaging code fails when a variable has no good values #453

Open mhidas opened 4 years ago

mhidas commented 4 years ago

The code used to average files from burst-sampling instruments crashes when it encounters a variable that has all its values (within a burst) flagged bad. It should just output a missing value for that burst.

The error in the pipeline log looks like this:

2020-02-27 12:13:35,957 INFO tasks.ANMN_AIMS_MOORING[32642604-00c4-4971-ad23-151321b2c731] Burst-processing IMOS_ANMN-NRS_FZ_20170929T003418Z_NRSYON_FV01_NRSYON-1709-SRF-ECO-PARSB-1.1_END-20180509T233213Z_C-20190308T030455Z.nc
2020-02-27 12:13:52,041 ERROR tasks.ANMN_AIMS_MOORING[32642604-00c4-4971-ad23-151321b2c731] TypeError: unorderable types: int() < NoneType()
Traceback (most recent call last):
  File "/mnt/ebs/pipeline/lib/python3.5/site-packages/aodncore/pipeline/handlerbase.py", line 1038, in run
    self.trigger(transition['trigger'])
  File "/mnt/ebs/pipeline/lib/python3.5/site-packages/transitions/core.py", line 65, in _get_trigger
    return machine.events[trigger_name].trigger(model, *args, **kwargs)
  File "/mnt/ebs/pipeline/lib/python3.5/site-packages/transitions/core.py", line 405, in trigger
    return self.machine._process(func)
  File "/mnt/ebs/pipeline/lib/python3.5/site-packages/transitions/core.py", line 1073, in _process
    return trigger()
  File "/mnt/ebs/pipeline/lib/python3.5/site-packages/transitions/core.py", line 423, in _trigger
    return self._process(event_data)
  File "/mnt/ebs/pipeline/lib/python3.5/site-packages/transitions/core.py", line 433, in _process
    if trans.execute(event_data):
  File "/mnt/ebs/pipeline/lib/python3.5/site-packages/transitions/core.py", line 279, in execute
    machine.callback(func, event_data)
  File "/mnt/ebs/pipeline/lib/python3.5/site-packages/transitions/core.py", line 1031, in callback
    func(*event_data.args, **event_data.kwargs)
  File "/mnt/ebs/pipeline/lib/python3.5/site-packages/aodndata/moorings/handlers.py", line 61, in preprocess
    product_path = create_burst_average_netcdf(f.src_path, self.products_dir)
  File "/mnt/ebs/pipeline/lib/python3.5/site-packages/aodndata/moorings/burst_average.py", line 263, in create_burst_average_netcdf
    burst_vars = create_burst_average_var(input_netcdf_obj)
  File "/mnt/ebs/pipeline/lib/python3.5/site-packages/aodndata/moorings/burst_average.py", line 56, in create_burst_average_var
    return trim_timestamps_burst_vars(burst_vars)
  File "/mnt/ebs/pipeline/lib/python3.5/site-packages/aodndata/moorings/burst_average.py", line 94, in trim_timestamps_burst_vars
    max_non_nan_idx = max(max_non_nan_idx, max_last_non_nan_var)
TypeError: unorderable types: int() < NoneType()
mhidas commented 4 years ago

Sample files that reproduce this issue are in 10-aws-syd:/mnt/wip/ANMN/error_files/burst_averaging_fails/

lbesnard commented 2 years ago

@mhidas not all the files in the folder you mentionned above have the same issue. I tried with the IMOS_ANMN-QLD_KUZ_20120130T120457Z_KIM050_FV01_KIM050-1202-ECO-FLNTUSB-25.5_END-20120807T114049Z_C-20171114T062340Z.nc and the create_burst_average_netcdf could create the file successfully

lbesnard commented 2 years ago

https://github.com/aodn/python-aodndata/pull/279