search

orcasound / orca-action-workflow

Github actions to automate Orcasound tasks.
MIT License
6 stars 6 forks source link

Error handling #3

Open Molkree opened 3 years ago

Molkree commented 3 years ago

Noticed that workflow failed once two weeks ago.

Here's the log.

  File "script.py", line 36, in <module>
    st.spectrogram(outfile=outfile, dbscale=True, wlen = 0.1)
  File "/opt/hostedtoolcache/Python/3.7.10/x64/lib/python3.7/site-packages/obspy/core/stream.py", line 1184, in spectrogram
    spec = tr.spectrogram(**kwargs)
  File "/opt/hostedtoolcache/Python/3.7.10/x64/lib/python3.7/site-packages/obspy/core/trace.py", line 961, in spectrogram
    return spectrogram(data=self.data, **kwargs)
  File "/opt/hostedtoolcache/Python/3.7.10/x64/lib/python3.7/site-packages/obspy/imaging/spectrogram.py", line 134, in spectrogram
    pad_to=mult, noverlap=nlap)
  File "/opt/hostedtoolcache/Python/3.7.10/x64/lib/python3.7/site-packages/matplotlib/mlab.py", line 777, in specgram
    mode=mode)
  File "/opt/hostedtoolcache/Python/3.7.10/x64/lib/python3.7/site-packages/matplotlib/mlab.py", line 374, in _spectral_helper
    result = stride_windows(x, NFFT, noverlap, axis=0)
  File "/opt/hostedtoolcache/Python/3.7.10/x64/lib/python3.7/site-packages/matplotlib/mlab.py", line 252, in stride_windows
Error:     raise ValueError('noverlap must be less than n')
ValueError: noverlap must be less than n

I've looked into it a bit and it seems like it failed on an extremely short recording, just 36 samples. OO.HYVM2..YDH | 2021-04-28T14:15:00.000000Z - 2021-04-28T14:15:00.000547Z | 64000.0 Hz, 36 samples

Might be interesting that it only fails if we do st.decimate(factor=10) step, otherwise it works fine and produces this spectrogram:

image

But I'm not even sure we are interested in such short recordings for this script (or ever).

The above is specific to the spectrogram script but looking at the overall picture I think we need to decide how to handle this kind of situation in future.

Maybe something like this would work: extract looping over .mseed files into separate method, applying target script doing actual work to each recording in the loop. Wrap that script into try...except and set some environmental variable in except block. This way we will process all recordings without failing early and at the same time will be able to fail the whole workflow after uploading results to examine what happened.

But this assumes that our useful script works on individual files. So I can either just modify the existing spectrogram script or extract looping over files for future scripts working on individual files.

Molkree commented 3 years ago

The above is no longer relevant as OOI processing was changed.

At the same time sometimes we have this: Data Segment https://rawdata.oceanobservatories.org/files/RS01SBPS/PC01A/08-HYDBBA103/2021/07/23/./OO-HYVM2--YDH-2021-07-23T02:20:00.000000.mseed Broken I am not sure what this means, this is what OOIPy reports.

And sometimes OOI doesn't return data at all: Client response: Cannot connect to host rawdata.oceanobservatories.org:443 ssl:default [Connect call failed ('128.6.240.153', 443)]