dsa110 / dsa110-issues

Issue tracker for all DSA-110 work
0 stars 0 forks source link

T3 processing error not handled well #200

Open caseyjlaw opened 1 year ago

caseyjlaw commented 1 year ago

Candidate 221108aaea was an RFI candidate during an RFI storm. It produced a slack message of: Timeout while waiting for 221108aaea filfiles DM=656.388 ibox=32

However, what happens in T3 logging is something like:

Nov 21 19:22:08 lxd110h23 dask-worker[26161]: Function:  run_hdf5copy
Nov 21 19:22:08 lxd110h23 dask-worker[26161]: args:      (DSAEvent(mjds=59891.74387969, snr=10.7801, ibox=32, dm=96.2913, ibeam=64, cntb=1, cn
Nov 21 19:22:08 lxd110h23 dask-worker[26161]: kwargs:    {'lock': <distributed.lock.Lock object at 0x7f7fcfe9ae50>}
Nov 21 19:22:08 lxd110h23 dask-worker[26161]: Exception: "FileNotFoundError('Only 0 of 384.0 hdf5 files found.')"

We should log this better in slack.