search

dsa110 / dsa110-issues

Issue tracker for all DSA-110 work
0 stars 0 forks source link

T3 manager dask failures #152

Open caseyjlaw opened 2 years ago

caseyjlaw commented 2 years ago

This issue will help track issues with T3_manager.py.

  1. On h23?
Mar 10 06:29:42 lxd110h23 dask-worker[19597]: Could not make filplot 220310aaam due to UnboundLocalError. Callback:
Mar 10 06:29:42 lxd110h23 dask-worker[19597]:  File “/home/ubuntu/anaconda3/envs/casa/lib/python3.6/site-packages/dsaT3/T3_manager.py”, line 146, in run_nowait
Mar 10 06:29:42 lxd110h23 dask-worker[19597]:   output_dict[‘candplot’], output_dict[‘probability’] = filf.filplot_entry(datestring,a)
Mar 10 06:29:42 lxd110h23 dask-worker[19597]:  File “/home/ubuntu/anaconda3/envs/casa/lib/python3.6/site-packages/dsaT3/filplot_funcs.py”, line 717, in filplot_entry
Mar 10 06:29:42 lxd110h23 dask-worker[19597]:   fake=fake)
  1. On h23, in the log of dask-worker1:
Mar 10 14:37:54 lxd110h23 dask-worker[23524]: flagging a la DSA
Mar 10 14:38:26 lxd110h23 dask-worker[23524]: /home/ubuntu/anaconda3/envs/casa/lib/python3.6/site-packages/dsaT3/filplot_funcs.py:129: UserWar
Mar 10 14:38:26 lxd110h23 dask-worker[23524]:   fig = plt.figure(figsize=(8,10))
Mar 10 14:38:27 lxd110h23 dask-worker[23524]: {'prob': 9.147103519069333e-09, 'snr_dm0_ibeam': FilterbankBlock(4.2978873, dtype=float32), 'snr
Mar 10 14:38:27 lxd110h23 dask-worker[23524]: Could not make filplot 220310aaaw due to TypeError.  Callback:
Mar 10 14:38:27 lxd110h23 dask-worker[23524]:   File "/home/ubuntu/anaconda3/envs/casa/lib/python3.6/site-packages/dsaT3/T3_manager.py", line 
Mar 10 14:38:27 lxd110h23 dask-worker[23524]:     output_dict['candplot'], output_dict['probability'] = filf.filplot_entry(datestring,a)
Mar 10 14:38:27 lxd110h23 dask-worker[23524]:   File "/home/ubuntu/anaconda3/envs/casa/lib/python3.6/site-packages/dsaT3/filplot_funcs.py", li
Mar 10 14:38:27 lxd110h23 dask-worker[23524]:     fake=fake)
Mar 10 14:38:27 lxd110h23 dask-worker[23524]:   File "/home/ubuntu/anaconda3/envs/casa/lib/python3.6/site-packages/dsaT3/filplot_funcs.py", li
Mar 10 14:38:27 lxd110h23 dask-worker[23524]:     fake=fake)
Mar 10 14:38:27 lxd110h23 dask-worker[23524]:   File "/home/ubuntu/anaconda3/envs/casa/lib/python3.6/site-packages/dsaT3/filplot_funcs.py", li
Mar 10 14:38:27 lxd110h23 dask-worker[23524]:     if classification_dict['snr_dm0_allbeam']>7.0:
Mar 10 14:39:24 lxd110h23 dask-worker[23524]: /data/dsa110/T1/corr01/2022_3_10_1_19_25/fil_220310aaas/220310aaas_9.fil

dask-worker3 shows similar.

This may be relevant to the first one: https://github.com/dask/distributed/issues/3412

caseyjlaw commented 2 years ago

Looks like we have a TypeError when comparing two values: if classification_dict['snr_dm0_allbeam']>7.0:

I'd bet that the "snr_dm0_allbeam" key is set to None or something.