Failed to find any hits in a Mars ATA HDF5 file with normalisation enabled

[texadactyl]

The HDF5 file was supplied by @wfarah (ATA). HDF5 file directory: /mnt_blpd20/scratch/rawspec_testing/wael_mars (any data centre compute node)

--- File Info ---
DIMENSION_LABELS : [b'time' b'feed_id' b'frequency']
        az_start :                              0.0
       data_type :                                1
            fch1 :                      8355.75 MHz
            foff :          1.9073486328125e-06 MHz
           ibeam :                               -1
      machine_id :                               20
          nbeams :                                1
           nbits :                               32
          nchans :                         50331648
            nfpc :                           262144
            nifs :                                1
     rawdatafile : guppi_59688_63255_3379455_mars_0001-beam0000.0000.raw
     source_name :                             mars
         src_dej :                   -11:39:52.6509
         src_raj :                   335:38:04.0293
    telescope_id :                               -1
           tsamp :                         1.048576
   tstart (ISOT) :          2022-04-19T17:34:15.000
    tstart (MJD) :                59688.73211805556
        za_start :                              0.0

Num ints in file :                               57
      File shape :                (57, 1, 50331648)
--- Selection Info ---
Data selection shape :                (57, 1, 50331648)
Minimum freq (MHz) :                          8355.75
Maximum freq (MHz) :                8451.749998092651

turboSETI found 167 top hits. Output .dat file: dat.txt

Versions 0.0.6 and 0.0.7 findET found 0 hits. Command line used: findET -M 30 -s 10 -g 2 *h5 Used the default gulp exponent of 18 and default kernel of ddsk.

Plots are attached.

[texadactyl]

I tried the other 2 hyperseti kernels but no good news.

[texadactyl]

Tried command line: findET -M 30 -s 10 -g 2 -b 4 *h5 E.T. = 34.6 minutes Still no hits found.

[texadactyl]

Tried this: findET guppi_59688_63255_3379455_mars_0001-beam0000.rawspec.0000.h5 -M 30 -s 10 -g 2 -k ddsk -N

Suppressing normalisation (-N) to the original command line gets a much different result:

find_et: hyperseti version 0.0.7
find_et: TOTAL ELAPSED TIME: 610.14s
findET: Output dataframe:
      drift_rate      f_start           snr  driftrate_idx  channel_idx  beam_idx  boxcar_size      ddsk  n_integration
60    29.103830  8355.992443  1.412186e+11          912.0     127110.0       0.0          1.0  0.910263           57.0
68    29.103830  8356.022957  1.403839e+11          912.0     143108.0       0.0          1.0  0.835296           57.0
58    29.359127  8355.988089  1.402837e+11          920.0     124827.0       0.0          1.0  1.394906           57.0
57    29.710160  8355.984846  1.402820e+11          931.0     123127.0       0.0          1.0  1.070693           57.0
63    29.103830  8356.006792  1.402713e+11          912.0     134633.0       0.0          1.0  0.754337           57.0
..          ...          ...           ...            ...          ...       ...          ...       ...            ...
116   29.997369  8451.735167  7.020863e+10          940.0   50323871.0       0.0          1.0  1.072887           57.0
119   29.582512  8451.745583  7.018886e+10          927.0   50329332.0       0.0          1.0  1.010095           57.0
118   29.933545  8451.741478  6.988912e+10          938.0   50327180.0       0.0          1.0  1.065434           57.0
2     29.263391  8451.254757  6.954344e+10          917.0   50071998.0       0.0          1.0  1.266721           57.0
0     29.231479  8451.250082  6.922097e+10          916.0   50069547.0       0.0          1.0  1.063222           57.0

[22964 rows x 9 columns]

It seems that the issue might be in normalize.py or its sole caller in hyperseti.py:run_pipeline.

[texadactyl]

Tried this: findET guppi_59688_63255_3379455_mars_0001-beam0000.rawspec.0000.h5 -M 30 -s 10 -g 2 -k ddsk -F

Suppressing spectral kurtosis flagging but keeping normalisation (-F) to the original command line gets a 3rd result:

find_et: hyperseti version 0.0.7
find_et: TOTAL ELAPSED TIME: 345.27s
findET: Output dataframe:
     drift_rate      f_start         snr  driftrate_idx  channel_idx  beam_idx  boxcar_size      ddsk  n_integration
0     7.786551  8430.762636  447.462463          244.0   39328225.0       0.0          1.0  0.884867           57.0
0     7.786551  8431.262636  447.760498          244.0   39590369.0       0.0          1.0  0.877170           57.0
3    15.605014  8445.562977   21.179022          489.0   47087866.0       0.0          1.0  1.508339           57.0
1    15.573102  8445.530624   16.936077          488.0   47070904.0       0.0          1.0  1.948037           57.0
5    15.573102  8445.577690   16.721777          488.0   47095580.0       0.0          1.0  2.325269           57.0
..         ...          ...         ...            ...          ...       ...          ...       ...            ...
30   15.573102  8446.730648   11.412288          488.0   47700062.0       0.0          1.0  2.227722           57.0
25   15.636926  8446.655651   11.062871          490.0   47660742.0       0.0          1.0  2.347433           57.0
29   15.573102  8446.718868   10.911513          488.0   47693886.0       0.0          1.0  2.083316           57.0
20   15.573102  8446.360062   10.551646          488.0   47505768.0       0.0          1.0  2.356349           57.0
26   15.668839  8446.687998   10.045910          491.0   47677701.0       0.0          1.0  1.982852           57.0

[106 rows x 9 columns]

Note that findET did not turn up any hits beyond a drift rate of 16 Hz/s even though we know that there are many. So, findET did find Maven, I guess.

[telegraphic]

FYI, I'm looking into this again. It's a bug in the spectral kurtosis flagging -- will look into it more next week

[telegraphic]

Related technote: https://github.com/UCBerkeleySETI/hyperseti/blob/master/docs/technotes/09_sk_flagging_revisited.ipynb

[telegraphic]

Notes in https://github.com/UCBerkeleySETI/hyperseti/blob/master/docs/technotes/10_mars_hyperseti_0p3.ipynb