IndexError: too many indices for array: array is 1-dimensional, but 3 were indexed

[texadactyl]

Expectation The program below would run to conclusions without errors or it would clearly and consisely log a setigen limitation before throwing an exception.

Input file https://storage.googleapis.com/ata_test_data/turbo_seti_test/guppi_59196_68468_000762_Unknown_0001.rawspec.0000.h5

My program

import setigen as stg
from setigen.waterfall_utils import min_freq, max_freq
from blimpy import Waterfall

DIR = '/seti_data/turbo_seti_issue_135/'
INFILE = DIR + 'guppi_59196_68468_000762_Unknown_0001.rawspec.0000.h5'

def import_bl_file(path_infile):
    r'''
    Using setigen, import a BL file (.fil, .h5).
    '''
    print('Input file:', path_infile)
    wf = Waterfall(path_infile)
    print('Waterfall data loaded?', wf.container.load_data)
    print(wf.info())

    # Instantiate a setigen Frame object
    frame = stg.Frame(waterfall=wf)
    print('Drift rate:', frame.get_drift_rate(0, -1))
    print('Noise stats:', frame.get_noise_stats())
    print('min freq:', min_freq(wf))
    print('max freq:', max_freq(wf))

import_bl_file(INFILE)

Stdout

Input file: /seti_data/turbo_seti_issue_135/guppi_59196_68468_000762_Unknown_0001.rawspec.0000.h5
blimpy.io.base_reader WARNING  Selection size of 7.50 GB, exceeding our size limit 1.00 GB. Instance created, header loaded, but data not loaded, please try another (t,v) selection.

--- File Info ---
DIMENSION_LABELS :   ['time' 'feed_id' 'frequency']
        az_start :                              0.0
       data_type :                                1
            fch1 :                     8437.625 MHz
            foff :          1.9073486328125e-06 MHz
           ibeam :                               -1
      machine_id :                               20
          nbeams :                                1
           nbits :                               32
          nchans :                         33554432
            nifs :                                1
     rawdatafile : guppi_59196_68468_000762_Unknown_0001.0000.raw
     source_name :                          Unknown
         src_dej :                          0:00:00
         src_raj :                          0:00:00
    telescope_id :                               -1
           tsamp :                         2.097152
   tstart (ISOT) :          2020-12-13T19:01:08.000
    tstart (MJD) :                 59196.7924537037
        za_start :                              0.0

Num ints in file :                               60
      File shape :                (60, 1, 33554432)
--- Selection Info ---
Data selection shape :                (60, 1, 33554432)
Minimum freq (MHz) :                         8437.625
Maximum freq (MHz) :                8501.624998092651
None
Traceback (most recent call last):

  File "/home/elkins/seti_testing/blimpy_testing/stg_bug.py", line 24, in <module>
    import_bl_file(INFILE)

  File "/home/elkins/seti_testing/blimpy_testing/stg_bug.py", line 18, in import_bl_file
    frame = stg.Frame(waterfall=wf)

  File "/home/elkins/.local/lib/python3.8/site-packages/setigen/frame.py", line 120, in __init__
    self.data = waterfall_utils.get_data(self.waterfall)[:, ::-1]

  File "/home/elkins/.local/lib/python3.8/site-packages/setigen/waterfall_utils.py", line 75, in get_data
    return waterfall.data[:, 0, :]

IndexError: too many indices for array: array is 1-dimensional, but 3 were indexed

[texadactyl]

Other observations:

• The data array is 3-dimensional even though the middle dimension is never used. But, both the blimpy io hdf_reader and the fil_reader read_data() functions initialize "heavy" data arrays to [0] i.e. a 1-dimensional array with a single element of 0. Maybe this is the cause of the crash? But, setigen Frame should check to see if the Waterfall.container.load_data = True before going after data.
• Not all of the data will fit in RAM as a practical matter. Blimpy handles that with a threshold and a buffering technique. What should setigen do when a Frame is being instantiated from a Waterfall object that has "heavy" data?