search

bbrzycki / setigen

Python library for generating and injecting artificial narrow-band signals into radio frequency data
https://setigen.readthedocs.io
MIT License
25 stars 11 forks source link

setigen crashes when large data arrays are required #17

Closed texadactyl closed 3 years ago

texadactyl commented 3 years ago

It is impractical to download or even access every H5 or Filterbank file where trouble is reported. In order to investigate, it is highly desirable to be able to synthesize Filterbank files that resemble the file in question. E.g. a large data array file from observations of Tabby's Star (KIC8462852):

--- File Info ---
      machine_id :                               20
    telescope_id :                                6
       data_type :                                1
            fch1 :           19626.464842353016 MHz
            foff :      -2.7939677238464355e-06 MHz
          nchans :                        134217728
     source_name :                           maserA
         src_raj :                     20:39:29.112
         src_dej :                       11:12:59.4
        az_start :                              0.0
        za_start :                              0.0
           nbits :                               32
   tstart (ISOT) :          2016-12-19T20:44:33.000
    tstart (MJD) :               57741.864270833335
           tsamp :                     18.253611008
            nifs :                                1
Num ints in file :                                5
      File shape :                (5, 1, 134217728)
--- Selection Info ---
Data selection shape :                (5, 1, 134217728)
Minimum freq (MHz) :               19251.464845146984
Maximum freq (MHz) :               19626.464842353016

synthia.py.zip

To reproduce this issue, run the source file inside the attached ZIP file like this inside a terminal window: python3 synthia.py

Linux reports: killed on an octacore with 16 GB RAM (mostly free).

Just change the value of the DIR variable to point to an output directory for saving the Filterbank file. No other parameters needed.

texadactyl commented 3 years ago

When I set fchans = 2^26 = 67108864, I see this result:

blimpy.waterfall INFO     Detecting high frequency resolution data.
blimpy.waterfall INFO     Using 1 n_blobs to write the data.
blimpy.waterfall INFO     Reading 1 of 1
blimpy.io.base_reader INFO     DD shape != blob shape.
Traceback (most recent call last):
  File "synthia.py", line 122, in <module>
    gen_fil(INFILE, n_signals=1)
  File "synthia.py", line 117, in gen_fil
    frame.save_fil(arg_path)
  File "/home/elkins/.local/lib/python3.8/site-packages/setigen/frame.py", line 825, in save_fil
    self.waterfall.write_to_fil(filename)
  File "/usr/local/lib/python3.8/dist-packages/blimpy-2.0.6-py3.8.egg/blimpy/waterfall.py", line 323, in write_to_fil
    write_to_fil(self, filename_out)
  File "/usr/local/lib/python3.8/dist-packages/blimpy-2.0.6-py3.8.egg/blimpy/io/fil_writer.py", line 24, in write_to_fil
    __write_to_fil_heavy(wf, filename_out)
  File "/usr/local/lib/python3.8/dist-packages/blimpy-2.0.6-py3.8.egg/blimpy/io/fil_writer.py", line 57, in __write_to_fil_heavy
    bob = wf.container.read_blob(blob_dim, n_blob=ii)
  File "/usr/local/lib/python3.8/dist-packages/blimpy-2.0.6-py3.8.egg/blimpy/io/fil_reader.py", line 223, in read_blob
    blob = dd.reshape((int(dd.shape[0]/blob_dim[self.freq_axis]),blob_dim[self.beam_axis],blob_dim[self.freq_axis]))
ValueError: cannot reshape array of size 32768 into shape (0,1,76800000)
texadactyl commented 3 years ago

fchans = 2^25 = 33554432 succeeds as expected.