search

thepetabyteproject / your

Your Unified Reader
https://thepetabyteproject.github.io/your/
GNU General Public License v3.0
35 stars 25 forks source link

IndexError: index 0 is out of bounds for axis 0 with size 0 #64

Closed devanshkv closed 3 years ago

devanshkv commented 3 years ago

Describe the bug Can't read a bunch of fits files together for some reason?

To Reproduce

your_header.py -f /hyrule/data/users/dagarwal/SGR1935/L_Band/vegas_59261_60229_J1934+2145*fits -v
[02/20/21 11:00:17] DEBUG    2021-02-20 11:00:17,100 - __init__ -your.your - DEBUG - Reading the file(s):                                                                                         your.py:70
                             ['/hyrule/data/users/dagarwal/SGR1935/L_Band/vegas_59261_60229_J1934+2145_0011_0001.fits',
                             '/hyrule/data/users/dagarwal/SGR1935/L_Band/vegas_59261_60229_J1934+2145_0011_0002.fits',
                             '/hyrule/data/users/dagarwal/SGR1935/L_Band/vegas_59261_60229_J1934+2145_0011_0003.fits',
                             '/hyrule/data/users/dagarwal/SGR1935/L_Band/vegas_59261_60229_J1934+2145_0011_0004.fits',
                             '/hyrule/data/users/dagarwal/SGR1935/L_Band/vegas_59261_60229_J1934+2145_0011_0005.fits',
                             '/hyrule/data/users/dagarwal/SGR1935/L_Band/vegas_59261_60229_J1934+2145_0011_0006.fits',
                             '/hyrule/data/users/dagarwal/SGR1935/L_Band/vegas_59261_60229_J1934+2145_0011_0007.fits',
                             '/hyrule/data/users/dagarwal/SGR1935/L_Band/vegas_59261_60229_J1934+2145_0011_0008.fits',
                             '/hyrule/data/users/dagarwal/SGR1935/L_Band/vegas_59261_60229_J1934+2145_0011_0009.fits',
                             '/hyrule/data/users/dagarwal/SGR1935/L_Band/vegas_59261_60229_J1934+2145_0011_0010.fits',
                             '/hyrule/data/users/dagarwal/SGR1935/L_Band/vegas_59261_60229_J1934+2145_0011_0011.fits',
                             '/hyrule/data/users/dagarwal/SGR1935/L_Band/vegas_59261_60229_J1934+2145_0011_0012.fits',
                             '/hyrule/data/users/dagarwal/SGR1935/L_Band/vegas_59261_60229_J1934+2145_0011_0013.fits',
                             '/hyrule/data/users/dagarwal/SGR1935/L_Band/vegas_59261_60229_J1934+2145_0011_0014.fits',
                             '/hyrule/data/users/dagarwal/SGR1935/L_Band/vegas_59261_60229_J1934+2145_0011_0015.fits',
                             '/hyrule/data/users/dagarwal/SGR1935/L_Band/vegas_59261_60229_J1934+2145_0011_0016.fits',
                             '/hyrule/data/users/dagarwal/SGR1935/L_Band/vegas_59261_60229_J1934+2145_0011_0017.fits',
                             '/hyrule/data/users/dagarwal/SGR1935/L_Band/vegas_59261_60229_J1934+2145_0011_0018.fits',
                             '/hyrule/data/users/dagarwal/SGR1935/L_Band/vegas_59261_60229_J1934+2145_0011_0019.fits',
                             '/hyrule/data/users/dagarwal/SGR1935/L_Band/vegas_59261_60229_J1934+2145_0011_0020.fits',
                             '/hyrule/data/users/dagarwal/SGR1935/L_Band/vegas_59261_60229_J1934+2145_0011_0021.fits',
                             '/hyrule/data/users/dagarwal/SGR1935/L_Band/vegas_59261_60229_J1934+2145_0011_0022.fits',
                             '/hyrule/data/users/dagarwal/SGR1935/L_Band/vegas_59261_60229_J1934+2145_0011_0023.fits',
                             '/hyrule/data/users/dagarwal/SGR1935/L_Band/vegas_59261_60229_J1934+2145_0011_0024.fits',
                             '/hyrule/data/users/dagarwal/SGR1935/L_Band/vegas_59261_60229_J1934+2145_0011_0025.fits',
                             '/hyrule/data/users/dagarwal/SGR1935/L_Band/vegas_59261_60229_J1934+2145_0011_0026.fits']
                    WARNING  2021-02-20 11:00:17,136 - __init__ -your.formats.psrfits - WARNING - 'TELESCOP' values don't match for files 0 (G) and 1(GBT)!                                   psrfits.py:477
                    WARNING  2021-02-20 11:00:17,154 - __init__ -your.formats.psrfits - WARNING - 'TELESCOP' values don't match for files 0 (G) and 2(GBT)!                                   psrfits.py:477
                    WARNING  2021-02-20 11:00:17,171 - __init__ -your.formats.psrfits - WARNING - 'TELESCOP' values don't match for files 0 (G) and 3(GBT)!                                   psrfits.py:477
                    WARNING  2021-02-20 11:00:17,189 - __init__ -your.formats.psrfits - WARNING - 'TELESCOP' values don't match for files 0 (G) and 4(GBT)!                                   psrfits.py:477
                    WARNING  2021-02-20 11:00:17,206 - __init__ -your.formats.psrfits - WARNING - 'TELESCOP' values don't match for files 0 (G) and 5(GBT)!                                   psrfits.py:477
                    WARNING  2021-02-20 11:00:17,223 - __init__ -your.formats.psrfits - WARNING - 'TELESCOP' values don't match for files 0 (G) and 6(GBT)!                                   psrfits.py:477
                    WARNING  2021-02-20 11:00:17,241 - __init__ -your.formats.psrfits - WARNING - 'TELESCOP' values don't match for files 0 (G) and 7(GBT)!                                   psrfits.py:477
                    WARNING  2021-02-20 11:00:17,258 - __init__ -your.formats.psrfits - WARNING - 'TELESCOP' values don't match for files 0 (G) and 8(GBT)!                                   psrfits.py:477
                    WARNING  2021-02-20 11:00:17,275 - __init__ -your.formats.psrfits - WARNING - 'TELESCOP' values don't match for files 0 (G) and 9(GBT)!                                   psrfits.py:477
                    WARNING  2021-02-20 11:00:17,292 - __init__ -your.formats.psrfits - WARNING - 'TELESCOP' values don't match for files 0 (G) and 10(GBT)!                                  psrfits.py:477
                    WARNING  2021-02-20 11:00:17,309 - __init__ -your.formats.psrfits - WARNING - 'TELESCOP' values don't match for files 0 (G) and 11(GBT)!                                  psrfits.py:477
                    WARNING  2021-02-20 11:00:17,326 - __init__ -your.formats.psrfits - WARNING - 'TELESCOP' values don't match for files 0 (G) and 12(GBT)!                                  psrfits.py:477
                    WARNING  2021-02-20 11:00:17,343 - __init__ -your.formats.psrfits - WARNING - 'TELESCOP' values don't match for files 0 (G) and 13(GBT)!                                  psrfits.py:477
                    WARNING  2021-02-20 11:00:17,360 - __init__ -your.formats.psrfits - WARNING - 'TELESCOP' values don't match for files 0 (G) and 14(GBT)!                                  psrfits.py:477
                    WARNING  2021-02-20 11:00:17,377 - __init__ -your.formats.psrfits - WARNING - 'TELESCOP' values don't match for files 0 (G) and 15(GBT)!                                  psrfits.py:477
                    WARNING  2021-02-20 11:00:17,394 - __init__ -your.formats.psrfits - WARNING - 'TELESCOP' values don't match for files 0 (G) and 16(GBT)!                                  psrfits.py:477
                    WARNING  2021-02-20 11:00:17,411 - __init__ -your.formats.psrfits - WARNING - 'TELESCOP' values don't match for files 0 (G) and 17(GBT)!                                  psrfits.py:477
                    WARNING  2021-02-20 11:00:17,428 - __init__ -your.formats.psrfits - WARNING - 'TELESCOP' values don't match for files 0 (G) and 18(GBT)!                                  psrfits.py:477
                    WARNING  2021-02-20 11:00:17,445 - __init__ -your.formats.psrfits - WARNING - 'TELESCOP' values don't match for files 0 (G) and 19(GBT)!                                  psrfits.py:477
                    WARNING  2021-02-20 11:00:17,462 - __init__ -your.formats.psrfits - WARNING - 'TELESCOP' values don't match for files 0 (G) and 20(GBT)!                                  psrfits.py:477
                    WARNING  2021-02-20 11:00:17,479 - __init__ -your.formats.psrfits - WARNING - 'TELESCOP' values don't match for files 0 (G) and 21(GBT)!                                  psrfits.py:477
                    WARNING  2021-02-20 11:00:17,495 - __init__ -your.formats.psrfits - WARNING - 'TELESCOP' values don't match for files 0 (G) and 22(GBT)!                                  psrfits.py:477
                    WARNING  2021-02-20 11:00:17,512 - __init__ -your.formats.psrfits - WARNING - 'TELESCOP' values don't match for files 0 (G) and 23(GBT)!                                  psrfits.py:477
                    WARNING  2021-02-20 11:00:17,529 - __init__ -your.formats.psrfits - WARNING - 'TELESCOP' values don't match for files 0 (G) and 24(GBT)!                                  psrfits.py:477
                    WARNING  2021-02-20 11:00:17,546 - __init__ -your.formats.psrfits - WARNING - 'TELESCOP' values don't match for files 0 (G) and 25(GBT)!                                  psrfits.py:477
Traceback (most recent call last):
  File "/home/dagarwal/soft/conda/envs/your_env/bin/your_header.py", line 104, in <module>
    read_header(values.files, values.no_table)
  File "/home/dagarwal/soft/conda/envs/your_env/bin/your_header.py", line 55, in read_header
    y = Your(f)
  File "/home/dagarwal/soft/conda/envs/your_env/lib/python3.8/site-packages/your/your.py", line 72, in __init__
    self.formatclass.__init__(self, self.your_file)
  File "/home/dagarwal/soft/conda/envs/your_env/lib/python3.8/site-packages/your/formats/psrfits.py", line 112, in __init__
    self.specinfo = SpectraInfo(psrfitslist)
  File "/home/dagarwal/soft/conda/envs/your_env/lib/python3.8/site-packages/your/formats/psrfits.py", line 557, in __init__
    first_subint = subint_hdu.data[0]
  File "/home/dagarwal/.local/lib/python3.8/site-packages/astropy/io/fits/fitsrec.py", line 512, in __getitem__
    out = self.view(np.recarray)[key]
  File "/home/dagarwal/soft/conda/envs/your_env/lib/python3.8/site-packages/numpy/core/records.py", line 519, in __getitem__
    obj = super(recarray, self).__getitem__(indx)
IndexError: index 0 is out of bounds for axis 0 with size 0
devanshkv commented 3 years ago

The last file is empty, which's causing this issue.

KshitijAggarwal commented 3 years ago

Yes, the last file is causing this. It has 0 subints. Just running your_header on it gives the following. Need to find a better way to catch this error. 

your_header.py -f /hyrule/data/users/dagarwal/SGR1935/L_Band/vegas_59261_60229_J1934+2145_0011_0026.fits -v

[02/22/21 11:52:41] DEBUG    2021-02-22 11:52:41,419 - __init__ -your.your - DEBUG - Reading the file(s): ['/hyrule/data/users/dagarwal/SGR1935/L_Band/vegas_59261_60229_J1934+2145_0011_0026.fits']                                                                your.py:70
WARNING: non-ASCII characters are present in the FITS file header and have been replaced by "?" characters [astropy.io.fits.util]
                    WARNING  2021-02-22 11:52:41,450 - _showwarning -astropy - WARNING - non-ASCII characters are present in the FITS file header and have been replaced by "?" characters                                                                       logger.py:204
WARNING: non-ASCII characters are present in the FITS file header and have been replaced by "?" characters [astropy.io.fits.util]
[02/22/21 11:52:42] WARNING  2021-02-22 11:52:42,333 - _showwarning -astropy - WARNING - non-ASCII characters are present in the FITS file header and have been replaced by "?" characters                                                                       logger.py:204
Traceback (most recent call last):
  File "/home/kshitij/anaconda3/envs/grbfrb/lib/python3.6/site-packages/astropy/io/fits/hdu/hdulist.py", line 1111, in _try_while_unread_hdus
    return func(*args, **kwargs)
IndexError: list index out of range

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/kshitij/anaconda3/envs/grbfrb/lib/python3.6/site-packages/astropy/io/fits/hdu/base.py", line 406, in _readfrom_internal
    header_str, header = _BasicHeader.fromfile(data)
  File "/home/kshitij/anaconda3/envs/grbfrb/lib/python3.6/site-packages/astropy/io/fits/header.py", line 2077, in fromfile
    header_str, cards = parse_header(fileobj)
  File "astropy/io/fits/_utils.pyx", line 38, in astropy.io.fits._utils.parse_header
UnicodeDecodeError: 'ascii' codec can't decode byte 0xd5 in position 1: ordinal not in range(128)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/kshitij/anaconda3/envs/grbfrb/bin/your_header.py", line 4, in <module>
    __import__('pkg_resources').run_script('your==0.6.3', 'your_header.py')
  File "/home/kshitij/anaconda3/envs/grbfrb/lib/python3.6/site-packages/pkg_resources/__init__.py", line 650, in run_script
    self.require(requires)[0].run_script(script_name, ns)
  File "/home/kshitij/anaconda3/envs/grbfrb/lib/python3.6/site-packages/pkg_resources/__init__.py", line 1446, in run_script
    exec(code, namespace, namespace)
  File "/home/kshitij/anaconda3/envs/grbfrb/lib/python3.6/site-packages/your-0.6.3-py3.6.egg/EGG-INFO/scripts/your_header.py", line 104, in <module>
    read_header(values.files, values.no_table)
  File "/home/kshitij/anaconda3/envs/grbfrb/lib/python3.6/site-packages/your-0.6.3-py3.6.egg/EGG-INFO/scripts/your_header.py", line 55, in read_header
    y = Your(f)
  File "/home/kshitij/anaconda3/envs/grbfrb/lib/python3.6/site-packages/your-0.6.3-py3.6.egg/your/your.py", line 72, in __init__
    self.formatclass.__init__(self, self.your_file)
  File "/home/kshitij/anaconda3/envs/grbfrb/lib/python3.6/site-packages/your-0.6.3-py3.6.egg/your/formats/psrfits.py", line 112, in __init__
    self.specinfo = SpectraInfo(psrfitslist)
  File "/home/kshitij/anaconda3/envs/grbfrb/lib/python3.6/site-packages/your-0.6.3-py3.6.egg/your/formats/psrfits.py", line 462, in __init__
    self.hdu_names = [hdu.name for hdu in hdus]
  File "/home/kshitij/anaconda3/envs/grbfrb/lib/python3.6/site-packages/your-0.6.3-py3.6.egg/your/formats/psrfits.py", line 462, in <listcomp>
    self.hdu_names = [hdu.name for hdu in hdus]
  File "/home/kshitij/anaconda3/envs/grbfrb/lib/python3.6/site-packages/astropy/io/fits/hdu/hdulist.py", line 261, in __iter__
    yield self[idx]
  File "/home/kshitij/anaconda3/envs/grbfrb/lib/python3.6/site-packages/astropy/io/fits/hdu/hdulist.py", line 314, in __getitem__
    self._positive_index_of(key))
  File "/home/kshitij/anaconda3/envs/grbfrb/lib/python3.6/site-packages/astropy/io/fits/hdu/hdulist.py", line 1113, in _try_while_unread_hdus
    if self._read_next_hdu():
  File "/home/kshitij/anaconda3/envs/grbfrb/lib/python3.6/site-packages/astropy/io/fits/hdu/hdulist.py", line 1154, in _read_next_hdu
    hdu = _BaseHDU.readfrom(fileobj, **kwargs)
  File "/home/kshitij/anaconda3/envs/grbfrb/lib/python3.6/site-packages/astropy/io/fits/hdu/base.py", line 327, in readfrom
    **kwargs)
  File "/home/kshitij/anaconda3/envs/grbfrb/lib/python3.6/site-packages/astropy/io/fits/hdu/base.py", line 414, in _readfrom_internal
    endcard=not ignore_missing_end)
  File "/home/kshitij/anaconda3/envs/grbfrb/lib/python3.6/site-packages/astropy/io/fits/header.py", line 514, in fromfile
    padding)[1]
  File "/home/kshitij/anaconda3/envs/grbfrb/lib/python3.6/site-packages/astropy/io/fits/header.py", line 596, in _from_blocks
    raise OSError('Header missing END card.')
OSError: Header missing END card.
KshitijAggarwal commented 3 years ago

I have a possible fix:

adding ignore_missing_end=True to the pyfits.open statements and an hdus.verify() after this: https://github.com/thepetabyteproject/your/blob/a678c8da7894d1ee9bfcab9250920a66772007c6/your/formats/psrfits.py#L459 and an if condition to catch is self.num_subint[ii] == 0 https://github.com/thepetabyteproject/your/blob/a678c8da7894d1ee9bfcab9250920a66772007c6/your/formats/psrfits.py#L539 We would have to change the number of files, fitslist, etc if the last file is found to have 0 subints. What if the user is reading just that file with 0 subints? and what if some file in between has 0 subints?

Also, we use pyfits.open multiple times in psrfits.py, so before deploying this change we need to think of better ways to test if this breaks something else.

KshitijAggarwal commented 3 years ago

First attempt at fixing this: 42e10ff511d63e96ed46b5857c0cf9211bb99026

KshitijAggarwal commented 3 years ago

Added tests for this too: 45d212e8bb9425375647b41edf4d78d54dc92a9d I think this issue is resolved now.