Error in the TS map notebook

[NicoloParmiggiani]

Hello,

in this notebook https://github.com/cositools/cosipy/blob/main/docs/tutorials/ts_map/Parallel_TS_map_computation_DC2.ipynb

I have an error in the step "Getting the binned Crab data" of example 3:

binning data...
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
Cell In[26], line 5
      3 # "Crab_bkg_galactic_inputs.yaml" can be used for both Crab and background binning since the only useful information in the yaml file is the binning of CDS
      4 analysis = BinnedData("Crab_bkg_galactic_inputs.yaml")
----> 5 analysis.get_binned_data(unbinned_data = "Crab_DC2_3months_unbinned_data.fits.gz", 
      6                          make_binning_plots=False, 
      7                          output_name = "Crab_galactic_CDS_binned", 
      8                          psichi_binning = "galactic")
     10 # After you generate the binned data files, it should be saved to the same directory of this notebook

File ~/.conda/envs/cosipy-dc2/lib/python3.10/site-packages/cosipy/data_io/BinnedData.py:62, in BinnedData.get_binned_data(self, unbinned_data, output_name, make_binning_plots, psichi_binning, event_range)
     60 if unbinned_data:
     61     if self.unbinned_output == 'fits':
---> 62         self.cosi_dataset = self.get_dict_from_fits(unbinned_data)
     63     if self.unbinned_output == 'hdf5':
     64         self.cosi_dataset = self.get_dict_from_hdf5(unbinned_data)

File ~/.conda/envs/cosipy-dc2/lib/python3.10/site-packages/cosipy/data_io/UnBinnedData.py:540, in UnBinnedData.get_dict_from_fits(self, input_fits)
    538 # Fill dictionary from input fits file:
    539 hdu = fits.open(input_fits,memmap=True)
--> 540 cols = hdu[1].columns
    541 data = hdu[1].data
    542 for i in range(0,len(cols)):

File ~/.conda/envs/cosipy-dc2/lib/python3.10/site-packages/astropy/io/fits/hdu/hdulist.py:395, in HDUList.__getitem__(self, key)
    392 except IndexError as e:
    393     # Raise a more helpful IndexError if the file was not fully read.
    394     if self._read_all:
--> 395         raise e
    396     else:
    397         raise IndexError(
    398             "HDU not found, possibly because the index "
    399             "is out of range, or because the file was "
    400             "closed before all HDUs were read"
    401         )

File ~/.conda/envs/cosipy-dc2/lib/python3.10/site-packages/astropy/io/fits/hdu/hdulist.py:389, in HDUList.__getitem__(self, key)
    385 # Originally this used recursion, but hypothetically an HDU with
    386 # a very large number of HDUs could blow the stack, so use a loop
    387 # instead
    388 try:
--> 389     return self._try_while_unread_hdus(
    390         super().__getitem__, self._positive_index_of(key)
    391     )
    392 except IndexError as e:
    393     # Raise a more helpful IndexError if the file was not fully read.
    394     if self._read_all:

File ~/.conda/envs/cosipy-dc2/lib/python3.10/site-packages/astropy/io/fits/hdu/hdulist.py:1258, in HDUList._try_while_unread_hdus(self, func, *args, **kwargs)
   1256 while True:
   1257     try:
-> 1258         return func(*args, **kwargs)
   1259     except Exception:
   1260         if self._read_next_hdu():

IndexError: list index out of range

I am using cosipy v0.0.2a5

[ckarwin]

Hi @NicoloParmiggiani, I just tested this and things ran fine for me. I'm using the same cosipy version (v0.0.2a5). Specifically, I ran the following two cells (I believe this is where you have the error, but let me know otherwise):

%%capture
# 3-month unbinned Crab data ~619.22 MB
fetch_wasabi_file("COSI-SMEX/DC2/Data/Sources/Crab_DC2_3months_unbinned_data.fits.gz")

# Here is the code I used to bin the Crab data if you want to generate it by yourself.
from cosipy import BinnedData
# "Crab_bkg_galactic_inputs.yaml" can be used for both Crab and background binning since the only useful information in the yaml file is the binning of CDS
analysis = BinnedData("Crab_bkg_galactic_inputs.yaml")
analysis.get_binned_data(unbinned_data = "Crab_DC2_3months_unbinned_data.fits.gz", 
                         make_binning_plots=False, 
                         output_name = "Crab_galactic_CDS_binned", 
                         psichi_binning = "galactic")

# After you generate the binned data files, it should be saved to the same directory of this notebook

I think your error might be due to the fits file not being successfully downloaded from wasabi. I've seen similar problems before, where the download gets interrupted for some reason. In this case there is no indication that the download failed (which indeed is problematic). Can you check the file size of Crab_DC2_3months_unbinned_data.fits.gz (should be ~584 MB), and also check that it can be opened ok? Re-downloading the file might resolve the issue. Please let me know if the problem still persists.

[NicoloParmiggiani]

Hi @ckarwin,

your suggestion solved the issue, thanks.

The file was only 300 MB more or less. I removed and downloaded it again.

Nicolò