Position from DL3?

[matthew-w-lundy]

Position from DL3 will occasionally fail when there's a folder in your out_dir or if your out_dir isn't only fit files. Perhaps this could either be cleaned up or there could be user recommendations.

    if config["run_selection"]["pos_from_DL3"]:  # get position from DL3 header
        hdul = fits.open(config["io"]["out_dir"]+os.listdir(config["io"]["out_dir"])[0])
        source_pos = SkyCoord(hdul[1].header["RA_OBJ"]*u.deg, hdul[1].header["DEC_OBJ"]*u.deg)
    else:  # get position from ra/dec [deg]
        source_pos = SkyCoord(
            config["run_selection"]["source_ra"],
            config["run_selection"]["source_dec"],
            frame="icrs",
            unit="deg",
        )

[steob92]

This only effects the analysis scripts (gammapy_tools/analysis). Can you check any of the example notebooks show the same issue?

They should get the position from the index files or specified by the user.

[matthew-w-lundy]

The example notebooks (analysis_notebook.ipynb) will show this depending on your config file. You can set it to take the ra and dec specified by the user but if you use "pos_from_DL3" in your config then this error will be thrown depending on whatever is the first file in your os.listdir(config["io"]["out_dir"])

run_selection:

Runs of interests

# Source of interest
source_name : "Crab"
pos_from_DL3: True

[matthew-w-lundy]

---------------------------------------------------------------------------
IsADirectoryError                         Traceback (most recent call last)
Cell In[21], line 1
----> 1 spectral_points, flux, flux_err, spectral_model, cumulative_time, cumulative_sig = make_spectrum_RE(config, plot=False)

File ~/anaconda3/envs/mamba/envs/gammapy1/lib/python3.11/site-packages/gammapy_tools/analysis/data_products.py:72, in make_spectrum_RE(config, plot, return_stacked)
     69 observations = datastore.get_observations()
     71 if config["run_selection"]["pos_from_DL3"]:  # get position from DL3 header
---> 72     hdul = fits.open(config["io"]["out_dir"]+os.listdir(config["io"]["out_dir"])[0])
     73     source_pos = SkyCoord(hdul[1].header["RA_OBJ"]*u.deg, hdul[1].header["DEC_OBJ"]*u.deg)
     74 else:  # get position from ra/dec [deg]

File ~/anaconda3/envs/mamba/envs/gammapy1/lib/python3.11/site-packages/astropy/io/fits/hdu/hdulist.py:214, in fitsopen(name, mode, memmap, save_backup, cache, lazy_load_hdus, ignore_missing_simple, use_fsspec, fsspec_kwargs, **kwargs)
    211 if not name:
    212     raise ValueError(f"Empty filename: {name!r}")
--> 214 return HDUList.fromfile(
    215     name,
    216     mode,
    217     memmap,
    218     save_backup,
    219     cache,
    220     lazy_load_hdus,
    221     ignore_missing_simple,
    222     use_fsspec=use_fsspec,
    223     fsspec_kwargs=fsspec_kwargs,
    224     **kwargs,
    225 )

File ~/anaconda3/envs/mamba/envs/gammapy1/lib/python3.11/site-packages/astropy/io/fits/hdu/hdulist.py:482, in HDUList.fromfile(cls, fileobj, mode, memmap, save_backup, cache, lazy_load_hdus, ignore_missing_simple, **kwargs)
    462 @classmethod
    463 def fromfile(
    464     cls,
   (...)
    472     **kwargs,
    473 ):
    474     """
    475     Creates an `HDUList` instance from a file-like object.
    476 
   (...)
    479     documentation for details of the parameters accepted by this method).
    480     """
--> 482     return cls._readfrom(
    483         fileobj=fileobj,
    484         mode=mode,
    485         memmap=memmap,
    486         save_backup=save_backup,
    487         cache=cache,
    488         ignore_missing_simple=ignore_missing_simple,
    489         lazy_load_hdus=lazy_load_hdus,
    490         **kwargs,
    491     )

File ~/anaconda3/envs/mamba/envs/gammapy1/lib/python3.11/site-packages/astropy/io/fits/hdu/hdulist.py:1170, in HDUList._readfrom(cls, fileobj, data, mode, memmap, cache, lazy_load_hdus, ignore_missing_simple, use_fsspec, fsspec_kwargs, **kwargs)
   1167 if fileobj is not None:
   1168     if not isinstance(fileobj, _File):
   1169         # instantiate a FITS file object (ffo)
-> 1170         fileobj = _File(
   1171             fileobj,
   1172             mode=mode,
   1173             memmap=memmap,
   1174             cache=cache,
   1175             use_fsspec=use_fsspec,
   1176             fsspec_kwargs=fsspec_kwargs,
   1177         )
   1178     # The Astropy mode is determined by the _File initializer if the
   1179     # supplied mode was None
   1180     mode = fileobj.mode

File ~/anaconda3/envs/mamba/envs/gammapy1/lib/python3.11/site-packages/astropy/io/fits/file.py:218, in _File.__init__(self, fileobj, mode, memmap, overwrite, cache, use_fsspec, fsspec_kwargs)
    216     self._open_fileobj(fileobj, mode, overwrite)
    217 elif isinstance(fileobj, (str, bytes)):
--> 218     self._open_filename(fileobj, mode, overwrite)
    219 else:
    220     self._open_filelike(fileobj, mode, overwrite)

File ~/anaconda3/envs/mamba/envs/gammapy1/lib/python3.11/site-packages/astropy/io/fits/file.py:628, in _File._open_filename(self, filename, mode, overwrite)
    625     self._overwrite_existing(overwrite, None, True)
    627 if os.path.exists(self.name):
--> 628     with open(self.name, "rb") as f:
    629         magic = f.read(4)
    630 else:

IsADirectoryError: [Errno 21] Is a directory: '/home/matthew/mydl3/crabtest/results'

[steob92]

These two notebooks should still work and do 90% of what needs to be done:

• prepare_dataset.ipynb
• RBM_analysis.ipynb

[matthew-w-lundy]

I believe gammapy_tools/analysis/validation_analysis.py may also have problems. If this file should not be used in its current state then it should probably be removed.

I have been able to run analysis successfully with all of these notebooks (including analysis_notebook.pynb) but I'm just recording the errors that will pop up anytime any notebook references make_spectrum_RE() with pos_from_DL3. The other notebooks don't contain this helper function so the error does not occur. The RBM_analysis notebook is able to still get outputs that this helper function should generate by rewriting some of the functions that the wrapper should handle for you so I don't think it running is a good benchmark for testing the full functionality of the helper functions included in this package, it's more of a test of just using gammapy.

[samanthalwong]

Yeah validation_analysis.py is really outdated - I'll update it before the next release