cylammarco / ASPIRED

Automated SpectroPhotometric Image REDuction (ASPIRED)
https://aspired.readthedocs.io/en/latest/
BSD 3-Clause "New" or "Revised" License
30 stars 4 forks source link

Transcribing extensive FITS headers from the raw data #23

Closed rjs3273 closed 3 years ago

rjs3273 commented 4 years ago

Just starting a discussion.

Currently ASPIRED generates FITS with a bare minimal header. The input files of course include extensive data about the observation circumstances, telescope and the instrument. In order to integrate the data into a complete workflow there needs to be some way of propagating all those data through into the headers of the resulting data. As a trivial example, I have just reduced a couple of hundred SPRAT frames through ASPIRED, but now have no easy way of telling which were obtained in the BLUE config and which in the RED.

So my questions is whether there is any plan for this sort of feature in ASPIRED and also whether it makes any sense anyway?

It would be very straight forward to have ASPIRED transcribe everything from the input FITS file, but you could easily end up with invalid FITS headers. If ASPIRED just copies the full header for some arbitrary instrument it has no way of knowing whether or not those values make sense after extraction.

You could have an option to transcribe certain nominated values to the output FITS, but it might end up an enormous list that is difficult to keep up to date.

Is it better to just say that the user needs to write all that into their own wrapper and handle it for themselves. ASPIRED only ever creates a barest minimum header of parameters that it controls itself.

If the latter, then the pipeline workflow is something like

cylammarco commented 4 years ago

I have been struggling to sort out the FITS header for the past year. However, my recent change in how the reduced data are stored have made header handling much easier.

Each reduced spectrum is stored in a separate Spectrum1D object: https://github.com/cylammarco/ASPIRED/blob/160b4d6df7710b8cb8d5dfc3c946c8c10d383944/aspired/spectral_reduction.py#L29 so it is possible to migrate the entire FITS handling to the Spectrum1D class and this will allow much easier user-defined header information handling. Do you think this can resolve all the problem? @rjs3273

cylammarco commented 4 years ago

Some progress is made to improve the flexibility in handling the header, it is now (in the dev-fits branch) possible to do that with some rather long and ugly syntax. More design and development on the API is needed to enable easier header handling.

cylammarco commented 3 years ago

The bug fixing for #27 has brought us further away from being able to add custom header key-value pairs. This is not a complex problem, however, it takes a lot of time effort. The requirement is to separate the FITS creation, recreation, saving, and the conversion into CSV into multiple processes.

cylammarco commented 3 years ago

Solved in d1a041c.