Add functionality to write all columns from csv files out to Gaia DR3 FITS files

geordie666 commented 9 months ago

This PR has two main purposes:

It adds functionality to, optionally, write out the full set of Gaia columns to the Gaia DR3 FITS files instead of just the subset used for DESI targeting.
It adds a new higher-priority bit for the diffuse interstellar bands BACKUP program (see DESI project 398).

While writing this PR, I discovered a bug that should be included in the DESI DR1 paper:

The string form of Booleans changed for the Gaia csv files moving from Gaia DR2 to Gaia EDR3, necessitating an update to how these strings were parsed. This update was not made, however, meaning that no Boolean columns could be interpreted as True for Gaia EDR3 or Gaia DR3. Therefore, all EDR3 or DR3 Gaia FITS files made by versions of desitarget prior to this PR had Boolean columns set entirely to False. The only column this would have affected for DESI targeting would be the Gaia DUPLICATED_SOURCE column — i.e. effectively no source in DESI targeted using Gaia EDR3 (see section 4.1.4 of the desitarget paper) would have been masked on DUPLICATED_SOURCE.

This PR also adds a boolfix kwarg to gaia_csv_to_fits() allowing this bug to be corrected when generating Gaia EDR3 files. The default behavior of gaia_csv_to_fits() subsequent to this PR is:

To always correct for Gaia DR3 (and we should probably update the relevant Gaia DR3 FITS files at NERSC).
But to not correct the Gaia EDR3 files by default. The reason for this choice is that the Gaia DR3 files were never used for DESI targeting, but the Gaia EDR3 files were.

The bug does not affect Gaia DR2 files, either way.

coveralls commented 9 months ago

coverage: 56.026% (-0.05%) from 56.075% when pulling 6291b7ca833176974643cb8ce7f43c78561ae1c5 on ADM-gaia-full into ab1267e6e37fafc469a3afdcc27cad6e4b3c9f3e on main.

geordie666 commented 9 months ago

I'll leave this PR open until tomorrow evening and then I'll merge and make a new tag, mainly so we can deploy the work from PR #808 as part of daily operations.

moustakas commented 9 months ago

While writing this PR, I discovered a bug that should be included in the DESI DR1 paper:

@geordie666, nice sleuthing. Can I please trouble to add (and self-assign) a ticket here-- https://github.com/desihub/dr1paper/issues

desihub / desitarget

Add functionality to write all columns from csv files out to Gaia DR3 FITS files #809