search

LSSTDESC / DC2-production

Configuration, production, validation specifications and tools for the DC2 Data Set.
BSD 3-Clause "New" or "Revised" License
11 stars 7 forks source link

Raw files at NERSC with .fits.grid extensions #428

Open jchiang87 opened 2 years ago

jchiang87 commented 2 years ago

While ingesting the data in /global/cfs/cdirs/lsst/shared/DC2-prod/Run2.2i/sim/y4-wfd/ into a gen3 repo at NERSC, the ingest code encountered files with .fits.grid extensions in /global/cfs/cdirs/lsst/shared/DC2-prod/Run2.2i/sim/y4-wfd/00741720/. These appear to be copies of files in the same directory with .fits extensions. If they are copies, they should be deleted (or at least moved elsewhere) since their presence causes problems for the ingest. I'm about to ingest the y5-wfd data and will report here if I see similar issues.

heather999 commented 2 years ago

Just want to reach out to @airnandez for confirmation that the .fits.grid files are indeed just copies of the .fits. I also believe that is the case. If so, I don't see any reason to retain the .fits.grid files and we can remove them.

jchiang87 commented 2 years ago

For the record, I didn't see any similar issues with the ingest of the y5-wfd data, so it appears that the y4-wfd/00741720 folder is the only instance of this.

airnandez commented 2 years ago

I looked at directory y4-wfd/00741720 at CC-IN2P3 and I see 12 files with .fits.grid extension. Their contents is not identical to their .fits counterpart. For instance:

$ cd /sps/lsst/datasets/desc/DC2/Run2.2i/sim/y4-wfd/00741720

$ md5sum lsst_a_741720_R32_S01_u.fits.grid lsst_a_741720_R32_S01_u.fits
bc2c3bbbb71944767a720c838f7c2d53  lsst_a_741720_R32_S01_u.fits.grid
bd0a2fd67126cfd8b55547857d4ccdf1  lsst_a_741720_R32_S01_u.fits

To my knowledge those .grid files were not used at all for DC2. I observe that the .fits files are newer than the ones with .fits.grid extension. If I remember correctly, there were some issues with some simulation jobs of Y4 and Y5 using the grid which required to resubmit failed jobs, which had already produced their outputs and transferred to CC-IN2P3 for permanent storage. The issue was found the simulation was resubmitted either to the grid or executed at NERSC.

I think this issue is related to this Slack conversation.

For the specific issue of ingesting raws into a butler gen3 registry, it is possible to specify *.fits as an argument of the butler ingest-raws ... command to avoid it picking the .grid files.