Missing Run2.2i raw files at CC-IN2P3

[jchiang87]

As a follow-up to discussion at DM-31036, I did a census of the Run2.2i raw FITS files at CC-IN2P3 in

/sps/lsst/datasets/desc/DC2/Run2.2i/sim

and compared that list to the Run2.2i raw files that we have at NERSC in

/global/cfs/cdirs/lsst/shared/DC2-prod/Run2.2i/sim

In summary, there are 3185 raw files at NERSC that are not at CC-IN2P3. Conversely, all of the raw files at CC-IN2P3 are also at NERSC. The list of files that are missing at CC-IN2P3 is attached.

[johannct]

So we need to compare with the list of files that were flagged as bad and for which transfer to CC was prevented, right? @heather999 @villarrealas ?

[boutigny]

With @johannct we checked randomly a few visits having missing files and they are all intersecting the footprint boundary. So my best guess is that these sensors were discarded from the processing because they are outside of the footprint. If my memory is correct at some point we systematically checked whether a sensor was in the footprint or outside before simulating it rather than discarding it afterwards.

[jchiang87]

As recorded in https://github.com/LSSTDESC/ImageProcessingPipelines/issues/133 , the Run2.2i y2-wfd transfers from NERSC to CC-IN2P3 were done in 3 stages (the last of the three owing to some missing visits in the infamous DR3 "hole"). Looking at the log file for the first transfer, /sps/lsst/datasets/desc/DC2/Run2.2i/logs/2020-01-11-nersc-to-in2p3.txt, the visits with missing CCDs in the list I posted above were all transferred during the first of those three campaigns. Adding up the number of transferred files in all three logs and comparing to the number of y2-wfd files at NERSC, I recover the deficit of 3185 files. A spot check of the files for visit 457681 show that the three raw files for the noted CCDs are indeed missing from 2020-01-11-nersc-to-in2p3.txt while the other 186 raw files are listed. So, I conclude, that for whatever reason, the missing files were never transferred to CC-IN2P3.

[jchiang87]

Looking at the tracking db in /global/cfs/cdirs/lsst/projecta/lsst/production/DC2_ImSim/Run2.2i/sim/sqltracking_run2.2i.dbMAR302020, and selecting on raft R41 and visit=457681, I see that the three missing files were created after the initial campaign (probably as part of the sims completed later, as noted here):

sqlite> select file_path, raft_id, sensor_id, quality_controlled, created_on from files where visit_id=457681 and raft_id='41'; 
lsst_a_457681_R41_S22_i.fits|41|22|0|2020-01-06T15:49:22.792996
lsst_a_457681_R41_S00_i.fits|41|00|0|2020-01-06T15:49:54.297321
lsst_a_457681_R41_S10_i.fits|41|10|0|2020-01-06T15:49:28.489933
lsst_a_457681_R41_S01_i.fits|41|01|0|2020-01-06T15:49:53.879597
lsst_a_457681_R41_S21_i.fits|41|21|0|2020-01-06T15:49:17.185884
lsst_a_457681_R41_S02_i.fits|41|02|0|2020-01-06T15:49:18.361559
centroid_457681_R41_S00_i.txt.gz|41|00|0|2020-01-06T15:51:35.144340
centroid_457681_R41_S21_i.txt.gz|41|21|0|2020-01-06T15:51:33.917821
centroid_457681_R41_S01_i.txt.gz|41|01|0|2020-01-06T15:51:25.975025
centroid_457681_R41_S22_i.txt.gz|41|22|0|2020-01-06T15:51:34.323624
centroid_457681_R41_S02_i.txt.gz|41|02|0|2020-01-06T15:51:31.411522
centroid_457681_R41_S10_i.txt.gz|41|10|0|2020-01-06T15:51:17.954718
lsst_a_457681_R41_S11_i.fits|41|11|0|2020-01-13T12:26:06.614296
lsst_a_457681_R41_S20_i.fits|41|20|0|2020-01-13T12:26:03.096799
lsst_a_457681_R41_S12_i.fits|41|12|0|2020-01-13T12:26:03.521908
centroid_457681_R41_S20_i.txt.gz|41|20|0|2020-01-13T12:26:09.661075
centroid_457681_R41_S12_i.txt.gz|41|12|0|2020-01-13T12:26:07.622804
centroid_457681_R41_S11_i.txt.gz|41|11|0|2020-01-13T12:26:05.745043

Apparently, these were not included in the subsequent two transfer campaigns since those campaigns seem to have been driven by visit lists and not at the individual file level.

[airnandez]

I checked in my transfer campaign logs and confirm the findings by @jchiang87.

The 3 files not present at CC-IN2P3 from visit 457681 (year 2) were generated after the transfer campaign on 2020-01-11 was executed. That campaign included all the quality-controlled files of completed visits of year 2, up to that point in time.

The next transfer campaign from NERSC to IN2P3 was executed on 2020-01-19 and included quality-controlled files of completed visits of year 1 and year 2 of 1095 specific visits. Visit 457681 was not included in the list of visits to transfer.

The next transfer was executed on 2020-01-31 which involved quality-controlled files of completed visits of year 3 only.

[airnandez]

The missing 3,185 files of year 2 were transferred to CC-IN2P3 on 2021-07-15. See details here.

[jchiang87]

Since this seems to be resolved, I'm closing this issue. If there's anything else that need to be considered for this, we can re-open it.