search

AstroHuntsman / huntsman-drp

Imaging pipeline tools and flow.
MIT License
0 stars 2 forks source link

`DuplicateKeyError` when ingesting files if FITS header changes #174

Closed danjampro closed 3 years ago

danjampro commented 3 years ago 
ges/fields/Norma_Cluster/Norma_Cluster/1815420013090900/20210621T120755/20210621T121738.fits.fz'}.
2021-06-24 23:31:30.520 | DEBUG    | huntsman.drp.collection:find:118 - Find operation returned 0 results.
2021-06-24 23:31:30.521 | DEBUG    | huntsman.drp.collection:insert_one:150 - Inserting document into raw_exposures (RawExposureCollection): {'taiObs': '2021-06-21T12:18:04.069', 'ccdTemp': 0.5, 'expId': 'HUNTS01_1815420013090900_20210621T121738', 'ccd': 1, 'field': 'norma_cluster', 'dateObs': '2021-06-21', 'dataType': 'science', 'filename': '/data/nifi/huntsman_priv/images/fields/Norma_Cluster/Norma_Cluster/1815420013090900/20210621T120755/20210621T121738.fits', 'expTime': 300.0, 'filter': 'r_band', 'visit': 20210621121804069}.
2021-06-24 23:31:30.525 | ERROR    | huntsman.drp.services.base:_wrap_process_func:58 - Exception while processing /data/nifi/huntsman_priv/images/fields/Norma_Cluster/Norma_Cluster/1815420013090900/20210621T120755/20210621T121738.fits: DuplicateKeyError('E11000 duplicate key error collection: huntsman_team.raw_data
index: filename_1 dup key: { filename: "/data/nifi/huntsman_priv/images/fields/Norma_Cluster/Norma_Cluster/1815420013090900/20210621T120755/20210621T121738.fits" }, full error: {\'index\': 0, \'code\': 11000, \'keyPattern\': {\'filename\': 1}, \'keyValue\': {\'filename\': \'/data/nifi/huntsman_priv/images/fields/Norma_Cluster/Norma_Cluster/1815420013090900/20210621T120755/20210621T121738.fits\'}, \'errmsg\': \'E11000 duplicate key error collection: huntsman_team.raw_data index: filename_1 dup key: { filename: "/data/nifi/huntsman_priv/images/fields/Norma_Cluster/Norma_Cluster/1815420013090900/20210621T120755/20210621T121738.fits" }\'}')
2021-06-24 23:31:30.588 | DEBUG    | huntsman.drp.services.ingestor:_process_file:25 - Processing file: /data/nifi/huntsman_priv/images/fields/Frb190711/361d420013090900/20200914T231857/20200914T231900.fits.
2021-06-24 23:31:30.590 | INFO     | huntsman.drp.services.base:_process_results:307 - Finished processing /data/nifi/huntsman_priv/images/fields/Norma_Cluster/Norma_Cluster/1815420013090900/20210621T120755/20210621T121738.fits (fail).

The problem is that the WCS keys were added to the FITS header, so the update call was not matching with the original document and insert was being used instead.