0.7.290 likely a bug with 24BQ08-Sep17 BD8B60B4C5_pp_shapel.fits and BD8B60B4C5a_pp_shapel.fits

[larnoldgithub]

Hi @njcuk9999

I'm reprocessing the Sept run. I have removed from the calib/ and red/(nights) files that I suspect are problematic.

I'm reprocessing Sep17 night. Apero did correctly rebuild the db at launch. It redoes all the calibrations processing. When processing the shape, Apero saves file BD8B60B4C5a_pp_shapel.fits in calib/

a few min later, when Apero process the Flat, the processing failed because it apparently looks for BD8B60B4C5_pp_shapel.fits which doesn't exist in red/ or calib/, only BD8B60B4C5a_pp_shapel.fits is available.

[larnoldgithub]

20:35:13.416- |FF[00173]| Reading shape dxmap file: 20:35:13.416- |FF[00173]| /data/spirou4/apero-data/quicklook290/calib/3444961B5D_pp_shapex.fits 20:35:13.564- |FF[00173]| Reading shape dymap file: 20:35:13.564- |FF[00173]| /data/spirou4/apero-data/quicklook290/calib/3444961B5D_pp_shapey.fits 20:35:13.679- |FF[00173]| Calibration SHAPEL within delta time threshold (0.015331631700973958<7.0 days) 20:35:13.679- |FF[00173]| File: /data/spirou4/apero-data/quicklook290/calib/BD8B60B4C5_pp_shapel.fits 20:35:13.733-!!|FF[00173]| E[01-001-00012]: Read Error: File 'BD8B60B4C5_pp_shapel.fits' could not be 20:35:13.733-!!|FF[00173]| found in directory '/data/spirou4/apero-data/quicklook290/calib'. 20:35:13.733-!!|FF[00173]| function = io.drs_fits.py.readfits() 20:35:14.112-|FF[00173]| 20:35:14.136-@!|FF[00173]| W[40-003-00005]: Recipe apero_flat_spirou has NOT been successfully completed 20:35:14.156-|FF[00173]|

[njcuk9999]

Is this a mixed version? i.e. are there entries from v0.7.288?

If so the database may think the file should be BD8B60B4C5_pp_shapel.fits (which is true for v0.7.288) but apero saves the new file as BD8B60B4C5a_pp_shapel.fits

You'll have to check in your calibration database (not the files on disk) it may be there are some entries there which don't exist on disk any more.

How did you remove files? You may want to try apero_remove and remove BD8B60B4C5_pp_shapel.fits or use apero_database and completely reset the calibration database and get it to build it again (which uses the files on disk to rebuild).

It may be something more subtle but probably best to try these things first.

[larnoldgithub]

yes calib/ had files from the 288 and new files from the 290 for the september run. I have reset the calibration/ and calibDB for the quicklook running the version 290. I'm reprocessing the september run. It seems to work now.

[larnoldgithub]

After the reset of the calibration db, I still put the ref night files from the 288 in the new reset calibration/ directory, but did a apero_database --update for the calibration db. This rebuilt the calibdb with 62 files. I don't have yet a ref night calculted with the 290.

[njcuk9999]

So it may be expecting the 123456a file to exist which v0.7.288 wont have. It may work just changing the filenames to add the "a" or "c" or "d" etc and then resetting the calibration database.

Note there is a difference between --update and --reset, --update may not remove old entries that no longer appear on disk, --reset will clean the whole database.

[larnoldgithub]

I did a apero_reset --only_calib=True first, then rebuilt a minimal calibdb/ with the ref night of the 288 then did a apero_database --update -dbkind=calib . I'm reprocesing the Sept nights, no issues so far, I will verify the results later this morning.

[larnoldgithub]

I had aslo removed the sept night from red/; the index file db has been updated.