njcuk9999
commented
6 months ago You are correct that setting the DEBUG_XXX in the user_constants.ini should stop the creation of all DEBUG files.
I will look into your two bugs, but I confirm
- the flat recipe should not produce DEBUG_e2dsll files if DEBUG_e2dsll is set to False
- the wave solution shouldn't be using debug files - so it may be we need another calibration for the wave solution to use - those files should not start with DEBUG_ and should also go into the calibration database
larnoldgithub
I tried to process 23BQ10+Q12+Q14 (48nights), october to december nights without the DEBUG files to save disk space (up to 10Tb for a full SPI reduction). I set DEBUG_E2DSLL_FILE = False in my user_constant.ini and it failed.
I did then a test for one single night 23BQ10-Oct26 trying to disable DEBUG* files to save disk space (up to 10Tb for a full SPI reduction)
1/ I first set DEBUG_E2DSLL_FILE = False in my user_constant.ini but wave_night ends with an error looking for DEBUG_4BECB7874B_pp_e2dsll_AB.fits (HC_HC) to update its wave solution.
2/ I then realized that the file /apero/apero-drs/apero/core/instruments/spirou/default_constants.py also has a setting DEBUG_E2DSLL_FILE.value = True and I turned it to False (line 2813).
With DEBUG_E2DSLL_FILE set to False in both user_constant.ini and default_constants.py above, 1/ FF: I see that the flat extraction still produces DEBUGe2dsll files. 2/ WAVE: wave_night fails, still looking for DEBUG_4BECB7874B_pp_e2dsll_AB.fits (HC_HC) which is missing. It looks like a bug.
For this test, I had first deleted the night folder to be sure APERO would start from scratch for that night - APERO rebuilt its DB ('File no longer on disk removing from file index database: etc.'). My file runnewSPInights.ini had the calibrations RUN all set to True and SKIP_ to False. But wave_night failed.
So I had to come back to DEBUG_E2DSLL_FILE =True in user_constant.ini and DEBUG_E2DSLL_FILE.value = True in defaultconstants.py. Again, to get a redcution done, I had to delete the existing night folder, even with all calibration RUN set to True and SKIP_ to False; I don't understand that, otherwise I had APERO looking for missing DEBUG*.
To get the night properly reduced, the only solution I found was to delete the 23BQ10-12-14 nights in /red that failed earlier. I proceeded by step of 10 nights max to avoid MySQL to crash with ++thousand of files to remove from its tables at once. I was running a runnothing.ini with all RUN to False , so APERO was doing nothing but updating the red database, and it worked.
It would be usefull to have a simple way to disable DEBUG files : rm -v red/night/DEBUG* lead to pb with the DB because the number of removed files is too large. More generally a robust way to remove large amount of files (like deleting tens of nights in red/ for example) without compromising the DB update would be welcome. Why not saving the list of files somewhere and get apero and mysql proceeding by chunks of 1000 files for example. No worries if it's take some time.