Open larnoldgithub opened 7 months ago
You are correct that setting the DEBUG_XXX in the user_constants.ini should stop the creation of all DEBUG files.
I will look into your two bugs, but I confirm
Okay so the wave solution DEBUG file is just that we update the wave solution in the headers. I'll just make the DEBUG_E2DSLL file optional and deal with not finding it better.
Hi @njcuk9999 is it possible to just update a couple files to fix that bug in the .288 instead of making a fresh install with the .289-live ? I don't want to break my current .288 for the moement, although I'll will be ready to upgrade to .289 in a few weeks; but for the moment I want to stick with the .288 AND get rid of the DEBUG option issue. Thanks for your help. Copying @cusher
okay this is in a special branch called v0.7.288-luc-fix
honestly though now you've deleted all debug files it shouldn't be a big problem having a few created!
I tried to process 23BQ10+Q12+Q14 (48nights), october to december nights without the DEBUG files to save disk space (up to 10Tb for a full SPI reduction). I set DEBUG_E2DSLL_FILE = False in my user_constant.ini and it failed.
I did then a test for one single night 23BQ10-Oct26 trying to disable DEBUG* files to save disk space (up to 10Tb for a full SPI reduction)
1/ I first set DEBUG_E2DSLL_FILE = False in my user_constant.ini but wave_night ends with an error looking for DEBUG_4BECB7874B_pp_e2dsll_AB.fits (HC_HC) to update its wave solution.
2/ I then realized that the file /apero/apero-drs/apero/core/instruments/spirou/default_constants.py also has a setting DEBUG_E2DSLL_FILE.value = True and I turned it to False (line 2813).
With DEBUG_E2DSLL_FILE set to False in both user_constant.ini and default_constants.py above, 1/ FF: I see that the flat extraction still produces DEBUGe2dsll files. 2/ WAVE: wave_night fails, still looking for DEBUG_4BECB7874B_pp_e2dsll_AB.fits (HC_HC) which is missing. It looks like a bug.
For this test, I had first deleted the night folder to be sure APERO would start from scratch for that night - APERO rebuilt its DB ('File no longer on disk removing from file index database: etc.'). My file runnewSPInights.ini had the calibrations RUN all set to True and SKIP_ to False. But wave_night failed.
So I had to come back to DEBUG_E2DSLL_FILE =True in user_constant.ini and DEBUG_E2DSLL_FILE.value = True in defaultconstants.py. Again, to get a redcution done, I had to delete the existing night folder, even with all calibration RUN set to True and SKIP_ to False; I don't understand that, otherwise I had APERO looking for missing DEBUG*.
To get the night properly reduced, the only solution I found was to delete the 23BQ10-12-14 nights in /red that failed earlier. I proceeded by step of 10 nights max to avoid MySQL to crash with ++thousand of files to remove from its tables at once. I was running a runnothing.ini with all RUN to False , so APERO was doing nothing but updating the red database, and it worked.
It would be usefull to have a simple way to disable DEBUG files : rm -v red/night/DEBUG* lead to pb with the DB because the number of removed files is too large. More generally a robust way to remove large amount of files (like deleting tens of nights in red/ for example) without compromising the DB update would be welcome. Why not saving the list of files somewhere and get apero and mysql proceeding by chunks of 1000 files for example. No worries if it's take some time.