search

njcuk9999 / apero-drs

A PipelinE to Reduce Observations - The DRS for SPIRou (CFHT)
MIT License
12 stars 0 forks source link

0.7.289-stable-test: lbl0.63.006 fails if calibrations are removed from red/ #759

Open larnoldgithub opened 4 months ago

larnoldgithub commented 4 months ago

I have processed the mini dataset 2 in 2 steps: 1/ up to the calibrations 2/ then I removed calibrations from red/ (using apero_reset) and ran the rest of the processing: EXTTELL, EXTRACT etc up to LBL

Everything was fine but the LBL at the end of the processing.

10:04:37.611-  |PROC| ***************************************************************************
10:04:37.618-  |PROC| Errors:
10:04:37.626-  |PROC| ***************************************************************************
10:04:37.634-  |PROC| 
10:04:37.643-  |PROC| 
10:04:37.651-  |PROC| ***************************************************************************
10:04:37.659-@!|PROC| W[40-503-00019]: Error found for ID='502'
10:04:37.666-@!|PROC|   apero_lbl_mask_spirou.py FP --crunfile=mini_run2_EXTRACT_etc.ini --program=LBLMASK_FP[00502] --recipe_kind=lbl-mask-fp --shortname=LBLMASK_FP --parallel=True
10:04:37.674-  |PROC| ***************************************************************************
10:04:37.682-  |PROC| 
 LBL Template Exception [FP_FP] <class 'lbl.core.base_classes.LblException'>:
        Object is classified as a calibration however none ofthe files provided have:
        DPRTYPE=FP_FP
 Please define a template and do not run the template code or add some valid
        files
10:04:37.692-  |PROC| 
10:04:37.699-  |PROC| 
10:04:37.707-  |PROC| ***************************************************************************
10:04:37.715-  |PROC| 
10:04:37.723-  |PROC| ***************************************************************************
10:04:37.731-@!|PROC| W[40-503-00019]: Error found for ID='503'
10:04:37.739-@!|PROC|   apero_lbl_compute_spirou.py FP --iteration 0 --total 15 --crunfile=mini_run2_EXTRACT_etc.ini --program=LBLCOMPUTE_FP[00503] --recipe_kind=lbl-compute-fp --shortname=LBLCOMPUTE_FP --parallel=True
10:04:37.747-  |PROC| ***************************************************************************
10:04:37.754-  |PROC| 
 LBL Compute Exception [FP_FP] <class 'lbl.core.base_classes.LblException'>:
        File /apero-data/test_helu/lbl/masks/FP_neg.fits cannot be found
10:04:37.763-  |PROC| 
10:04:37.770-  |PROC| 
10:04:37.778-  |PROC| ***************************************************************************
10:04:37.786-  |PROC| 
10:04:37.793-  |PROC| ***************************************************************************
10:04:37.801-@!|PROC| W[40-503-00019]: Error found for ID='504'
10:04:37.809-@!|PROC|   apero_lbl_compute_spirou.py FP --iteration 1 --total 15 --crunfile=mini_run2_EXTRACT_etc.ini --program=LBLCOMPUTE_FP[00504] --recipe_kind=lbl-compute-fp --shortname=LBLCOMPUTE_FP --parallel=True
10:04:37.817-  |PROC| ***************************************************************************
10:04:37.825-  |PROC| 
 LBL Compute Exception [FP_FP] <class 'lbl.core.base_classes.LblException'>:
        File /apero-data/test_helu/lbl/masks/FP_neg.fits cannot be found
10:04:37.833-  |PROC| 
10:04:37.841-  |PROC| 
10:04:37.849-  |PROC| ***************************************************************************
10:04:37.857-  |PROC| 
10:04:37.864-  |PROC| ***************************************************************************
10:04:37.872-@!|PROC| W[40-503-00019]: Error found for ID='505'
10:04:37.880-@!|PROC|   apero_lbl_compute_spirou.py FP --iteration 2 --total 15 --crunfile=mini_run2_EXTRACT_etc.ini --program=LBLCOMPUTE_FP[00505] --recipe_kind=lbl-compute-fp --shortname=LBLCOMPUTE_FP --parallel=True
10:04:37.888-  |PROC| ***************************************************************************
10:04:37.896-  |PROC| 
 LBL Compute Exception [FP_FP] <class 'lbl.core.base_classes.LblException'>:
        File /apero-data/test_helu/lbl/masks/FP_neg.fits cannot be found
10:04:37.904-  |PROC| 
10:04:37.912-  |PROC| 
10:04:37.920-  |PROC| ***************************************************************************
10:04:37.928-  |PROC| 
10:04:37.936-  |PROC| ***************************************************************************
10:04:37.944-@!|PROC| W[40-503-00019]: Error found for ID='506'
10:04:37.952-@!|PROC|   apero_lbl_compute_spirou.py FP --iteration 3 --total 15 --crunfile=mini_run2_EXTRACT_etc.ini --program=LBLCOMPUTE_FP[00506] --recipe_kind=lbl-compute-fp --shortname=LBLCOMPUTE_FP --parallel=True
10:04:37.960-  |PROC| ***************************************************************************
10:04:37.968-  |PROC| 
 LBL Compute Exception [FP_FP] <class 'lbl.core.base_classes.LblException'>:
        File /apero-data/test_helu/lbl/masks/FP_neg.fits cannot be found
10:04:37.977-  |PROC| 
10:04:37.984-  |PROC| 
10:04:37.992-  |PROC| ***************************************************************************
10:04:38.000-  |PROC| 
10:04:38.008-  |PROC| ***************************************************************************
10:04:38.016-@!|PROC| W[40-503-00019]: Error found for ID='507'
10:04:38.024-@!|PROC|   apero_lbl_compute_spirou.py FP --iteration 4 --total 15 --crunfile=mini_run2_EXTRACT_etc.ini --program=LBLCOMPUTE_FP[00507] --recipe_kind=lbl-compute-fp --shortname=LBLCOMPUTE_FP --parallel=True
10:04:38.032-  |PROC| ***************************************************************************
10:04:38.039-  |PROC| 
 LBL Compute Exception [FP_FP] <class 'lbl.core.base_classes.LblException'>:
        File /apero-data/test_helu/lbl/masks/FP_neg.fits cannot be found
10:04:38.048-  |PROC| 
10:04:38.056-  |PROC| 
10:04:38.064-  |PROC| ***************************************************************************
10:04:38.072-  |PROC| 
10:04:38.080-  |PROC| ***************************************************************************
10:04:38.088-@!|PROC| W[40-503-00019]: Error found for ID='508'
10:04:38.095-@!|PROC|   apero_lbl_compute_spirou.py FP --iteration 5 --total 15 --crunfile=mini_run2_EXTRACT_etc.ini --program=LBLCOMPUTE_FP[00508] --recipe_kind=lbl-compute-fp --shortname=LBLCOMPUTE_FP --parallel=True
10:04:38.103-  |PROC| ***************************************************************************
10:04:38.111-  |PROC| 
 LBL Compute Exception [FP_FP] <class 'lbl.core.base_classes.LblException'>:
        File /apero-data/test_helu/lbl/masks/FP_neg.fits cannot be found
10:04:38.120-  |PROC| 
10:04:38.127-  |PROC| 
10:04:38.135-  |PROC| ***************************************************************************
10:04:38.143-  |PROC| 
10:04:38.151-  |PROC| ***************************************************************************
10:04:38.159-@!|PROC| W[40-503-00019]: Error found for ID='509'
10:04:38.167-@!|PROC|   apero_lbl_compute_spirou.py FP --iteration 6 --total 15 --crunfile=mini_run2_EXTRACT_etc.ini --program=LBLCOMPUTE_FP[00509] --recipe_kind=lbl-compute-fp --shortname=LBLCOMPUTE_FP --parallel=True
10:04:38.175-  |PROC| ***************************************************************************
10:04:38.182-  |PROC| 
 LBL Compute Exception [FP_FP] <class 'lbl.core.base_classes.LblException'>:
        File /apero-data/test_helu/lbl/masks/FP_neg.fits cannot be found
10:04:38.191-  |PROC| 
10:04:38.199-  |PROC| 
10:04:38.207-  |PROC| ***************************************************************************
10:04:38.214-  |PROC| 
10:04:38.222-  |PROC| ***************************************************************************
10:04:38.230-@!|PROC| W[40-503-00019]: Error found for ID='510'
10:04:38.238-@!|PROC|   apero_lbl_compute_spirou.py FP --iteration 7 --total 15 --crunfile=mini_run2_EXTRACT_etc.ini --program=LBLCOMPUTE_FP[00510] --recipe_kind=lbl-compute-fp --shortname=LBLCOMPUTE_FP --parallel=True
10:04:38.246-  |PROC| ***************************************************************************
10:04:38.254-  |PROC| 
 LBL Compute Exception [FP_FP] <class 'lbl.core.base_classes.LblException'>:
        File /apero-data/test_helu/lbl/masks/FP_neg.fits cannot be found
10:04:38.263-  |PROC| 
10:04:38.271-  |PROC| 
10:04:38.279-  |PROC| ***************************************************************************
10:04:38.286-  |PROC| 
10:04:38.294-  |PROC| ***************************************************************************
10:04:38.302-@!|PROC| W[40-503-00019]: Error found for ID='511'
10:04:38.310-@!|PROC|   apero_lbl_compute_spirou.py FP --iteration 8 --total 15 --crunfile=mini_run2_EXTRACT_etc.ini --program=LBLCOMPUTE_FP[00511] --recipe_kind=lbl-compute-fp --shortname=LBLCOMPUTE_FP --parallel=True
10:04:38.318-  |PROC| ***************************************************************************
10:04:38.326-  |PROC| 
 LBL Compute Exception [FP_FP] <class 'lbl.core.base_classes.LblException'>:
        File /apero-data/test_helu/lbl/masks/FP_neg.fits cannot be found
10:04:38.335-  |PROC| 
10:04:38.343-  |PROC| 
10:04:38.351-  |PROC| ***************************************************************************
10:04:38.359-  |PROC| 
10:04:38.367-  |PROC| ***************************************************************************
10:04:38.375-@!|PROC| W[40-503-00019]: Error found for ID='512'
10:04:38.383-@!|PROC|   apero_lbl_compute_spirou.py FP --iteration 9 --total 15 --crunfile=mini_run2_EXTRACT_etc.ini --program=LBLCOMPUTE_FP[00512] --recipe_kind=lbl-compute-fp --shortname=LBLCOMPUTE_FP --parallel=True
10:04:38.391-  |PROC| ***************************************************************************
10:04:38.399-  |PROC| 
 LBL Compute Exception [FP_FP] <class 'lbl.core.base_classes.LblException'>:
        File /apero-data/test_helu/lbl/masks/FP_neg.fits cannot be found
10:04:38.407-  |PROC| 
10:04:38.415-  |PROC| 
10:04:38.423-  |PROC| ***************************************************************************
10:04:38.431-  |PROC| 
10:04:38.439-  |PROC| ***************************************************************************
10:04:38.447-@!|PROC| W[40-503-00019]: Error found for ID='513'
10:04:38.455-@!|PROC|   apero_lbl_compute_spirou.py FP --iteration 10 --total 15 --crunfile=mini_run2_EXTRACT_etc.ini --program=LBLCOMPUTE_FP[00513] --recipe_kind=lbl-compute-fp --shortname=LBLCOMPUTE_FP --parallel=True
10:04:38.463-  |PROC| ***************************************************************************
10:04:38.471-  |PROC| 
 LBL Compute Exception [FP_FP] <class 'lbl.core.base_classes.LblException'>:
        File /apero-data/test_helu/lbl/masks/FP_neg.fits cannot be found
10:04:38.480-  |PROC| 
10:04:38.488-  |PROC| 
10:04:38.496-  |PROC| ***************************************************************************
10:04:38.504-  |PROC| 
10:04:38.512-  |PROC| ***************************************************************************
10:04:38.520-@!|PROC| W[40-503-00019]: Error found for ID='514'
10:04:38.527-@!|PROC|   apero_lbl_compute_spirou.py FP --iteration 11 --total 15 --crunfile=mini_run2_EXTRACT_etc.ini --program=LBLCOMPUTE_FP[00514] --recipe_kind=lbl-compute-fp --shortname=LBLCOMPUTE_FP --parallel=True
10:04:38.535-  |PROC| ***************************************************************************
10:04:38.543-  |PROC| 
 LBL Compute Exception [FP_FP] <class 'lbl.core.base_classes.LblException'>:
        File /apero-data/test_helu/lbl/masks/FP_neg.fits cannot be found
10:04:38.554-  |PROC| 
10:04:38.561-  |PROC| 
10:04:38.569-  |PROC| ***************************************************************************
10:04:38.577-  |PROC| 
10:04:38.585-  |PROC| ***************************************************************************
10:04:38.593-@!|PROC| W[40-503-00019]: Error found for ID='515'
10:04:38.601-@!|PROC|   apero_lbl_compute_spirou.py FP --iteration 12 --total 15 --crunfile=mini_run2_EXTRACT_etc.ini --program=LBLCOMPUTE_FP[00515] --recipe_kind=lbl-compute-fp --shortname=LBLCOMPUTE_FP --parallel=True
10:04:38.609-  |PROC| ***************************************************************************
10:04:38.617-  |PROC| 
 LBL Compute Exception [FP_FP] <class 'lbl.core.base_classes.LblException'>:
        File /apero-data/test_helu/lbl/masks/FP_neg.fits cannot be found
10:04:38.625-  |PROC| 
10:04:38.633-  |PROC| 
10:04:38.641-  |PROC| ***************************************************************************
10:04:38.649-  |PROC| 
10:04:38.657-  |PROC| ***************************************************************************
10:04:38.665-@!|PROC| W[40-503-00019]: Error found for ID='516'
10:04:38.673-@!|PROC|   apero_lbl_compute_spirou.py FP --iteration 13 --total 15 --crunfile=mini_run2_EXTRACT_etc.ini --program=LBLCOMPUTE_FP[00516] --recipe_kind=lbl-compute-fp --shortname=LBLCOMPUTE_FP --parallel=True
10:04:38.681-  |PROC| ***************************************************************************
10:04:38.689-  |PROC| 
 LBL Compute Exception [FP_FP] <class 'lbl.core.base_classes.LblException'>:
        File /apero-data/test_helu/lbl/masks/FP_neg.fits cannot be found
10:04:38.697-  |PROC| 
10:04:38.705-  |PROC| 
10:04:38.713-  |PROC| ***************************************************************************
10:04:38.721-  |PROC| 
10:04:38.729-  |PROC| ***************************************************************************
10:04:38.737-@!|PROC| W[40-503-00019]: Error found for ID='517'
10:04:38.745-@!|PROC|   apero_lbl_compute_spirou.py FP --iteration 14 --total 15 --crunfile=mini_run2_EXTRACT_etc.ini --program=LBLCOMPUTE_FP[00517] --recipe_kind=lbl-compute-fp --shortname=LBLCOMPUTE_FP --parallel=True
10:04:38.753-  |PROC| ***************************************************************************
10:04:38.761-  |PROC| 
 LBL Compute Exception [FP_FP] <class 'lbl.core.base_classes.LblException'>:
        File /apero-data/test_helu/lbl/masks/FP_neg.fits cannot be found
10:04:38.770-  |PROC| 
10:04:38.778-  |PROC| 
10:04:38.785-  |PROC| ***************************************************************************
10:04:38.793-  |PROC| 
10:04:38.801-  |PROC| ***************************************************************************
10:04:38.809-@!|PROC| W[40-503-00019]: Error found for ID='520'
10:04:38.817-@!|PROC|   apero_lbl_compute_spirou.py GL699 --iteration -1 --total -1 --crunfile=mini_run2_EXTRACT_etc.ini --program=LBLCOMPUTE_SCI[00520] --recipe_kind=lbl-compute-sci --shortname=LBLCOMPUTE_SCI --parallel=True
10:04:38.825-  |PROC| ***************************************************************************
10:04:38.833-  |PROC| 
 LBL Compute Exception [GL699_GL699] <class
        'lbl.core.base_classes.LblException'>: File
        /apero-data/test_helu/lbl/calib/140E95EF27_pp_blaze_AB.fits cannot be found

 LBL Compute Exception [GL699_GL699] <class
        'lbl.core.base_classes.LblException'>: File
        /apero-data/test_helu/lbl/calib/140E95EF27_pp_blaze_AB.fits cannot be found
10:04:38.843-  |PROC| 
10:04:38.851-  |PROC| 
10:04:38.858-  |PROC| ***************************************************************************
10:04:38.866-  |PROC| 
10:04:38.874-  |PROC| ***************************************************************************
10:04:38.882-@!|PROC| W[40-503-00019]: Error found for ID='521'
10:04:38.890-@!|PROC|   apero_lbl_compile_spirou.py GL699 --crunfile=mini_run2_EXTRACT_etc.ini --program=LBLCOMPILE_SCI[00521] --recipe_kind=lbl-compile-sci --shortname=LBLCOMPILE_SCI --parallel=True
10:04:38.898-  |PROC| ***************************************************************************
10:04:38.905-  |PROC| 
 LBL Compile Exception [GL699_GL699] <class
        'lbl.core.base_classes.LblException'>: No lblrv files found in
        /apero-data/test_helu/lbl/lblrv/GL699_GL699. Please run lbl_compute first.

 LBL Compile Exception [GL699_GL699] <class
        'lbl.core.base_classes.LblException'>: No lblrv files found in
        /apero-data/test_helu/lbl/lblrv/GL699_GL699. Please run lbl_compute first.
10:04:38.915-  |PROC| 
10:04:38.923-  |PROC| 
10:04:38.931-  |PROC| ***************************************************************************
10:04:38.939-  |PROC| 
10:04:39.134-**|PROC| ***************************************************************************
10:04:39.142-**|PROC| Recipe apero_processing has been successfully completed
10:04:39.150-**|PROC| ***************************************************************************
larnoldgithub commented 4 months ago

@njcuk9999 Interestingly Apero is complaining about not finding a blaze in lbl/calib/ but I have checked the blaze is present in the main calib/ directory. There is definitely an issue with the last version of lbl and the building of the sym links; Maybe this test where I removed all cals from red/ will help you locating the bug.

njcuk9999 commented 4 months ago

I've no idea whats going on...

Do you have SCIENCE_TARGETS set in mini_run2_EXTRACT_etc.ini maybe that is blocking FP_FP and blaze files from being moved (as they don't match your SCIENCE_TARGET object name).... its just a theory but it might explain it?

larnoldgithub commented 4 months ago

SCIENCE_TARGETS = Gl699 in the mini_run2_EXTRACT_etc.ini

that was working in my previous tests. it fails now only after apero_reset having cleaned up ONLY red/ After that reset, Apero needs to update its databases as usual when files are removed/added but I didn't see any issue.

larnoldgithub commented 4 months ago

I see that when I have rebuilt the container, the git pull get the LBL VERSION: 0.63.007, not 006. Is this ok ?

larnoldgithub commented 4 months ago

test_helu is my profile

apero-data/test_helu/lbl/calib/ only contains sample_wave_grid_spirou.fits. No blazes etc.

njcuk9999 commented 4 months ago

Okay, could you run apero_lbl_ref.py and send me the full log (probably by email) I need to see what apero_lbl_ref_spirou.py is doing.

njcuk9999 commented 4 months ago

So from your log this is the problem:

04:31:50.722-  |GET| ***************************************************************************
04:31:50.729-  |GET| *
04:31:50.730-  |GET| *  SPIROU @PID-00017122051105899020-9M6H (V0.7.289)
04:31:50.730-  |GET| *         py3.9.18  git:v0.7.289-stable-test
04:31:50.730-  |GET| *
04:31:50.736-  |GET| ***************************************************************************
04:31:50.743-  |GET|       █████╗ ██████╗ ███████╗██████╗  ██████╗ 
04:31:50.750-  |GET|      ██╔══██╗██╔══██╗██╔════╝██╔══██╗██╔═══██╗
04:31:50.757-  |GET|      ███████║██████╔╝█████╗  ██████╔╝██║   ██║
04:31:50.763-  |GET|      ██╔══██║██╔═══╝ ██╔══╝  ██╔══██╗██║   ██║
04:31:50.770-  |GET|      ██║  ██║██║     ███████╗██║  ██║╚██████╔╝
04:31:50.777-  |GET|      ╚═╝  ╚═╝╚═╝     ╚══════╝╚═╝  ╚═╝ ╚═════╝ 
04:31:50.783-  |GET| ***************************************************************************
04:31:50.792-**|GET| DRS Setup:
04:31:50.799-**|GET|
04:31:50.799-**|GET|     DRS_ROOT: /opt/apero-drs/apero
04:31:50.799-**|GET|     DRS_DATA_RAW: /apero-data/test_helu/raw
04:31:50.800-**|GET|     DRS_DATA_REDUC: /apero-data/test_helu/red
04:31:50.800-**|GET|     DRS_DATA_WORKING: /apero-data/test_helu/tmp
04:31:50.800-**|GET|     DRS_CALIB_DB: /apero-data/test_helu/calib
04:31:50.800-**|GET|     DRS_TELLU_DB: /apero-data/test_helu/tellu
04:31:50.800-**|GET|     DRS_DATA_ASSETS: /apero-data/test_helu/asset
04:31:50.800-**|GET|     DRS_DATA_OUT: /apero-data/test_helu/out
04:31:50.800-**|GET|     DRS_DATA_MSG: /apero-data/test_helu/log
04:31:50.800-**|GET|     DRS_DATA_RUN: /apero-data/test_helu/run
04:31:50.801-**|GET|     DRS_DATA_PLOT: /apero-data/test_helu/plot
04:31:50.801-**|GET|     DRS_DATA_OTHER: /apero-data/test_helu/other
04:31:50.801-**|GET|     LBL_PATH: /apero-data/test_helu/lbl
04:31:50.801-**|GET|     DRS_CONFIG: /opt/apero-drs/config/test_helu/user_config.ini
04:31:50.801-**|GET|     DRS_CONFIG: /opt/apero-drs/config/test_helu/user_constants.ini
04:31:50.801-**|GET|     DRS_CONFIG: apero.constants.default.default_config
04:31:50.801-**|GET|     DRS_CONFIG: apero.constants.default.default_keywords
04:31:50.801-**|GET|     DRS_CONFIG: [config.instruments.spirou.default_config.py](http://config.instruments.spirou.default_config.py/)
04:31:50.802-**|GET|     DRS_CONFIG: [config.instruments.spirou.default_keywords.py](http://config.instruments.spirou.default_keywords.py/)
04:31:50.802-**|GET|     DRS_CONFIG: [core.instruments.default.default_constants.py](http://core.instruments.default.default_constants.py/)
04:31:50.802-**|GET|     DRS_CONFIG: [core.instruments.spirou.default_constants.py](http://core.instruments.spirou.default_constants.py/)
04:31:50.802-**|GET|     DATABASE: MYSQL
04:31:50.802-**|GET|     DATABASE-calib: spirou@pupus-mysql:calib_test_helu_db
04:31:50.802-**|GET|     DATABASE-tellu: spirou@pupus-mysql:tellu_test_helu_db
04:31:50.802-**|GET|     DATABASE-findex: spirou@pupus-mysql:findex_test_helu_db
04:31:50.802-**|GET|     DATABASE-log: spirou@pupus-mysql:log_test_helu_db
04:31:50.802-**|GET|     DATABASE-astrom: spirou@pupus-mysql:astrom_test_helu_db
04:31:50.803-**|GET|     DATABASE-lang: spirou@pupus-mysql:lang_test_helu_db
04:31:50.803-**|GET|     DATABASE-reject: spirou@pupus-mysql:reject_test_helu_db
04:31:50.803-**|GET|     PRINT_LEVEL: all
04:31:50.803-**|GET|     LOG_LEVEL: all
04:31:50.803-**|GET|     DRS_PLOT: 0
04:31:50.810-  |GET| ***************************************************************************
04:31:50.817-**|GET| Arguments used:
04:31:50.823-**|GET|
04:31:50.823-**|GET|     --OUTPATH: /apero-data/test_helu/lbl/calib
04:31:50.824-**|GET|     --SYMLINKS: True
04:31:50.824-**|GET|     --OBJNAMES: None
04:31:50.824-**|GET|     --OUTTYPES: FF_BLAZE,WAVE_NIGHT
04:31:50.824-**|GET|     --FIBERS: C
04:31:50.824-**|GET|     --SINCE: None
04:31:50.824-**|GET|     --NOSUBDIR: True
04:31:50.824-**|GET|     --TEST: False
04:31:50.831-  |GET| ***************************************************************************
04:31:50.999-  |GET| Loading file index database
04:31:51.010-  |GET| Loading astrometric database
04:31:51.020-  |GET| Loading log database
04:31:51.030-  |GET| Processing KW_OBJNAME=
04:31:51.061-  |GET|     Found no entries
04:31:51.069-  |GET|     Excluded 0 files for failing QC
04:31:51.076-**|GET| ***************************************************************************
04:31:51.084-**|GET| Recipe apero_get has been successfully completed
04:31:51.092-**|GET| ***************************************************************************
04:31:51.173-**|LBLREF[00000]| QUALITY CONTROL SUCCESSFUL - Well Done -
04:31:51.188-**|LBLREF[00000]| ***************************************************************************
04:31:51.196-**|LBLREF[00000]| Recipe apero_lbl_ref_spirou has been successfully completed
04:31:51.204-**|LBLREF[00000]| ***************************************************************************

Inside apero_lbl_ref we use apero_get to symlink the files - the last one copies the FF_BLAZE and WAVE_NIGHT it find no files thus you'll obviously have no blaze or wave in the lbl directory. There is no error given as apero_get just gets what you ask for, if it isn't there it continues to the next query - or in this case just does nothing.

apero_get uses the file index database not the calibration or telluric database so that is the problem... you removed the files from the red directory (and thus the file index database) so as far as apero_get is concerned you have no blaze or wave files.

larnoldgithub commented 4 months ago

But there are reduced o.fits in red/, how apero_get is not finding them ? only the calib files have been removed before the extraction etc. Are you saying that for each object found in red/, apero symlink the required Blaze and Wave from red/night/ and not from calib/ ? Sorry I don't understand the way it works.

Also how would you explain the other problem I have with Apero missing blaze files, while there are in calib/ ? I will run apero_lbl_ref only as above, and look at the log.

njcuk9999 commented 4 months ago

Are you saying that for each object found in red/, apero symlink the required Blaze and Wave from red/night/ and not from calib/ ?

Yes, for LBL only it is apero_get that does this which is called inside apero-lbl-ref_spirou.py

The problem is the following, apero_get only deals with the "file index database". You'll recall there are 7 tables in the database:

The problem is apero_get was designed for the file index database only - it requires the full path to files, some key header values including DRSOBJN and DRSOUTID and deals with things in a "observation night" directory - based on some input from the user it gets files from the file index database and puts them where the user requires (either as symlinks or hard copies). These header keys and full path are not in the calibration or telluric database.

The calibration directory and calibration database are completely separate from the file index database - there are no entries in the file index database for files in the calibration (or telluric) database - therefore if you remove BLAZE and WAVE files from the tmp/red/out/lbl directories (file indexed locations) apero_get thinks those files do not exist anymore anywhere.

I would have to re-write apero_lbl_ref_spirou.py to somehow get files from the calibration database but currently, this isn't easy as we never use the calibration or telluric database this way (we never copy these files just open them when needed) I cannot make lbl do this as lbl is a separate code - so I have to somehow find a way to symlink these files from the calibration database instead of from the tmp/red/out/lbl directories (file indexed locations).

Currently, any file you require in lbl will break if you remove it from the tmp/red/out/lbl directories (file indexed locations) as they are all symlinked.

I will try to add an option to "not symlink lbl files" that will hard copy all files, but this would mean you could never do an apero_reset.py on the lbl directory (as it would remove files that then would not be recopied from the tmp/red/out/lbl directories as you would have deleted them) and would also increase disk usage as you would have copies of all lbl files in the lbl directory.

I guess my best solution would actually be to have an option to only symlink from the telluric/calib directories (which you are not deleting) but this would mean you only get files that pass QC and any files that failed QC would not be copied. (Note that you may want to run lbl on files that fail lbl telluric correction).

Currently, most of these options are a lot of work.