Closed alexandrineLH closed 8 months ago
Okay so actually some of these "non weird" ones may be "weird".
I updated the astrometric database today and removed duplicates (I'm tagging @CharlesCadieux as he has some interest in this)
These are a list of files that had duplicates. I did one of three things:
BDP02_1729 <-- Not a duplicate but had a bad alias (empty string)
DX_CNC <-- Duplicate of GJ1111 (kept DX_CNC)
EE_LEO <-- Duplicate of GJ402 (kept EE_LEO)
GJ1111 <-- Duplicate of GJ1111 (kept DX_CNC)
GJ2 <-- Not a duplicate but had a bad alias (empty string)
GJ402 <-- Duplicate of EE_LEO (kept EE_LEO)
GJ411 <-- Duplicate of GL411 (kept GJ411)
GJ4247 <-- Duplicate of V374PEG (kept GJ4227)
GJ555 <-- Duplicate of HN_LIB (kept GJ555)
GJ617B <-- Duplicate of GL617B (kept GJ617B)
GJ628 <-- Duplicate of V2306_OPH (kept GJ628)
GJ752A <-- Duplicate of GL752A (kept GH752A)
GJ87 <-- Duplicate of GL87 (kept GJ87)
GJ876A <-- Duplicate of GL876 (kept GJ876A)
GJ905 <-- Duplicate of GL905 (kept GJ905)
GJ9520 <-- Duplicate of OTSER (kept GJ9520)
GL15A <-- Duplicate of GX_AND (kept GL15A)
GL15B <-- Duplicate of GQ_AND (kept GL15B)
GL176 <-- Duplicate of HD_285968 (kept GL176)
GL411 <-- Duplicate of GJ411 (kept GJ411)
GL617B <-- Duplicate of GL617B (kept GJ617B)
GL752A <-- Duplicate of GL752A (kept GH752A)
GL83_1 <-- Duplicate of TZ_ARI (kept TZ_ARI)
GL87 <-- Duplicate of GJ87 (kept GJ87)
GL876 <-- Duplicate of GJ876A (kept GJ876A)
GL905 <-- Duplicate of GJ905 (kept GJ905)
GQ_AND <-- Duplicate of GL15B (kept GL15B)
GX_AND<-- Duplicate of GX_AND (kept GL15A)
HD18978 <-- Duplicate of TAU3_ERI (kept HD18978)
HD_285968 <-- Duplicate of GL176 (kept GL176)
HN_LIB <-- Duplicate of GJ555 (kept GJ555)
OTSER <-- Duplicate of GJ9520 (kept GJ9520)
SCR_J1546M5534 <-- ignored as duplicate
SCR_J1845M6357 <-- ignored as duplicate
TAU3_ERI <-- Duplicate of HD18978 (kept HD18978)
TOIM837 <-- Not a duplicate but had a bad alias (empty string)
TZ_ARI <-- Duplicate of GL83_1 (kept TZ_ARI)
V2306_OPH <-- Duplicate of GJ628 (kept GJ628)
V374PEG <-- Duplicate of GJ4227 (kept GJ4227)
Could I be a pain and get you (@alexandrineLH) to check which ones of these are in your errors.
Also BETAPICMB
is in the reject list (it is a planet and we don't actually want to do LBL on a planet) so could you (@alexandrineLH ) also check any others (that aren't in the list above) against the reject list (https://docs.google.com/spreadsheets/d/1dOogfEwC7wAagjVFdouB1Y1JdF9Eva4uDW6CTZ8x2FM/edit?usp=sharing)
If @CharlesCadieux could just go over the merges I did (first list) I'd appreciate a second pair of eyes that these are correct.
If you want to run the code I used its in apero-utils/general/apero_astrometrics/merge_duplicate_rows.py
I can send you the bad_object yaml. If you dont' want to run the code here is my log as well so you can see the comparison for each object.
Here are all the objects for the three types of errors we get. None of them are on the duplicate list and a few are on the reject list (name preceded by --).
Missing Teff
--BETAPICMB
--BETAPICMCALIB
GJ199A
GJ9085
HD37258B
--LUHMAN_16B
NIGHT_SKYMSKY_M_HA
--NO_NAME
SKY
UCAC4_136M000283
No lblrv files BDP07_2692 GJ157B GJ199B HD1 HD98736B --LUHMAN_16 ROSS_477 TIC382602147 TOI4201 TOI551 TYC_8638M366M1
Zero-size array GJ1164A GJ157A GJ166A GJ570A GL666A GL666B GL783_2A GL797A HD115404 HD117579 HD120066 HD154363 HD16141B
Ah I think I was confusing BDP07_2692
with BDP02_1729
definitely very different names!
Thanks for this I'm kind of thinging now the no lblrv files are due to SNR stuff but the Zero-size array needs a better error message.
I got the same errors for the reductions today (2024-02-29 for HA and 2024-02-27 to 29 for HE).
-- on the reject list on the duplicate list
Missing Teff —— 2024-02-29 HA GJ527B —— 2024-02-27 to 29 HE GJ402 _GL831 --MOON NIGHT_SKYMSKY_HE --SKY_SKY_04_10 SKYMSKY NIGHT_SKYMSKY_M_HE SKY _TAU3ERI UCAC4_136M000283 TOIM837
No lblrv file I feel like this is all the targets... —— 2024-02-27 to 29 HE BET_PIC DENIS_J104814_6M395606 GJ3693 GJ328 GJ3622 GJ3252 GL317 HD9626 LP885M35 LP888M18 LP_500M35 LP_851M346 --LUHMAN_16 _SCRJ1546M5534 SIPS_J1259M4336 TIC304392811 TIC318801864 TIC382602147 TOI210 TOI2952 TOI2341 TOI3397 TOI1080 TOI2781 TOI3463 TOI3884 TOI4201 TOI4339 TOI4616 TOI4817 TOI5511 TOI552 TOI5007 TOI4863 TOI5295 TOI551 TOI5524 TOI736 TOIM2455 UPM_J1055M7356 UCAC3_152M281185 WASP17 WASP52
Zero-size array —— 2024-02-27 to 29 HE GL297_2A HD16141 HD50281 HD50281B
Two targets had a cannot load template fits file error —— 2024-02-27 to 29 HE TOI210 TOI2952 The log for this error:
21:40:18.662-**|PROC| ***************************************************************************
21:40:18.669-**|PROC|I[40-503-00018]: * GROUP 150/684 (MKTELLU1)
21:40:18.677-**|PROC| ***************************************************************************
21:40:18.685-**|PROC|
21:40:18.897-**|PROC|ID00745|C01/05| apero_mk_tellu_nirps_ha.py 2023-01-19 NIRPS_2023-01-20T09_00_04_634_pp_e2dsff_A.fits --crunfile=science_run_custom.ini --program=MKTELLU1[00745] --recipe_kind=tellu-hotstar --shortname=MKTELLU1 --parallel=True
21:40:19.085-**|PROC|ID00746|C02/05| apero_mk_tellu_nirps_ha.py 2023-01-19 NIRPS_2023-01-20T08_59_25_620_pp_e2dsff_A.fits --crunfile=science_run_custom.ini --program=MKTELLU1[00746] --recipe_kind=tellu-hotstar --shortname=MKTELLU1 --parallel=True
21:40:19.324-**|PROC|ID00747|C03/05| apero_mk_tellu_nirps_ha.py 2023-01-19 NIRPS_2023-01-20T08_58_46_606_pp_e2dsff_A.fits --crunfile=science_run_custom.ini --program=MKTELLU1[00747] --recipe_kind=tellu-hotstar --shortname=MKTELLU1 --parallel=True
21:40:19.610-**|PROC|ID00748|C04/05| apero_mk_tellu_nirps_ha.py 2023-01-19 NIRPS_2023-01-20T08_58_07_592_pp_e2dsff_A.fits --crunfile=science_run_custom.ini --program=MKTELLU1[00748] --recipe_kind=tellu-hotstar --shortname=MKTELLU1 --parallel=True
21:40:19.803-**|PROC|ID00749|C05/05| apero_mk_tellu_nirps_ha.py 2023-01-19 NIRPS_2023-01-20T02_54_27_771_pp_e2dsff_A.fits --crunfile=science_run_custom.ini --program=MKTELLU1[00749] --recipe_kind=tellu-hotstar --shortname=MKTELLU1 --parallel=True
(base) [nirps-client@rali nirps]$ cat /cosmos99/nirps/apero-data/nirps_he_online/msg/processing/APEROG-PID-00017093138008172900-VNOD_apero_processing_group/other/APEROL-PID-00017093203195601100-56XF_apero_lbl_compute_nirps_he.log
19:12:00.121- |LBLCOMPUTE_SCI[00957]| ***************************************************************************
19:12:00.130- |LBLCOMPUTE_SCI[00957]| *
19:12:00.131- |LBLCOMPUTE_SCI[00957]| * NIRPS_HE @PID-00017093203195601100-56XF (V0.7.289)
19:12:00.132- |LBLCOMPUTE_SCI[00957]| * py3.9.16 git:v0.7.289-live
19:12:00.132- |LBLCOMPUTE_SCI[00957]| *
19:12:00.141- |LBLCOMPUTE_SCI[00957]| * Group: APEROG-PID-00017093138008172900-VNOD_apero_processing_group
19:12:00.149- |LBLCOMPUTE_SCI[00957]| ***************************************************************************
19:12:00.158-**|LBLCOMPUTE_SCI[00957]|I[40-001-00006]: DRS Setup:
19:12:00.166-**|LBLCOMPUTE_SCI[00957]|I[
19:12:00.167-**|LBLCOMPUTE_SCI[00957]| DRS_ROOT: /cosmos99/nirps/git-bin/apero-drs-online/apero]:
19:12:00.167-**|LBLCOMPUTE_SCI[00957]| DRS_ROOT: /cosmos99/nirps/git-bin/apero-drs-online/apero
19:12:00.168-**|LBLCOMPUTE_SCI[00957]| DRS_DATA_RAW: /cosmos99/nirps/apero-data/nirps_he_online/raw
19:12:00.169-**|LBLCOMPUTE_SCI[00957]| DRS_DATA_REDUC: /cosmos99/nirps/apero-data/nirps_he_online/red
19:12:00.169-**|LBLCOMPUTE_SCI[00957]| DRS_DATA_WORKING: /cosmos99/nirps/apero-data/nirps_he_online/tmp
19:12:00.170-**|LBLCOMPUTE_SCI[00957]| DRS_CALIB_DB: /cosmos99/nirps/apero-data/nirps_he_online/calib
19:12:00.170-**|LBLCOMPUTE_SCI[00957]| DRS_TELLU_DB: /cosmos99/nirps/apero-data/nirps_he_online/tellu
19:12:00.171-**|LBLCOMPUTE_SCI[00957]| DRS_DATA_ASSETS: /cosmos99/nirps/apero-data/nirps_he_online/asset
19:12:00.172-**|LBLCOMPUTE_SCI[00957]| DRS_DATA_OUT: /cosmos99/nirps/apero-data/nirps_he_online/out
19:12:00.172-**|LBLCOMPUTE_SCI[00957]| DRS_DATA_MSG: /cosmos99/nirps/apero-data/nirps_he_online/msg
19:12:00.173-**|LBLCOMPUTE_SCI[00957]| DRS_DATA_RUN: /cosmos99/nirps/apero-data/nirps_he_online/run
19:12:00.173-**|LBLCOMPUTE_SCI[00957]| DRS_DATA_PLOT: /cosmos99/nirps/apero-data/nirps_he_online/plot
19:12:00.174-**|LBLCOMPUTE_SCI[00957]| DRS_DATA_OTHER: /cosmos99/nirps/apero-data/nirps_he_online/other
19:12:00.175-**|LBLCOMPUTE_SCI[00957]| LBL_PATH: /cosmos99/nirps/apero-data/nirps_he_online/lbl
19:12:00.175-**|LBLCOMPUTE_SCI[00957]| DRS_CONFIG: /cosmos99/nirps/apero-settings/nirps_he_online/user_config.ini
19:12:00.176-**|LBLCOMPUTE_SCI[00957]| DRS_CONFIG: /cosmos99/nirps/apero-settings/nirps_he_online/user_constants.ini
19:12:00.176-**|LBLCOMPUTE_SCI[00957]| DRS_CONFIG: apero.constants.default.default_config
19:12:00.177-**|LBLCOMPUTE_SCI[00957]| DRS_CONFIG: apero.constants.default.default_keywords
19:12:00.178-**|LBLCOMPUTE_SCI[00957]| DRS_CONFIG: config.instruments.nirps_he.default_config.py
19:12:00.178-**|LBLCOMPUTE_SCI[00957]| DRS_CONFIG: config.instruments.nirps_he.default_keywords.py
19:12:00.179-**|LBLCOMPUTE_SCI[00957]| DRS_CONFIG: core.instruments.default.default_constants.py
19:12:00.180-**|LBLCOMPUTE_SCI[00957]| DRS_CONFIG: core.instruments.nirps_he.default_constants.py
19:12:00.180-**|LBLCOMPUTE_SCI[00957]| DATABASE: MYSQL
19:12:00.181-**|LBLCOMPUTE_SCI[00957]| DATABASE-calib: nirps@rali:calib_nirps_he_online_db
19:12:00.181-**|LBLCOMPUTE_SCI[00957]| DATABASE-tellu: nirps@rali:tellu_nirps_he_online_db
19:12:00.182-**|LBLCOMPUTE_SCI[00957]| DATABASE-findex: nirps@rali:findex_nirps_he_online_db
19:12:00.183-**|LBLCOMPUTE_SCI[00957]| DATABASE-log: nirps@rali:log_nirps_he_online_db
19:12:00.183-**|LBLCOMPUTE_SCI[00957]| DATABASE-astrom: nirps@rali:astrom_nirps_he_online_db
19:12:00.184-**|LBLCOMPUTE_SCI[00957]| DATABASE-lang: nirps@rali:lang_nirps_he_online_db
19:12:00.184-**|LBLCOMPUTE_SCI[00957]| DATABASE-reject: nirps@rali:reject_nirps_he_online_db
19:12:00.185-**|LBLCOMPUTE_SCI[00957]| PRINT_LEVEL: all
19:12:00.186-**|LBLCOMPUTE_SCI[00957]| LOG_LEVEL: all
19:12:00.186-**|LBLCOMPUTE_SCI[00957]| DRS_PLOT: 0
19:12:00.195- |LBLCOMPUTE_SCI[00957]| ***************************************************************************
19:12:00.206- |LBLCOMPUTE_SCI[00957]|A[40-001-00010]: System information
19:12:00.207- |LBLCOMPUTE_SCI[00957]| ***************************************************************************
19:12:00.208- |LBLCOMPUTE_SCI[00957]| Python version = '3.9.16'
19:12:00.209- |LBLCOMPUTE_SCI[00957]| Distribution date = 'main, Mar 8 2023, 14:00:05'
19:12:00.209- |LBLCOMPUTE_SCI[00957]| Other information = 'GCC 11.2.0'
19:12:00.210- |LBLCOMPUTE_SCI[00957]| Path = '/cosmos99/nirps/bin/miniconda3/envs/apero-env-07/bin/python'
19:12:00.211- |LBLCOMPUTE_SCI[00957]| Platform = 'linux'
19:12:00.211- |LBLCOMPUTE_SCI[00957]| Currently installed python modules:
19:12:00.212- |LBLCOMPUTE_SCI[00957]| astropy: 5.3 (req==5.3)
19:12:00.212- |LBLCOMPUTE_SCI[00957]| astroquery: 0.4.6 (req==0.4.6)
19:12:00.213- |LBLCOMPUTE_SCI[00957]| barycorrpy: 0.4.4 (req==0.4.4)
19:12:00.214- |LBLCOMPUTE_SCI[00957]| duckdb: 0.8.0 (req==0.8)
19:12:00.214- |LBLCOMPUTE_SCI[00957]| matplotlib: 3.7.1 (req==3.7.1)
19:12:00.215- |LBLCOMPUTE_SCI[00957]| numba: 0.57.0 (req==0.57.0)
19:12:00.216- |LBLCOMPUTE_SCI[00957]| numexpr: 2.8.4 (req==2.8.4)
19:12:00.216- |LBLCOMPUTE_SCI[00957]| numpy: 1.24.3 (req==1.24.3)
19:12:00.217- |LBLCOMPUTE_SCI[00957]| pandas: 2.0.2 (req==2.0.2)
19:12:00.217- |LBLCOMPUTE_SCI[00957]| pandastable: 0.13.1 (req==0.13.1)
19:12:00.218- |LBLCOMPUTE_SCI[00957]| psutil: 5.9.4 (req==5.9.4)
19:12:00.219- |LBLCOMPUTE_SCI[00957]| scipy: 1.10.0 (req==1.10.0)
19:12:00.219- |LBLCOMPUTE_SCI[00957]| tqdm: 4.65.0 (req==4.65.0)
19:12:00.220- |LBLCOMPUTE_SCI[00957]| yagmail: 0.15.293 (req==0.15.293)
19:12:00.220- |LBLCOMPUTE_SCI[00957]| lbl: 0.63.002 (git+https://github.com/njcuk9999/lbl.git)
19:12:00.221- |LBLCOMPUTE_SCI[00957]| Raw arguments used:
19:12:00.222- |LBLCOMPUTE_SCI[00957]| Arg 1 = 'apero_processing.py'
19:12:00.222- |LBLCOMPUTE_SCI[00957]| ***************************************************************************
19:12:00.231-**|LBLCOMPUTE_SCI[00957]|I[40-001-00017]: Arguments used:
19:12:00.239-**|LBLCOMPUTE_SCI[00957]|
19:12:00.240-**|LBLCOMPUTE_SCI[00957]| --OBJNAME: TOI210
19:12:00.241-**|LBLCOMPUTE_SCI[00957]| --ITERATION: -1
19:12:00.242-**|LBLCOMPUTE_SCI[00957]| --TOTAL: -1
19:12:00.242-**|LBLCOMPUTE_SCI[00957]| --DEBUG: None
19:12:00.243-**|LBLCOMPUTE_SCI[00957]| --PROGRAM: LBLCOMPUTE_SCI[00957]
19:12:00.244-**|LBLCOMPUTE_SCI[00957]| --RECIPE_KIND: lbl-compute-sci
19:12:00.244-**|LBLCOMPUTE_SCI[00957]| --PARALLEL: True
19:12:00.245-**|LBLCOMPUTE_SCI[00957]| --SHORTNAME: LBLCOMPUTE_SCI
19:12:00.245-**|LBLCOMPUTE_SCI[00957]| --IDEBUG: None
19:12:00.246-**|LBLCOMPUTE_SCI[00957]| --REF: None
19:12:00.247-**|LBLCOMPUTE_SCI[00957]| --CRUNFILE: online_run.ini
19:12:00.247-**|LBLCOMPUTE_SCI[00957]| --NOSAVE: False
19:12:00.248-**|LBLCOMPUTE_SCI[00957]| --FORCE_INDIR: None
19:12:00.249-**|LBLCOMPUTE_SCI[00957]| --FORCE_OUTDIR: None
19:12:00.258- |LBLCOMPUTE_SCI[00957]| ***************************************************************************
19:12:01.250-**|LBLCOMPUTE_SCI[00957]|7 DTEMP files found.
19:12:01.262-**|LBLCOMPUTE_SCI[00957]|Running LBL compute for TOI210_TOI210
19:12:02.305-**|LBLCOMPUTE_SCI[00957]|Running LBL compute for TOI210_GJ3737
19:12:07.231-**|LBLCOMPUTE_SCI[00957]|I[40-005-10001]: QUALITY CONTROL SUCCESSFUL - Well Done -
19:12:07.775-!!|LBLCOMPUTE_SCI[00957]|LBL Compute Exception [TOI210_TOI210] <class 'lbl.core.base_classes.LblException'>: File /cosmos99/nirps/apero-data/nirps_he_online/lbl/masks/TOI210_full.fits cannot be found
19:12:07.776-!!|LBLCOMPUTE_SCI[00957]|
19:12:07.777-!!|LBLCOMPUTE_SCI[00957]|LBL Compute Exception [TOI210_GJ3737] <class 'lbl.core.base_classes.LblException'>: Cannot load template fits file. Filename: /cosmos99/nirps/apero-data/nirps_he_online/lbl/templates/Template_s1dv_TOI210_sc1d_v_file_A.fits
19:12:07.777-!!|LBLCOMPUTE_SCI[00957]| <class 'FileNotFoundError'>: [Errno 2] No such file or directory: '/cosmos99/nirps/apero-data/nirps_he_online/lbl/templates/Template_s1dv_TOI210_sc1d_v_file_A.fits'
19:12:07.787-!!|LBLCOMPUTE_SCI[00957]|Traceback (most recent call last):
19:12:07.788-!!|LBLCOMPUTE_SCI[00957]| File "/cosmos99/nirps/git-bin/apero-drs-online/apero/core/utils/drs_startup.py", line 433, in run
19:12:07.790-!!|LBLCOMPUTE_SCI[00957]| llmain = func(recipe, params)
19:12:07.791-!!|LBLCOMPUTE_SCI[00957]| File "/cosmos99/nirps/git-bin/apero-drs-online/apero/recipes/nirps_he/apero_lbl_compute_nirps_he.py", line 231, in __main__
19:12:07.791-!!|LBLCOMPUTE_SCI[00957]| WLOG(params, 'error', '\n\n'.join(errors))
19:12:07.792-!!|LBLCOMPUTE_SCI[00957]| File "/cosmos99/nirps/git-bin/apero-drs-online/apero/core/core/drs_log.py", line 428, in __call__
19:12:07.793-!!|LBLCOMPUTE_SCI[00957]| raise drs_exceptions.LogExit(errorstring)
19:12:07.793-!!|LBLCOMPUTE_SCI[00957]|apero.core.core.drs_exceptions.LogExit: LBL Compute Exception [TOI210_TOI210] <class 'lbl.core.base_classes.LblException'>: File /cosmos99/nirps/apero-data/nirps_he_online/lbl/masks/TOI210_full.fits cannot be found
19:12:07.794-!!|LBLCOMPUTE_SCI[00957]|
19:12:07.794-!!|LBLCOMPUTE_SCI[00957]|LBL Compute Exception [TOI210_GJ3737] <class 'lbl.core.base_classes.LblException'>: Cannot load template fits file. Filename: /cosmos99/nirps/apero-data/nirps_he_online/lbl/templates/Template_s1dv_TOI210_sc1d_v_file_A.fits
19:12:07.795-!!|LBLCOMPUTE_SCI[00957]| <class 'FileNotFoundError'>: [Errno 2] No such file or directory: '/cosmos99/nirps/apero-data/nirps_he_online/lbl/templates/Template_s1dv_TOI210_sc1d_v_file_A.fits'
19:12:07.796-!!|LBLCOMPUTE_SCI[00957]|
19:12:07.796-!!|LBLCOMPUTE_SCI[00957]|
19:12:08.821-**|LBLCOMPUTE_SCI[00957]| ***************************************************************************
19:12:08.841-@!|LBLCOMPUTE_SCI[00957]|W[40-003-00005]: Recipe apero_lbl_compute_nirps_he has NOT been successfully completed
19:12:08.850-**|LBLCOMPUTE_SCI[00957]| ***************************************************************************
There was also a apero_processing error
"21:40:18.662-**|PROC| ***************************************************************************
21:40:18.669-**|PROC|I[40-503-00018]: * GROUP 150/684 (MKTELLU1)
21:40:18.677-**|PROC| ***************************************************************************
21:40:18.685-**|PROC|
21:40:18.897-**|PROC|ID00745|C01/05| apero_mk_tellu_nirps_ha.py 2023-01-19 NIRPS_2023-01-20T09_00_04_634_pp_e2dsff_A.fits --crunfile=science_run_custom.ini --program=MKTELLU1[00745] --recipe_kind=tellu-hotstar --shortname=MKTELLU1 --parallel=True
21:40:19.085-**|PROC|ID00746|C02/05| apero_mk_tellu_nirps_ha.py 2023-01-19 NIRPS_2023-01-20T08_59_25_620_pp_e2dsff_A.fits --crunfile=science_run_custom.ini --program=MKTELLU1[00746] --recipe_kind=tellu-hotstar --shortname=MKTELLU1 --parallel=True
21:40:19.324-**|PROC|ID00747|C03/05| apero_mk_tellu_nirps_ha.py 2023-01-19 NIRPS_2023-01-20T08_58_46_606_pp_e2dsff_A.fits --crunfile=science_run_custom.ini --program=MKTELLU1[00747] --recipe_kind=tellu-hotstar --shortname=MKTELLU1 --parallel=True
21:40:19.610-**|PROC|ID00748|C04/05| apero_mk_tellu_nirps_ha.py 2023-01-19 NIRPS_2023-01-20T08_58_07_592_pp_e2dsff_A.fits --crunfile=science_run_custom.ini --program=MKTELLU1[00748] --recipe_kind=tellu-hotstar --shortname=MKTELLU1 --parallel=True
21:40:19.803-**|PROC|ID00749|C05/05| apero_mk_tellu_nirps_ha.py 2023-01-19 NIRPS_2023-01-20T02_54_27_771_pp_e2dsff_A.fits --crunfile=science_run_custom.ini --program=MKTELLU1[00749] --recipe_kind=tellu-hotstar --shortname=MKTELLU1 --parallel=True"
Hi! Yer I think you'll continue getting the same errors, I think it is trying to redo all targets that failed every time. I haven't had chance to look into this so you'll continue to get errors like this.
Do you have any additional information on the "apero_proccessing error" that just looks like a list of ids for one of the group there doesn't seem to be a crash message or an error there?
There's no error message with the apero_processing error. The box is empty in the recipe table.
So first thing if there ever isn't an error message in the box (i.e. it says None) you can look at the log file (which shouldn't be blank)
The second thing to note "No lblrv file I feel like this is all the targets..." it's not all targets and most of these targets don't appear in the raw files for 2024-02-27 to 2024-02-29 so something is going wrong with apero selecting the right files (it may be trying all nights and redoing ones that failed previous... I'll try to figure this out)
Missing Teff means we don't have a Teff entry in the database and we need to find a Teff for this object.
The cannot load templates is weird for TOI2952 I see the tcorr files, they seemed to have passed but I don't see a template in apero (explains why lbl can't find one either) no idea why this is.
For the error message, what I wrote in the message from 2 days ago (last box, https://github.com/njcuk9999/apero-utils/issues/218#issuecomment-1974277950) is taken directly from the log file. It's the last few lines, but I looked through the whole thing and didn't see an error (here's the file for reference /cosmos99/nirps/apero-data/nirps_ha_online/msg/tool/other/APEROL-PID-00017093161623219610-FYMG_apero_processing.log)
Ah okay I see sorry that means python crashed/got killed and the log stopped dead (normally a Ctrl+C wouldn't do this unless multiple Ctrl+C were used) otherwise I'd blame a server restart (which I don't think happened) or a memory error
Looking at the top it looks like this was a "science_run_custom.ini" (probably @CharlesCadieux was running this?) so not anything you were running via manual_trigger.
If a recipe finished (either successfully or unsuccessfully it should finish with a statement saying "Code finishes successfully" or "Code finished unsuccessfully" even on a python error - anything else is weird (but again can be caused by multiple Ctrl+Cs or a memory error or python quiting being killed before it can "gracefully" exit)
I think we can ignore this error this time! Which just leaves all the lbl problems
For the ones that are missing templates:
Seems this is what happened:
I think this is a problem with the offline2online script not copying something to the log database as normally this isn't a problem. Normally if QC fails the template should be allowed to re-run -- though I need to confirm this.
I think the reason we had so many objects in the recipe log on 2024-02-27 is this is not sorted per "observation directory" but when things were processed - on this date I did the big "catch-up" run that spreads over the next few days - thus you get many more objects reporting errors.
Globally we get many lbl errors due to the following:
So turns out the template problem wasn't an offline2online one but a general problem - no templates will be calculated if they have been run before (even if QC fails) I've added a patch that should fix this. I'm running those templates now (+the rest of the telluric + lbl steps after)
Since 2024-02-28, I'm getting many APERO error messages linked to the apero_lbl_compile_nirps_ha recipe (shortname = LBLCOMPILE_SCI). Here is an example for every type of error message:
class apero.core.core.drs_exceptions.LogExit : No Teff found: -- For objects with "real" names: BETAPICMB
-- For objects with weird names: NIGHT_SKYMSKY_M_HA
LBL Compile Exception [BDP07_2692_BDP07_2692] class lbl.core.base_classes.LblException : No lblrv files found No weird object names here
LBL Compile Exception [GJ1164A_GJ1164A] class lbl.core.base_classes.LblException : Unexpected lbl_compile.py error: class ValueError : zero-size array to reduction operation minimum which has no identity LBL Compile Exception No weird object name here