search

desihub / desispec

DESI spectral pipeline
BSD 3-Clause "New" or "Revised" License
36 stars 24 forks source link

Nightqa crashes because of "special" survey tile #2358

Closed akremin closed 1 month ago

akremin commented 1 month ago

Nightqa crashes because of "special" survey tile in Kibo processing.

ERROR:tile_qa_plot.py:1114:get_tilecov: surveys=special not in allowed_surveys=sv1,sv2,sv3,main,catchall
Traceback (most recent call last):
  File "/global/common/software/desi/perlmutter/desiconda/20240425-2.2.0/code/desispec/0.66.2/bin/desi_night_qa", line 210, in <module>
    main()
  File "/global/common/software/desi/perlmutter/desiconda/20240425-2.2.0/code/desispec/0.66.2/bin/desi_night_qa", line 192, in main
    create_petalnz_pdf(outfns["petalnz"], args.night, args.prod, unq_tileids, unq_surveys, dchi2_threshold=25,
  File "/global/common/software/desi/perlmutter/desiconda/20240425-2.2.0/code/desispec/0.66.2/lib/python3.10/site-packages/desispec/night_qa.py", line 1796, in create_petalnz_pdf
    _, pix_ntilecovs, _, _, _ = get_tilecov(tileid, surveys=survey, programs=faprgrm.upper(), lastnight=night)
  File "/global/common/software/desi/perlmutter/desiconda/20240425-2.2.0/code/desispec/0.66.2/lib/python3.10/site-packages/desispec/tile_qa_plot.py", line 1115, in get_tilecov
    raise ValueError(msg)
ValueError: surveys=special not in allowed_surveys=sv1,sv2,sv3,main,catchall

Log: /global/cfs/cdirs/desi/spectro/redux/kibo/run/scripts/tsnr_nightqa/logs/nightqa-kibo-20210917.log

Script: /global/cfs/cdirs/desi/spectro/redux/kibo/run/scripts/tsnr_nightqa/scripts/nightqa-kibo-20210917.slurm

Command: desi_night_qa --nproc=64 --recompute -p kibo -n 20210917 -o /global/cfs/cdirs/desi/spectro/redux/kibo/nightqa/20210917

araichoor commented 1 month ago

ok, this one is a bug / typo in the code, the create_petalnz_pdf() function:

here: https://github.com/desihub/desispec/blob/33d11ac38df5c7e9b18cefe3d28b1cd1278b6dac/py/desispec/night_qa.py#L1670-L1678

it should be if sel.sum() != sel.size, not if sel.sum() > 0, i.e. apply the selection is there are some tiles to remove. I guess it didn t often trigger an error because the "error case" is when all tiles from the night are discarded. that s the case for 20210917:

INFO:night_qa.py:150:get_surveys_night_expids: for NIGHT=20210917 found 14 exposures from 10 tiles for SURVEY=cmx and 3 exposures from 2 tiles for SURVEY=special

note that this error also occured in jura (and maybe earlier): /global/cfs/cdirs/desi/spectro/redux/jura/run/scripts/qa/night-qa-20210917.log https://data.desi.lbl.gov/desi/spectro/redux/jura/nightqa/20210917/

akremin commented 1 month ago

Resolved with Pull Request #2361