NightQA script error

[rongpu]

The nightqa script failed with the following error

############################################
# tail of nightqa log nightqa-daily-20240327.log:#
############################################
INFO:night_qa.py:142:get_surveys_night_expids: for NIGHT=20240327 found 34 exposures from 32 tiles for SURVEY=main and 2 exposures from 2 tiles for SURVEY=special
INFO:night_qa.py:168:get_dark_night_expid: proctable_fn = /global/cfs/cdirs/desi/spectro/redux/daily/processing_tables/processing_table_daily-20240327.csv
Traceback (most recent call last):
  File "/global/common/software/desi/perlmutter/desiconda/20230111-2.1.0/code/desispec/main/bin/desi_night_qa", line 208, in <module>
    main()
  File "/global/common/software/desi/perlmutter/desiconda/20230111-2.1.0/code/desispec/main/bin/desi_night_qa", line 144, in main
    dark_expid = get_dark_night_expid(args.night, args.prod)
  File "/global/common/software/desi/perlmutter/desiconda/20230111-2.1.0/code/desispec/main/py/desispec/night_qa.py", line 185, in get_dark_night_expid
    expid = int(str(d["EXPID"][sel][0]).strip("|"))
ValueError: invalid literal for int() with base 10: '232598|232668|232665'

[araichoor]

from a quick look:

in the night_qa.py script, I identify the 300s dark exposure from the processing_table with this: https://github.com/desihub/desispec/blob/46d8d51e6561cbcd63fa23f2eaee142e32fc83db/py/desispec/night_qa.py#L173C9-L173C70

        sel = (d["OBSTYPE"] == "dark") & (d["JOBDESC"] == "ccdcalib")

it should return one row (which it does here), but for 20240327, the EXPID field is 232598|232668|232665|, i.e. contains three exposures instead of one...:

• 232598: DARK, CALIB Dark 5min (it is the 300s dark exposure the code is looking for)
• 232668: FLAT, LED03 1s flat for CTE check
• 232665: FLAT, CALIB DESI-CALIB-03 LEDs only

@akremin : do we know why is that? the two "extra-exposures" are FLAT, not DARK? (I guess/think that it s the first time night_qa crashes in this mode)

[akremin]

Yes that is expected due to changes in the pipeline. The cte correction is also bundled into the ccdcalib and we now correctly list the flat exposures included in those fits. The dark should be the first expid in the list, if there is more than one and the OBSTYPE is dark. So the logic looks good except we now need to parse the line for the first entry. I'll take care of this.

[araichoor]

I see, thanks.

[akremin]

I fixed the original problem in PR #2206 , but now the code is crashing at a later step:

Traceback (most recent call last):
  File "/global/common/software/desi/perlmutter/desiconda/20230111-2.1.0/code/desispec/main/bin/desi_night_qa", line 208, in <module>
    main()
  File "/global/common/software/desi/perlmutter/desiconda/20230111-2.1.0/code/desispec/main/bin/desi_night_qa", line 156, in main
    create_dark_pdf(outfns["dark"], args.night, args.prod, dark_expid, args.nproc, bkgsub_science_cameras=args.dark_bkgsub_science_cameras)
  File "/global/common/software/desi/perlmutter/desiconda/20230111-2.1.0/code/desispec/main/py/desispec/night_qa.py", line 525, in create_dark_pdf
    proc_expids = [int(expid.strip("|")) for expid in d["EXPID"]]
  File "/global/common/software/desi/perlmutter/desiconda/20230111-2.1.0/code/desispec/main/py/desispec/night_qa.py", line 525, in <listcomp>
    proc_expids = [int(expid.strip("|")) for expid in d["EXPID"]]
ValueError: invalid literal for int() with base 10: '232598|232668|232665'

@anand I'm happy to fix this too, but can you give me guidance on what nightqa is doing at line 525? Is this the code for processing the 1200s morning dark? Or is this trying to find the 300s evening dark? Depending on what it's trying to find, I may approach the fix differently.

    # if set to None will judge necessity for preprocessing according to proctable                                                                                                                                                                                     
    # but allows manual override e.g. for cases where no proctable should be there                                                                                                                                                                                     
    if run_preproc is None:
        if not os.path.isfile(proctable_fn):
            run_preproc = True
        else:
            d = Table.read(proctable_fn)
            sel = d["OBSTYPE"] == "dark"
            d = d[sel]
            proc_expids = [int(expid.strip("|")) for expid in d["EXPID"]]
            if dark_expid not in proc_expids:
                run_preproc = True
            else:
                run_preproc = False
        # AR run preproc?  

[araichoor]

ah crap, sorry I didn t think hard about other places in the code...

this part of the code is common for dark/morningdark (it s a piece of code to see in the preproc needs to be run or not). I guess we d want to replace:

proc_expids = [int(expid.strip("|")) for expid in d["EXPID"]]

by something like:

proc_expids = []
for expid in d["EXPID"]:
   proc_expids += [int(_) for _ in expid.split("|")]

I didn t check the code, but you get the idea, hopefully!

[akremin]

Second issue is solved in PR #2207 . The code ran successfully at NERSC, so closing this issue.