deep wavecal recipe fails on looking up DQ frame

[semaphoreP]

Trying to process the latest wavecals taken this month using the Data Cruncher/Data Parser, I've run into the following error:

On closer inspection, the appropriate line of code is the following line (the one that isn't a comment): ;READ IN BAD PIXEL MAP ; assume this is already present in the DQ extension. badpix=*dataset.currdq GT 0

Have there been recent changes to how the DQ frame works? Have anyone else hit this issue?

Here's the recipe to reproduce the bug: <?xml version="1.0" encoding="UTF-8"?>

[mperrin]

attention @swolff9 @kfollette

[mperrin]

@semaphoreP the recipe you pasted didn't seem to show up cleanly here in github

[semaphoreP]

Whoops, here's the recipe file as a text file:

wavecal_dq_fail.txt

[swolff9]

OK, so I've traced the problem, but it doesn't have an obvious solution. At the end of the quick wavecal primitive, the current frame is set to the shifted wavecal so that the new wavecal can be written. The DQ extensions are also reset at this point: *dataset.currframe = shiftedwavecal ptr_free, dataset.currDQ, dataset.currUncert

When we go to reduce the full wavelength solution, it doesn't get the raw files it was expecting, and the thing fails. It actually fails first when trying to call the DQ extension, but it will still fail even if we don't clear the data quality extension in the quick wavecal primitive because the current frame is incorrect.

The goal of adding the quick wavelength solution primitive to the developer code was to handle the all too common case where there is a 3 pixel shift from one deep wavecal sequence to another. The fit will fail because the boxes to extract the spectra for fitting are drawn in the wrong place. Generally I fix this manually by running the quick wavelength solution on the data which is more robust to 3 pixel shifts, but less accurate.

It seems the obvious solution would be to just run two recipes on all wavecal files. First the quick, then the full developer version. However, unless I am mistaken the data parser logic only allows one recipe template name to be returned. Am I mistaken in this? Anyone see a better solution?

[swolff9]

For now I've pushed a change to the recipe template to remove the quick wavecal portion.

[mperrin]

Actually it turns out that we did work out a way for the data parser to make more than one recipe using different templates on the same data set. Take a look at how the pol calibrations are created in parsercore. Instead of trying to pass back multiple template names we just create the several recipes right away inside the case statement. So you can definitely set it up with two recipes for the wavecal reductions. Thanks for figuring this out.

[swolff9]

Great, I've made the changes and am running a test now. It looks like I've generated the recipes in the correct order, so that the quick wavecals will be generated before the deep ones. If all goes well, I will push the changes to the repository in a few minutes. Jason: will you have a chance to test this on the Data Cruncher before the run, or should I generate the wavecals?

[swolff9]

OK, changes pushed

[semaphoreP]

Cool! I can run them through the data cruncher later today and check them tomorrow. Thanks for figure it out!

[semaphoreP]

H band deep wavecals were successful so I think the workaround should be fine. Closing the issue now!

[swolff9]

thanks