Process Run3 data through DM level 2 with image differencing

LSSTDESC / Twinkles

10 years. 6 filters. 1 tiny patch of sky. Thousands of time-variable cosmological distance probes.

MIT License

13 stars 12 forks source link

Process Run3 data through DM level 2 with image differencing #421

Open tony-johnson opened 7 years ago

tony-johnson commented 7 years ago

The DM pipeline should be adapted to run the workflow from the newly update Image Differencing cookbook, using 3.00 (pixels) as the value for modelPsf.defaultFwhm and select.maxPsfFwhm.

This should then be run on the entire Run3 output at NERSC.

First step is for @SimonKrughoff to make sure that the cookbook is complete, then Tony will run the workflow.

In more detail, here are the issues in this epic:

[x] Image differencing recipe #248
[x] DIAObject production recipe #296
[ ] Unified level 2 pipeline cookbook as a DESC Note #436
[ ] Config files to enable regular coadds etc, before we start the image differencing #422
[ ] Speed up ingestSimImage #409
[ ] Find good template generation recipe, and write into cookbook #435
[x] Get lists of visits for Year 1 (template-making) and Years 2-10 (science images) #441

SimonKrughoff commented 7 years ago

We have run into a problem with this. When the IQ is too good we end up with artifacts in the matched images. This results in unusable coadds. I am not sure if this is because of numerical instability as the PSF gets close to critically sampled or some other bug.

I think the thing to do in the short term is to use a window in QI to make the coadds.

drphilmarshall commented 7 years ago

@SimonKrughoff Yes - now issued at #435

drphilmarshall commented 7 years ago

Hi @tony-johnson @SimonKrughoff

At the DESC meeting Twinkles session we decided to use the Year 1 visits to make the good seeing template image, and then difference all subsequent (Year 2 - Year 10) images against these 6 templates. It seems as though Simon's 2.6 - 3.0 pixel IQ window works pretty well, and we'll just have to swallow the loss of all visits with IQ better than 3.0 pixels (0.6") for the Run 3 analysis. What else, if anything, do you two need in order to run the 2-branch DM workflow that we saw last week? How did your reconnection conversation go?

Cheers

Phil

SimonKrughoff commented 7 years ago

I think things went really well. I think the workflow is in place. The remaining work is to figure out the data that go into each step.