search

LSSTDESC / Twinkles

10 years. 6 filters. 1 tiny patch of sky. Thousands of time-variable cosmological distance probes.
MIT License
13 stars 12 forks source link

Choose visit image IQ window for successful template generation #435

Open drphilmarshall opened 7 years ago

drphilmarshall commented 7 years ago

@SimonKrughoff Sounded like a minimum IQ of 2.5 pixels might work - keep us posted! When you find a set up that works, I guess we'll need the cookbook updated for @tony-johnson to work off. Good luck!

PS. Ill add this to the Run 3 DM Workflow epic #421

SimonKrughoff commented 7 years ago

Some background. I thought that u-band was not subject to the issue we were seeing. I.e. in r-band: image

But it also happens in u-band. image

It turns out that there are far fewer images in u-band with IQ in the range 2. - 2.5 pixels.

SimonKrughoff commented 7 years ago

Here is an r-band image with reported FWHM = 2.5: image

It seems to be right on the hairy edge. E.g. for another image with FWHM = 2.45: image

To be conservative, I'm going to suggest we make the window 2.6 < FWHM < 3.0. Here is an example of an image with FWHM = 2.63: image

I am wondering if this has something to do with a minimum size for the PSF basis set.

drphilmarshall commented 7 years ago

Could be. 2.6 pixels seems good for now - will that give us enough frames to make a decent DR1 co-add (for example)? How are you planning to communicate this to the rest of the DM team?

SimonKrughoff commented 7 years ago

I'll file a ticket. The problem is putting together a minimal "how to repeat".

SimonKrughoff commented 7 years ago

And I think it will have enough frames, but I don't have a histogram on hand to show that.

SimonKrughoff commented 7 years ago

O.K. I ran a coadd generation with makeCoaddTempExp.py /global/cscratch1/sd/desc/twinkles/work/6/output --output $SCRATCH/test_out --selectId filter=r --id filter=r --config modelPsf.defaultFwhm=3.00 select.minPsfFwhm=2.60 select.maxPsfFwhm=3.00 --clobber-config --no-versions. That seemed to work pretty well. It produces a coadd with 166 visits (over all 10 years) as opposed to 295 visits.

Here's a cutout: image

There are still a few artifacts I don't understand, but we should look at some diffims to see how they look.

I'll update the cookbook later today.

drphilmarshall commented 7 years ago

Splendid! Looks like you've cracked it. For DR1 I guess we can expect 10-20 images. Think that'll be enough?

How many years of data should we try to analyze then, before the DESC meeting? At the meeting we should discuss the DR schedule for Twinkles 1.

SimonKrughoff commented 7 years ago

Yes. 10-20 should be plenty. Of course, the fewer images that go into the template, the less sensitivity we have, but we'll still get something useful out.

SimonKrughoff commented 7 years ago

FYI, I didn't get the cookbook out yesterday, because I'm trying to update both at once since they are related.

drphilmarshall commented 7 years ago

No worries. I'm going to wait until I see what they both look like before deciding how to Note-ify them. Could be they just make a pair of Notes.

On Fri, Jan 27, 2017 at 9:41 AM, SimonKrughoff notifications@github.com wrote:

FYI, I didn't get the cookbook out yesterday, because I'm trying to update both at once since they are related.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/LSSTDESC/Twinkles/issues/435#issuecomment-275726052, or mute the thread https://github.com/notifications/unsubscribe-auth/AArY955HiqFKQk8uiAon3xNAqbi-94D8ks5rWixXgaJpZM4Lt8xE .

SimonKrughoff commented 7 years ago

O.K. so this ended up being a the place where I did a bunch of things. I now have a cookbook that sketches out both level 1 and level 2 processing using appropriate configs for each. There are still some things I'd like to try out: e.g. a different PSF estimation algorithm.

I don't know how to review this and get it merged, but my suggestion is that we use this cookbook to do run 3. I also suggest merging twinkles_#435 to twinkles_3 instead of merging it with twinkle_1, but I'm can certainly be talked out of that.

drphilmarshall commented 7 years ago

Great! I see that the diaObjectMaker doesn't take a filter as input - does this mean that it now makes multi-filter objects with a single centroid (as opposed to one centroid per filter)? Have you managed to make any such DIAObjects? I am excited :-)

On Fri, Feb 3, 2017 at 10:21 AM, SimonKrughoff notifications@github.com wrote:

O.K. so this ended up being a the place where I did a bunch of things. I now have a cookbook that sketches out both level 1 and level 2 processing using appropriate configs for each. There are still some things I'd like to try out: e.g. a different PSF estimation algorithm.

I don't know how to review this and get it merged, but my suggestion is that we use this cookbook to do run 3. I also suggest merging twinkles_#435 to twinkles_3 instead of merging it with twinkle_1, but I'm can certainly be talked out of that.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/LSSTDESC/Twinkles/issues/435#issuecomment-277322445, or mute the thread https://github.com/notifications/unsubscribe-auth/AArY94q31d_GkFoGX9RKCCuT7ybAT6Jfks5rY3BAgaJpZM4Lt8xE .

SimonKrughoff commented 7 years ago

So the diaObjectMaker.py can take any valid dataId, so you can build per filter DiaObjects, but if you don't specify a filter it will aggregate over all of them.

Just to be clear, I am not doing any sort of averaging over the source properties. That's hard enough to get right that I want to do that in the issue that covers the forced photometry. @wmwv may have ideas about that.

I have made a few single band DiaObjects, but one remaining issue is how to reject the epochs from the DiaObject that went into the coadd.

drphilmarshall commented 7 years ago

What happens if you don't reject them? Could you simply flag them instead of rejecting them, please?

I think we are probably all OK with the DIAObjects having just forced photometry at this point. As long as the Monitor can pull out all the DIASources that belong to each DIAObject then we can do some interesting multi-epoch analysis.

Also I am pretty sure that we will want multi-filter aggregation as long as the inter-band astrometric registration is good.

@jbkalmbach @rbiswas4 @wmwv PLease chip in with comments and feedback for Simon - he is enabling marvellous things here :-)

SimonKrughoff commented 7 years ago

I just updated the cookbook again. I'm considering this a fairly complete list of example command lines except forced phot for diaObjects.

drphilmarshall commented 7 years ago

@SimonKrughoff Looks like the cookbook recipe at https://github.com/LSSTDESC/Twinkles/blob/master/doc/Cookbook/Coadd_Diffim_Recipe.md needs finalizing to reflect the final scheme (including using Year 1 for the template and Years 2-10 for science).

SimonKrughoff commented 7 years ago

O.K. I will try to get to this today.

rbiswas4 commented 7 years ago

Since we are looking into this now again to understand what needs to be done.

@SimonKrughoff : Is this still the recommended way to go?

If I understand the last comment right, we need to build the template for coadd using the visits in year 1. Is that correct @drphilmarshall ? We have those obsHistIDs. Is the way to implement these here in adding the list of obsHistIDs to the makeDiscreteSkyMap flag --id @SimonKrughoff ?

Thanks!