search

LSSTDESC / DC2-production

Configuration, production, validation specifications and tools for the DC2 Data Set.
BSD 3-Clause "New" or "Revised" License
11 stars 7 forks source link

PhoSim Validation Tests #142

Closed salmanhabib closed 5 years ago

salmanhabib commented 6 years ago

Since #140 was getting too long, I opened this issue for discussing and writing down quantitative PhoSim validation tests. Where such tests already exist, please provide links to them.

katrinheitmann commented 6 years ago

To get more concrete, here is the plan that we have been all discussing over the last few days. If anybody sees something wrong with this plan, please comment!

We have now (i) raw phosim images for more than 20 phosim settings and (ii) data from these 20+ tests processed with processEimage.py, (iii) power spectra results, (iv) a table looking at the edge effects. (If you need pointers to any of these, please ask).

From the results so far we have come up with the following steps:

Just to reiterate Salman's point above we are very much in need of some quantitative criteria. One was pointed out by Zeljko in issue #140 already.

rmandelb commented 6 years ago

I heard this plan before and I think it's mostly a good one. Just one question: are we definitely sure that the issues that we saw with sky background might not result from some interaction between how sky and object photons are handled? If there could be some unforeseen interaction, then our "no sources" test might give us results that seem too good. It seems like a simple way to check would be to choose a setting that we had already flagged as being quite problematic, and see how it behaves in the "no sources" scenario, but I wasn't sure if this is something that has already been investigated.

rmandelb commented 6 years ago

Aside from my above objection/question, as I said, i think your plan is good. Let me offer a few updates and suggestions for how we can get a bit more quantitative:

You said "Get more input from the experts on the fall-off on the edge that Seth has calculated" -> This has happened on #140, and it seems to have converged towards this being a real effect, not a bug - according to contributions from Chris Stubbs, Chris Walter, Andy Connolly (and comments from Alex Drlica-Wagner on slack). It shows up as a sharper feature with full background and a smoother feature with quickbackground because the optimization works in a way that smooths out sharp features in the background. So the different between the two cases also makes sense, and the full background case is more realistic at least in terms of properly representing the sharpness of the feature.

Given this information, my proposal is that:

As a totally separate issue from this validation test, we can decide whether we want to have this edge roll-off feature in DC2 (if it is even something that can be turned off). In other words, just like b/f and tree rings, we made a decision as to whether we wanted it in the sims despite possibly not being able to correct for it in an optimal way, and we should think of the edge roll-off in a similar manner: it's a real sensor effect, so do we want it in DC2 or not?

But as for whether DM can handle this, I had a chat with @TallJimbo . His reaction was that if it's a real effect, then DM would be happy to incorporate code that deals with it by masking around the edges during ISR (or some solution like that). He also stated that the edges of chips do get masked at some point, "but primarily to indicate that we couldn't smooth them when performing detection, and in many contexts we do still use those pixels." I gather that masking during ISR would take care of this problem and avoid them ever using those pixels for background subtraction, source detection, etc.

astrostubbs commented 6 years ago

It has always been the default plan to mask out a picture frame around each CCD, for any edge rolloff that leaves unacceptable post-correction residuals.

C

Christopher Stubbs Samuel C. Moncher Professor of Physics and of Astronomy 17 Oxford Street Harvard University Cambridge MA 02138 stubbs@physics.harvard.edu

Think this email is terse? Here's why: http://www.emailcharter.org

On Mar 13, 2018, at 8:52 PM, Rachel Mandelbaum notifications@github.com wrote:

Aside from my above objection/question, as I said, i think your plan is good. Let me offer a few updates and suggestions for how we can get a bit more quantitative:

You said "Get more input from the experts on the fall-off on the edge that Seth has calculated" -> This has happened on #140, and it seems to have converged towards this being a real effect, not a bug - according to contributions from Chris Stubbs, Chris Walter, Andy Connolly (and comments from Alex Drlica-Wagner on slack). It shows up as a sharper feature with full background and a smoother feature with quickbackground because the optimization works in a way that smooths out sharp features in the background. So the different between the two cases also makes sense, and the full background case is more realistic at least in terms of properly representing the sharpness of the feature.

Given this information, my proposal is that:

our background validation tests should explicitly exclude the edges where this occurs we should do your above tests on the non-excluded part of the chips: check whether the DM background model can describe well the sky in the non-excluded parts of the chips As a totally separate issue from this validation test, we can decide whether we want to have this edge roll-off feature in DC2 (if it is even something that can be turned off). In other words, just like b/f and tree rings, we made a decision as to whether we wanted it in the sims despite possibly not being able to correct for it in an optimal way, and we should think of the edge roll-off in a similar manner: it's a real sensor effect, so do we want it in DC2 or not?

But as for whether DM can handle this, I had a chat with @TallJimbo . His reaction was that if it's a real effect, then DM would be happy to incorporate code that deals with it by masking around the edges during ISR (or some solution like that). He also stated that the edges of chips do get masked at some point, "but primarily to indicate that we couldn't smooth them when performing detection, and in many contexts we do still use those pixels." I gather that masking during ISR would take care of this problem and avoid them ever using those pixels for background subtraction, source detection, etc.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

cwwalter commented 6 years ago

It has always been the default plan to mask out a picture frame around each CCD, for any edge rolloff that leaves unacceptable post-correction residuals.

Yes, our plan now was to introduce masks for it in DC2 processing.

johnrpeterson commented 6 years ago

posted this on #140, but relevant here as well:

so, it looks like the backgamma is the critical parameter.

based on both the timings and the fidelity, we should use:

airglowvariation 0.0 backalpha 0.1 backbeta 6.0 backdelta 1.0 activebuffer 1000.0

and then we should set backgamma based on what is at least a factor of ~3 below realistic background variations. visually, it looks like

backgamma 300.0

would be the best for now, but we could test it quantitatively.

salmanhabib commented 6 years ago

Should we close this thread due to lack of input?

katrinheitmann commented 6 years ago

No, we shouldn't. Work is going to start again on this with Run 1.2p being complete now. @fjaviersanchez has started organizing the effort some more in the original overarching validation epic: https://github.com/LSSTDESC/DC2_Repo/issues/34 and this issue is one of the once listed in the tables there. So we should all help out and make sure Run 1.2p looks good in all aspects. We need volunteers!