"Twinkles: An LSST DESC Data Challenge Pathfinder" - talk slides, discussion

[drphilmarshall]

@DarkEnergyScienceCollaboration/twinkles

I started a slide set for our upcoming talk on July 8th, and then the Oxford meeting session a week and a half later. I put in a rough outline in the first 15 slides, and left the slides from our most recent "Twinkles as a pathfinder" talk (in April) later in the file. @jchiang87 @tony-johnson @richardxdubois @wmwv I think we should jump in to this and do what we can first (our names are mentioned!) and then use this thread to ping people we need new input from. What do you think?

[drphilmarshall]

Alright, we're off to a good start on this slide set, with some nice input from Jim on code development. Let's get some more content in there ASAP so we can see what we've got to work with!

[jchiang87]

I added some slides on the comparisons with the input truth and fitting with sncosmo to the slide set ( @rbiswas4 could you have a look and comment please?) It would be great if we could get the intro slides filled with content from @wmwv, @drphilmarshall, and @richardxdubois. @tony-johnson are you planning to present the slides on the workflow engine stuff? If not, then I'll write those slides, otherwise I'll just leave them to you.

[rbiswas4]

@jchiang87 That looks quite good! Here are a few comments:

• Generic comment about the slides: could you please add slide numbers on the slides. Useful to point to whenever we have remote attendance.
• Could you please point me to the SNcosmo calls you are using to fit the light curves? Slide 21:
• I don't understand the parameter n_sigma :
• x_0 is a multiplicative factor. So probably x_0 / model(x_0) is a better thing to plot rather than difference. In fact Delta(-2.5 log10(x0)) is what I would like to show.
• The Delta T plot is interesting. While the difference is ~ milli-day :) and that is small I am surprised by the diagonal structure. This might have something to do with the guess_t0 moral prior, and would be interesting to figure out.
• A good quantity to plot up would be Delta( -2.5 * log10(x0) + 0.11 x1 - 3.11 c) against z. This is very roughly speaking the distance modulus for each of the SN.
• I would say that the scale of x1 differences that is interesting is ~ 1, and for c that is around ~ 0.3. So, the plots show major outliers, but it is hard to look at the structure at the interesting scale. We should discuss a bit more about how to show these.

Other possibilities:

• Maybe scaled plots of the form Delta x/sigma(x)
• 2D plots.

I would be happy to help and probably have code lying around for some of this. The harder thing is to get all of it together. Some of this is in https://github.com/rbiswas4/sncosmo_lc_analysis. It might be good to try an offline/email discussion after tomorrow's meeting to list up things so that we can put ideas of what we would like together.

[tony-johnson]

Hi Jim, yes I will add slides on workflow, NERSC experience etc. Currently I can summarize the NERSC experience as "very frustrating":

[heather999]

Concerning NERSC - while I agree aspects have been frustrating (as has been dealing with DMstack installs which is completely unrelated to NERSC) - it would be helpful to point out that we should take greater advantage of coordinating with the NERSC folks - who can be very helpful when directly poked. (actually the same can be said of DM!) The upcoming Cori downtime starting in late Aug is a bit of a concern.. the timing has not been on our side.

[jchiang87]

@rbiswas4 Thanks for the comments! Here are some responses:

Generic comment about the slides: could you please add slide numbers on the slides. Useful to point to whenever we have remote attendance.

Done!

Could you please point me to the SNcosmo calls you are using to fit the light curves? Slide 21:

I actually went through as many as three iterations of fitting, but the calls were all essentially just this: https://github.com/jchiang87/desc_projects/blob/master/twinkles/refit_sncosmo.py#L33 The fits seem fairly sensitive to the redshift bounds, even when starting from the same initial value. If I set the bounds too wide (e.g., z in the range (0, 1)), I can get really poor fits. So I started out with z in (0.1, 0.4), and widened the bounds if the initial fit was pegged at either bound.

I don't understand the parameter n_sigma :

That's supposed to be an estimate of the quality of the fit in units of number of sigmas. I used an estimate that Phil gave me: n_sigma = sqrt(2chisq) - sqrt(2ndof-1). The vertical lines are upper bound cuts on n_sigma that I use to select different subsets of the data. The entries to the left of the blue line are the blue points in the subsequent plots.

The Delta T plot is interesting. While the difference is ~ milli-day :) and that is small I am surprised by the diagonal structure. This might have something to do with the guess_t0 moral prior, and would be interesting to figure out.

The Delta T units on the y-axis are actually supposed to be 1000s of days, so the diagonal guys are clearly mis-identifications.

I'll update my scripts to plot the other quantities you recommend. We can discuss further tomorrow.

[rbiswas4]

@jchiang87 Q1. You are reading in a results pickle. Is this able to access the reference light curves and truths I was supplying? Q2. Do you have access to the ra, dec values measured in this pickle file? think this is both an interesting (kind of where we want to be) discussion and a potentially long one. Does it make sense to actually have some document in the documentation to keep track of it. (I don't have a problem with using issues simultaneously, I just feel like it would be nice to have a flat document (without being broken up into replies) to collect ideas and thoughts.

[jchiang87]

@rbiswas4 The initial set of sncosmo fits just wrote out the fit parameters to an ascii file, and later I ran a version of this script to repackage those results into a DataFrame and also add the input truth parameters using your RefLightCurves.py code. I have the objectId in that DataFrame, so I've been resorting to db queries to get info like RA, Dec from the tables if I need them. All of the code to produce those plots is a bit disorganized and has been evolving. I plan to clean it up and reorganize it all into a sensible workflow so that I can rerun everything end-to-end, and then document it all. I hope I can find time to do that after Friday.

[rbiswas4]

@jchiang87 OK, I want to suggest some changes to the fitting procedure (though they will not take care of the huge outliers). I will try to put that in a branch with a few comments, and you can implement it appropriately based on the exact contents/interface. (FYI, when I said documentation I was not talking about code documentation, but the documentation directory of Twinkles, more as a planning document)

We can discuss more about how to understand this tomorrow. But I think it might help to start from a simpler task

• Assume that we have redshifts. I suggest this as a hypothetical exercise, but one could potentially get host spectroscopy to obtain decent redshifts for SN (though there are other problems to worry about there as well).
• Compare the size of uncertainties say of the distance modulus like quantity with fits for the reference light curves. I am trying to distinguish between the cases where the SN sampling is intrinsically poor (and would be reflected in large uncertainties to the reference light curve), and the cases where the problem is in some step between the reference light curve to obtaining the fluxes from the DM pipeline.

[rbiswas4]

@jchiang87

The suggested changes are in https://github.com/rbiswas4/desc_projects/blob/changeSNCosmoModel/twinkles/refit_sncosmo.py The place where they start is at https://github.com/rbiswas4/desc_projects/blob/changeSNCosmoModel/twinkles/refit_sncosmo.py

L19

I will make some more changes to the model over the next few days.

[jchiang87]

@rbiswas4 Thanks! Can you issue the pull-request?

[rbiswas4]

ok. I was trying not to because it includes some lines of pseudo-code rather than real code ( because I do not know the internals of results), but I will issue one if you find it more convenient.

[drphilmarshall]

Alright! The slides are coming along, but need finishing this afternoon. We have the following action items:

• [x] @wmwv Check Phil's intro slides to make sure we are representing SN properly. I emphasized the end-to-end nature of our accuracy test, and Jim will show his sncosmo SALT2 results - but it could be good to introduce all the sources of systematic error that we are trying to probe (DM object definition and measurement is just one?)
• [x] @tony-johnson You were going to write a high level conclusions slide to replace 18, which will be shuffled to the back of the pack for the Q&A
• [x] @drphilmarshall Sort out slides 7 and 8
• [x] @rbiswas4 Include slides on using CatSim, the sprinkled objects, and PhoSim please? Currently we don't say very much about what we are adding
• [x] @drphilmarshall Explain Run 1, 1.1 and 2, including selection of observation epochs; fit this in a more general introduction of Twinkles 1 and Twinkles 2, and the DC timeline.
• [x] @drphilmarshall Make an outlook slide based on the content in our Summer 2016 report, mention workflow system development, adjustment of over-population, image differencing.

Did I miss anything? Thanks!

[drphilmarshall]

Thanks all for your input! @jchiang87 @tony-johnson I put in some spacer slides, to help with our transitions tomorrow - see what you think. More comments welcome!

[jbkalmbach]

Lensed AGN image added to slide 33.

[drphilmarshall]

Cool, thanks @jbkalmbach! :-)

On Fri, Jul 8, 2016 at 1:35 AM, Bryce Kalmbach notifications@github.com wrote:

Lensed AGN image added to slide 33.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/DarkEnergyScienceCollaboration/Twinkles/issues/270#issuecomment-231305509, or mute the thread https://github.com/notifications/unsubscribe/AArY9zBtoKVyg6PoC_wKL3V4G2Vl0ZEvks5qTgvogaJpZM4JCm_N .

[drphilmarshall]

I think this talk went well - closing out!