Run 3 phoSim Workflow

[TomGlanzman]

This issue is intended to start a discussion on the elapsed time that will be needed for the next Twinkles cycle based on experience with the first Twinkles cycle.

Twinkles 1 was ~1250 visits sensors and took 1-2 weeks to finally conclude, with ~10% of jobs running out the clock.

Task: Twinkles-phoSimII

1227 total jobs (visits) 1124 (91.6%) completed successfully 103 ( 8.4%) ran out of time

Mean wallclock time for successful jobs = 1400 minutes (23.3 hours) with tails going out to 8,000 minutes (133.3 hours = 5.55 days), not including the jobs that ran out of time.

*\ Assume 2500 cores max (Fermi not busy)

*\ Assume 80% of visits complete within 3000 minutes (or 50 hours = ~2 days), the remaining 20% outliers to require 7 days.

*\ Assume visits are time-ordered in terms of processing dispatch (not optimized for any other reason)

To process 25,000 visits:

80% = 20,000 jobs. That is 8 full cycles of 2 days = 16 days.

The remaining 5000 jobs will be started, on average, in 625 job chunks every two days. Therefore, the last chunk will finish about 7 days after the next to last cycle, or (16-2)+7 = 21 days

Temporary work storage needed for 2500 running jobs is ~7 TB (~2.3 GB/running job) Permanent output storage is 4.8 TB (~200 MB/sensor/visit)

This is probably the most optimistic estimate imaginable: no start-up hiccups, smooth running, no extra-long outliers, no significant failure rate, no I/O overload requiring job throttling, etc.

More realistic: 1) trickling jobs to avoid a shock front of I/O load 2) start-up delays (validation, problem investigation, etc.) and pauses during the run 3) limiting the total simultaneous number of jobs 4) significant stragglers at the end due to rollbacks 5) competition of CPU resources from Fermi 6) lack of proper Pipeline support for checkpoints will cause confusion

Optimizations: 1) Move all phosim Input from wain032/25 to Lustre (Fermi scratch) 2) Tune jobs to ask for resources they will need (mem/swap) (which will likely reduce the number of simultaneously running jobs) 3) Use Seth's algorithm to better estimate expected number of checkpoints (to reduce startup I/O load)

Update

Here are all the issues (based on a list made by @TomGlanzman) that need to be resolved to get the Run 3 phoSim workflow operational, and hence form part of this Epic:

• [x] Understanding Phosim 3.5.2 vs. Phosim 3.4.2 results for Twinkles visits #288
• [x] Zero out off-diagonal elements of crosstalk matrix in segmentation.txt #330
• [x] Avoiding phoSim indexing failures in Run 3 #312
• [x] Prepend instance catalog generation to phosim pipeline task #313
• [x] Can we list a set of issues that lead to phosim crashes/long times? #314
• [ ] Implement a script to check instance catalog inputs from CatSim against PhoSim output #325
• [x] PhoSim "smoke test" #144
• [x] Turn off per amp images when running phosim for Run3 #339
• [x] update to phoSim instance catalog generation script #351
• [x] Tools to extract obsHistIDs from the OpSim db file #335

To check on these issues, click on the "Epic: Run 3 phoSim Workflow" label.

We also have the following separate but related Epics and sub-issues, that all have the same milestone (#10):

• Becoming able to generate phoSim instance catalogs at SLAC #320
  • Write an instance catalog generation task #305
• Twinkles 1 PhoSim Pipeline Review #323
  • Schedule the Pipeline Review #331
• Run 3 Science Inputs #321

[sethdigel]

In the release notes for v3.5.2 I don't see anything that looks obviously related to these features. Phosim v3.5.2 was also said to have an improved model for the sky brightness, predicting generally lower sky backgrounds. This did not seem to have a systematic effect on the CPU time requirements for the Twinkles runs, though. If we go with v3.4.2 we should be sure to specify the altitude angles of the observations correctly.

[jchiang87]

I was thinking along the same lines as Simon, i.e., keep using v3.4.x. As Seth notes re: the altitude snafu, this makes verifying the inputs vs outputs even more crucial for these runs.

[TomGlanzman]

3.5.3 does have a fix for internal checkpointing. But seeing as how that will not help us with long running jobs...

• Tom

[drphilmarshall]

I'm fine with going back to 3.4. But first we need to issue this problem on the phoSim repo - John should really reply there, so that the whole phoSim community can follow the thread. Seth, would you mind doing that, and re-posting your comparison image, please?

[sethdigel]

I've posted an issue on the phosim repository. If I've left out useful information please add comments.

[johnrpeterson]

its contamination (dust or debris) on the surface of the detector resulting in lower flux. if you use flats it will go away.

On Sep 30, 2016, at 2:01 PM, Seth Digel notifications@github.com<mailto:notifications@github.com> wrote:

I've posted an issuehttps://bitbucket.org/phosim/phosim_release/issues/14/artifacts-in-eimages-from-phosim-v352 on the phosim repository. If I've left out useful information please add comments.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/DarkEnergyScienceCollaboration/Twinkles/issues/315#issuecomment-250811632, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AJbT8nTFdBHpWUSoUHf55iCaDb5QwgWkks5qvU6IgaJpZM4J-buj.

———————————

John R. Peterson Assoc. Professor of Physics and Astronomy Department of Physics and Astronomy Purdue University 525 Northwestern Ave. West Lafayette, IN 47906 (765) 494-5193

[johnrpeterson]

also, if you want it off just do “contaminationmode 0” in the command file. but y'all should be using flats at this point…

On Sep 30, 2016, at 2:01 PM, Seth Digel notifications@github.com<mailto:notifications@github.com> wrote:

I've posted an issuehttps://bitbucket.org/phosim/phosim_release/issues/14/artifacts-in-eimages-from-phosim-v352 on the phosim repository. If I've left out useful information please add comments.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/DarkEnergyScienceCollaboration/Twinkles/issues/315#issuecomment-250811632, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AJbT8nTFdBHpWUSoUHf55iCaDb5QwgWkks5qvU6IgaJpZM4J-buj.

———————————

John R. Peterson Assoc. Professor of Physics and Astronomy Department of Physics and Astronomy Purdue University 525 Northwestern Ave. West Lafayette, IN 47906 (765) 494-5193

[TomGlanzman]

@SimonKrughoff I can provide 1 sec exposures for, say, the PhoSim-deep-pre2 (non-checkpointed) task, but given recent developments would that still be useful?

[SimonKrughoff]

@johnrpeterson ah o.k. I didn't see that in the release notes.

I'm still wondering if we want to just stick with v3.4 since we know it better and know how to make it do what we intend it to, but I'd like to get feedback from the rest of the team.

@TomGlanzman I guess it's not important to have the short integration exposures any more.

[jchiang87]

Unless we can use already existing flats, I don't think it's practical to generate them for Run3. This is something PhosimDeep will have to deal with so I think we should let it be sorted out in that context. Regarding 3.5.3 vs 3.4, I would suggest running a few visits with the contaminationmode 0 setting and compare to the corresponding visits in Run1 (maybe we even want to tweak the altitude in these test runs to be 89deg) and if they look ok, go with 3.5.3.

[drphilmarshall]

Agreed - let's pretend we will have uncontaminated chips! In Twinkles 2 we'll graduate from eimages, John. Thanks for catching this!

On Fri, Sep 30, 2016 at 2:00 PM, James Chiang notifications@github.com wrote:

Unless we can use already existing flats, I don't think it's practical to generate them for Run3. This is something PhosimDeep will have to deal with so I think we should let it be sorted out in that context. Regarding 3.5.3 vs 3.4, I would suggest running a few visits with the contaminationmode 0 setting and compare to the corresponding visits in Run1 (maybe we even want to tweak the altitude in these test runs to be 89deg) and if they look ok, go with 3.5.3.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/DarkEnergyScienceCollaboration/Twinkles/issues/315#issuecomment-250851128, or mute the thread https://github.com/notifications/unsubscribe-auth/AArY948FcPH24FVfWnwIHJ7IPv-h2IUcks5qvXh6gaJpZM4J-buj .

[TomGlanzman]

I've created a new task, Twinkles-phoSim-353 which is a clone of the Run1 task (Twinkles-phoSimII), but using phoSim v3.5.3 and adding in the "contaminationmode 0" command file directive.

@jchiang87 Do I simply modify the altitude in the instance catalog for a few visits?

[TomGlanzman]

For reference here is the first bit of the first instance catalog in Twinkles-phoSim-353, phosim_input_200.txt.gz:

Opsim_obshistid 200 SIM_SEED 10861 Unrefracted_RA 53.0091385 Unrefracted_Dec -27.4389488 Opsim_moonra 256.098498 Opsim_moondec -23.4579808 Opsim_rotskypos 256.415745 Opsim_filter 2 Opsim_rawseeing 0.74412 Opsim_sunalt -30.8094876 Opsim_moonalt -36.2916624 Opsim_dist2moon 124.437037 Opsim_moonphase 3.884229 Opsim_expmjd 59580.1257 Opsim_altitude 70.8431374 Opsim_azimuth 273.031979 exptime 30 airmass 1.058624 SIM_NSNAP 1 SIM_VISTIME 30 object 992887068677 53.0510624 -27.360073 26.7201543 starSED/phoSimMLT/lte027-3.0-0.0a+0.0.BT-Settl.spec.gz 0 0 0 0 0 0 point CCM 0.0236970962 3.1 none [...]

So is the altitude tweak to modify "Opsim_altitude" from 70.8431374 to 89.0?

[jchiang87]

So is the altitude tweak to modify "Opsim_altitude" from 70.8431374 to 89.0?

Yes, that's right.

[SimonKrughoff]

@jchiang87 We do not have flats for v3.4 and only 30 second flats for v3.5, so I don't think we have enough information to use the flats at the moment.

[TomGlanzman]

Five visits have been launched with v3.5.3 with the modifications above: http://srs.slac.stanford.edu/Pipeline-II/exp/LSST-DESC/task.jsp?task=41888396

[sethdigel]

The five visits that Tom launched ran fairly quickly in v3.5.2 (between 8 and ~18 CPU hours).

[TomGlanzman]

All five visits appear to have completed successfully.

Data are at SLAC in the /nfs/farm/g/desc/u1/Pipeline-tasks/Twinkles-phoSim-353/phosim_output directory tree.

[sethdigel]

Thanks, Tom. They took about the same CPU time as the Twinkles Run 1 (phosim v3.4.2) originals.

Looking at the first visit (corresponding to OpSim visit ID 200), I don't see any evidence of the dust (which is good). I do see some differences that may or may not be important. The new version has about 4x more pixels affected by cosmic rays, but still only a few hundred total (~400 vs. ~100 pixels, comparing the number of large positive pixels to the number of large negative pixels in a difference map).

The overall average number of electrons per pixel in the original is 834.4 vs. 862.9 in the new image. The image below compares 300x300 pixel regions in the old (left), new (middle), and the difference (new - old). The difference image is smoothed with a 2x2 boxcar and scaled to the range -50 to +50 electrons (after correcting for the difference in overall average). The star at the center (which is at x, y ~ 2026, 2083) is saturated (100,000 electrons) at the peak. The difference map has a slight dipole appearance for other bright stars as well.

[johnrpeterson]

no, the flats are co-added for longer exposure times. should be 300 seconds, i believe.

On Sep 30, 2016, at 5:47 PM, SimonKrughoff notifications@github.com<mailto:notifications@github.com> wrote:

@jchiang87https://github.com/jchiang87 We do not have flats for v3.4 and only 30 second flats for v3.5, so I don't think we have enough information to use the flats at the moment.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/DarkEnergyScienceCollaboration/Twinkles/issues/315#issuecomment-250860266, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AJbT8h9aDG3Dbw51nbVwSw5acjyu0mqHks5qvYODgaJpZM4J-buj.

———————————

John R. Peterson Assoc. Professor of Physics and Astronomy Department of Physics and Astronomy Purdue University 525 Northwestern Ave. West Lafayette, IN 47906 (765) 494-5193

[jchiang87]

I ran processEimage.py (v12_0) on these new v3.5.3 exposures using the configuration from #324 and compared some calexp results to those of the Run1.1 (v3.4.2) exposures:

PhoSim version: 3.5.3
visit   seeing  zeropoint  #CR  #CR pixels
  220   1.090   8.13e+12    13     88
  200   0.913   6.48e+12    24    272
  230   0.838   4.71e+12    22    170
  250   1.061   3.24e+12    28    222
  276   1.366   1.60e+12    56    422

PhoSim version: 3.4.2
visit   seeing  zeropoint  #CR  #CR pixels
  220   1.065   8.13e+12    22    208
  200   0.983   6.49e+12    22    199
  230   0.909   4.71e+12    14    126
  250   0.976   3.24e+12    39    220
  276   1.177   1.61e+12    27    217

These look consistent with one another, as far as I can tell.

FWIW, here are histograms comparing the distributions of source fluxes from icSrc for the two sets of runs: I'm not sure how to interpret these, but I don't see anything in particular to be concerned about. There are probably other tests that one can perform, but from my perspective, the images from v3.5.3 look as serviceable as those from v3.4.2.

[SimonKrughoff]

@johnrpeterson I think we want to create our own master flats. I'm surprised these are the equivalent of 300 sec since they only have ~6500 counts. They also have far more structure than I'd expect in a dome flat.

My recommendation is that we forego using these flats until we understand them better. We've spent a lot of time on flats in the past, and I'd rather not repeat that.

[johnrpeterson]

they are an average not a sum, that we do in order to save space.

we should be using flats at this stage. commissioning will be here soon.

john

On Oct 3, 2016, at 1:58 PM, SimonKrughoff notifications@github.com<mailto:notifications@github.com> wrote:

@johnrpetersonhttps://github.com/johnrpeterson I think we want to create our own master flats. I'm surprised these are the equivalent of 300 sec since they only have ~6500 counts. They also have far more structure than I'd expect in a dome flat.

My recommendation is that we forego using these flats until we understand them better. We've spent a lot of time on flats in the past, and I'd rather not repeat that.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/DarkEnergyScienceCollaboration/Twinkles/issues/315#issuecomment-251177746, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AJbT8lLWGsfTrC3zCNx0FM_uuT3xHfdTks5qwUJAgaJpZM4J-buj.

———————————

John R. Peterson Assoc. Professor of Physics and Astronomy Department of Physics and Astronomy Purdue University 525 Northwestern Ave. West Lafayette, IN 47906 (765) 494-5193

[SimonKrughoff]

@johnrpeterson I agree that it would be preferable to use flats, but I'm worried about doing that at this point because of the amount of time we've dedicated to flats past.

On the issues of these particular flats, I get more and more confused. Maybe getting a few questions answered would help.

1. How many are averaged? It seems like only two since the values are either integers or half way between integers.
2. Since they are averages, why are there still two snaps?
3. Are you saying you aren't willing to let us create our own master flats?
4. What motivates the massive amount of small scale structure in these?
5. Where is there documentation about how the flats are generated and what goes into them?

[johnrpeterson]

On Oct 3, 2016, at 2:17 PM, SimonKrughoff notifications@github.com<mailto:notifications@github.com> wrote:

@johnrpetersonhttps://github.com/johnrpeterson I agree that it would be preferable to use flats, but I'm worried about doing that at this point because of the amount of time we've dedicated to flats past.

not sure what you mean by this. the images have effects in them whether you correct using flats or not. so the measurements will be worse by definition, if you don’t use them.

i am still very worried about why there is not a DM calibration pipeline to use flats at this point. it seems everything is being done by hand. correct me if i’m wrong though. how would you use them?

On the issues of these particular flats, I get more and more confused. Maybe getting a few questions answered would help.

1. How many are averaged? It seems like only two since the values are either integers or half way between integers.
2. and they are median averaged.
3. Since they are averages, why are there still two snaps?

don’t know. you can still average those.

1. Are you saying you aren't willing to let us create our own master flats?

i said earlier we didn’t have public-accessible space to host them at the moment. nothing to do with “willingness"

1. What motivates the massive amount of small scale structure in these?

explain more??

1. Where is there documentation about how the flats are generated and what goes into them?

flat photons are not different than astrophysical photon other than optimizations. they have the default (realistic) settings on, so this could be a problem.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/DarkEnergyScienceCollaboration/Twinkles/issues/315#issuecomment-251182819, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AJbT8oeGsAzM-mwAJ7XdAbqby61nvk23ks5qwUargaJpZM4J-buj.

———————————

John R. Peterson Assoc. Professor of Physics and Astronomy Department of Physics and Astronomy Purdue University 525 Northwestern Ave. West Lafayette, IN 47906 (765) 494-5193

[SimonKrughoff]

@johnrpetersonhttps://github.com/johnrpeterson I agree that it would be preferable to use flats, but I'm worried about doing that at this point because of the amount of time we've dedicated to flats past.

not sure what you mean by this. the images have effects in them whether you correct using flats or not. so the measurements will be worse by definition, if you don’t use them.

We have spent a lot of time in the past trying to debug the flats. I'm afraid we are falling into that trap again because the flats don't look like any flat I've ever seen coming off an optical telescope. My preference would be to have a way to produce e-images that are perfectly flat fielded, so we don't have to worry about bugs in the flats.

i am still very worried about why there is not a DM calibration pipeline to use flats at this point. it seems everything is being done by hand. correct me if i’m wrong though. how would you use them?

That's not correct, we could apply flats trivially. In fact, we can make the master flats in a pipeline. I'd prefer to do that than to be handed a master flat because then I will know what went into the master frames.

1. How many are averaged? It seems like only two since the values are either integers or half way between integers.
2. and they are median averaged.

O.K. ten seems like it could be enough, but i'm still confused how there are only integer and half integer values in that flat.

Edit: I think Mario set me straight. Sorry for my confusion.

1. Are you saying you aren't willing to let us create our own master flats?

i said earlier we didn’t have public-accessible space to host them at the moment. nothing to do with “willingness"

I can come up with space to put them. Would that make it possible to share all the flats that go in?

1. What motivates the massive amount of small scale structure in these?

explain more??

Flats I'm used to seeing have much smoother light distributions except for a few dust motes. What motivated you to implement structure like this?

1. Where is there documentation about how the flats are generated and what goes into them?

flat photons are not different than astrophysical photon other than optimizations. they have the default (realistic) settings on, so this could be a problem.

It sounds as though you expect me to know where to look for this information. Assuming I don't can you give me a specific pointer where it describes both the optimizations and the realistic settings?

[johnrpeterson]

On Oct 4, 2016, at 12:19 PM, SimonKrughoff notifications@github.com<mailto:notifications@github.com> wrote:

@johnrpetersonhttps://github.com/johnrpetersonhttps://github.com/johnrpeterson I agree that it would be preferable to use flats, but I'm worried about doing that at this point because of the amount of time we've dedicated to flats past.

not sure what you mean by this. the images have effects in them whether you correct using flats or not. so the measurements will be worse by definition, if you don’t use them.

We have spent a lot of time in the past trying to debug the flats. I'm afraid we are falling into that trap again because the flats don't look like any flat I've ever seen coming off an optical telescope. My preference would be to have a way to produce e-images that are perfectly flat fielded, so we don't have to worry about bugs in the flats.

that doesnt make any sense. if there are bugs in the flats, then there are bugs in the eimages already. the two are self-consistent. so you might as well correct them.

i am still very worried about why there is not a DM calibration pipeline to use flats at this point. it seems everything is being done by hand. correct me if i’m wrong though. how would you use them?

That's not correct, we could apply flats trivially. In fact, we can make the master flats in a pipeline. I'd prefer to do that than to be handed a master flat because then I will know what went into the master frames.

but what are you doing about the non-multiplicative effects (tree rings, brighter fatter, etc.)?

1. How many are averaged? It seems like only two since the values are either integers or half way between integers.
2. and they are median averaged.

O.K. ten seems like it could be enough, but i'm still confused how there are only integer and half integer values in that flat.

when you have a median of an even number of things, it can be half integer, i believe

1. Are you saying you aren't willing to let us create our own master flats?

i said earlier we didn’t have public-accessible space to host them at the moment. nothing to do with “willingness"

I can come up with space to put them. Would that make it possible to share all the flats that go in?

yes

1. What motivates the massive amount of small scale structure in these?

explain more??

[image]https://cloud.githubusercontent.com/assets/945715/19082496/40bf279e-8a13-11e6-83f9-f8e8b7b68839.png

Flats I'm used to seeing have much smoother light distributions except for a few dust motes. What motivated you to implement structure like this?

dust and debris on detector.

1. Where is there documentation about how the flats are generated and what goes into them?

flat photons are not different than astrophysical photon other than optimizations. they have the default (realistic) settings on, so this could be a problem.

It sounds as though you expect me to know where to look for this information. Assuming I don't can you give me a specific pointer where it describes both the optimizations and the realistic settings?

realistic settings just means no commands in the command file. so this may be a problem to using them as is, was what i meant earlier. optimizations just mean the optimizations for the flat photons & background photons (simulating them in bunches).

so i mean you are thinking of flats as somehow different “products” of phosim, but it really is just running the photons through from a dome light instead of stars & galaxies.

john

You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/DarkEnergyScienceCollaboration/Twinkles/issues/315#issuecomment-251437363, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AJbT8vlA3UY9AscQYmndr5Bi7b9lry5yks5qwnyCgaJpZM4J-buj.

———————————

John R. Peterson Assoc. Professor of Physics and Astronomy Department of Physics and Astronomy Purdue University 525 Northwestern Ave. West Lafayette, IN 47906 (765) 494-5193

[SimonKrughoff]

We have spent a lot of time in the past trying to debug the flats. I'm afraid we are falling into that trap again because the flats don't look like any flat I've ever seen coming off an optical telescope. My preference would be to have a way to produce e-images that are perfectly flat fielded, so we don't have to worry about bugs in the flats. that doesnt make any sense. if there are bugs in the flats, then there are bugs in the eimages already. the two are self-consistent. so you might as well correct them.

It's not as simple as this. The last time we tried to unravel this, we found that there were artifacts in the flats that were not in the eimages. This was a pretty big time sink for me last time and I'm hesitant to do that again. If it's not possible to produce e-images that are perfectly flat fielded, we may have to go a different route.

That's not correct, we could apply flats trivially. In fact, we can make the master flats in a pipeline. I'd prefer to do that than to be handed a master flat because then I will know what went into the master frames. but what are you doing about the non-multiplicative effects (tree rings, brighter fatter, etc.)?

We can deal with these too, but don't have measurements for simulated images to do the correction at the moment.

I can come up with space to put them. Would that make it possible to share all the flats that go in? yes

How much space do you need? I think we have space at NCSA where you could put things.

Flats I'm used to seeing have much smoother light distributions except for a few dust motes. What motivated you to implement structure like this? dust and debris on detector.

This doesn't answer my question. What data, conversations, papers, etc. informed your decision to make the amount and morphology of this dust and debris as you did?

It sounds as though you expect me to know where to look for this information. Assuming I don't can you give me a specific pointer where it describes both the optimizations and the realistic settings? realistic settings just means no commands in the command file. so this may be a problem to using them as is, was what i meant earlier. optimizations just mean the optimizations for the flat photons & background photons (simulating them in bunches). so i mean you are thinking of flats as somehow different “products” of phosim, but it really is just running the photons through from a dome light instead of stars & galaxies.

Is there a pointer you can give me where this is all described in detail? Specifically, the optimizations and default effects enumerated.

[johnrpeterson]

On Oct 4, 2016, at 1:52 PM, SimonKrughoff notifications@github.com<mailto:notifications@github.com> wrote:

We have spent a lot of time in the past trying to debug the flats. I'm afraid we are falling into that trap again because the flats don't look like any flat I've ever seen coming off an optical telescope. My preference would be to have a way to produce e-images that are perfectly flat fielded, so we don't have to worry about bugs in the flats. that doesnt make any sense. if there are bugs in the flats, then there are bugs in the eimages already. the two are self-consistent. so you might as well correct them.

It's not as simple as this. The last time we tried to unravel this, we found that there were artifacts in the flats that were not in the eimages. This was a pretty big time sink for me last time and I'm hesitant to do that again.

please delegate it to someone then. for the good of the project, we need someone with some experience with calibration prior to real data. i think you are either referring to ancient versions of phosim.

That's not correct, we could apply flats trivially. In fact, we can make the master flats in a pipeline. I'd prefer to do that than to be handed a master flat because then I will know what went into the master frames. but what are you doing about the non-multiplicative effects (tree rings, brighter fatter, etc.)?

We can deal with these too, but don't have measurements for simulated images to do the correction at the moment.

this is intriguing. what do you mean “measurements for simulated images”? what kind of measurements does DM expect?

I can come up with space to put them. Would that make it possible to share all the flats that go in? yes

How much space do you need? I think we have space at NCSA where you could put things.

Flats I'm used to seeing have much smoother light distributions except for a few dust motes. What motivated you to implement structure like this? dust and debris on detector.

This doesn't answer my question. What data, conversations, papers, etc. informed your decision to make the amount and morphology of this dust and debris as you did?

there are project estimates for the contamination. v3.6 fixes a few of the details and adds internal scattering, but the total light loss in v3.5 should be realistic.

It sounds as though you expect me to know where to look for this information. Assuming I don't can you give me a specific pointer where it describes both the optimizations and the realistic settings? realistic settings just means no commands in the command file. so this may be a problem to using them as is, was what i meant earlier. optimizations just mean the optimizations for the flat photons & background photons (simulating them in bunches). so i mean you are thinking of flats as somehow different “products” of phosim, but it really is just running the photons through from a dome light instead of stars & galaxies.

Is there a pointer you can give me where this is all described in detail? Specifically, the optimizations and default effects enumerated.

optimizations described in the phosim paper (Peterson et al. 2015, ApJS 218, 14). i don’t think the algorithm has changed substantially since that was written (3 years ago), but have to look at it carefully.

default effects just means no command file options (i.e. all physics on)

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/DarkEnergyScienceCollaboration/Twinkles/issues/315#issuecomment-251462392, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AJbT8ivfD7acYbGDYxyeZPEapckoAtPcks5qwpJ0gaJpZM4J-buj.

———————————

John R. Peterson Assoc. Professor of Physics and Astronomy Department of Physics and Astronomy Purdue University 525 Northwestern Ave. West Lafayette, IN 47906 (765) 494-5193

[cwwalter]

Hi All... this is a meta comment.. but John, can you use the quote feature either in your email client or on github (select the text and then hit 'r')? It's really really hard to read the responses with everything at the same level. Thanks!

[SimonKrughoff]

It's not as simple as this. The last time we tried to unravel this, we found that there were artifacts in the flats that were not in the eimages. This was a pretty big time sink for me last time and I'm hesitant to do that again.

please delegate it to someone then. for the good of the project, we need someone with some experience with calibration prior to real data.

I don't have anyone to delegate debugging phosim flats to. We are gaining a lot of calibration experience, but this is a different thing.

but what are you doing about the non-multiplicative effects (tree rings, brighter fatter, etc.)?

We can deal with these too, but don't have measurements for simulated images to do the correction at the moment.

this is intriguing. what do you mean “measurements for simulated images”? what kind of measurements does DM expect?

We should talk about this in a different thread. In short, we need correlations measured from flats for the BF corrections. Tree rings have not been nailed down yet, so we could use advice as to what the best way to save those corrections is.

I can come up with space to put them. Would that make it possible to share all the flats that go in?

yes

How much space do you need? I think we have space at NCSA where you could put things.

How much space do you need? Is NCSA an acceptable solution?

there are project estimates for the contamination. v3.6 fixes a few of the details and adds internal scattering, but the total light loss in v3.5 should be realistic.

Where are the project estimates for contamination you are using as a requirement for phosim?

[sethdigel]

Where are the project estimates for contamination you are using as a requirement for phosim?

This may not be the source, but the Camera Contamination Control Plan (LCA-279) specifies that at the time of the Camera pre-ship review the maximum acceptable "surface coverage of particulates independent of particle size" is 0.6% (Sec. 10.1 CCD Cleanliness Specification). It goes on to say that "Cleaning of the CCD sensor surface is not factored into this cleanliness specification," by which I think it means that cleaning would be possible in principle but the plan is to have the whole assembly and test process clean enough to stay below this surface coverage.

[SimonKrughoff]

Wow @sethdigel! That is an outstanding reference. So, if I'm understanding John correctly that the spots in the flat are from particles on the actual detector surface, the v3.5 implementation of phosim overestimates the density of these by a factor of many.

[johnrpeterson]

no, that’s not correct. the throughput loss from dust on the detector in v3.5 is 0.25%. there are losses on the other surfaces too including condensation, but its not as easy to see in the flats.

john

On Oct 5, 2016, at 6:38 PM, SimonKrughoff notifications@github.com<mailto:notifications@github.com> wrote:

Wow @sethdigelhttps://github.com/sethdigel! That is an outstanding reference. So, if I'm understanding John correctly that the spots in the flat are from particles on the actual detector surface, the 3.5 implementation of phosim overestimates the density of these by a factor of many.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/DarkEnergyScienceCollaboration/Twinkles/issues/315#issuecomment-251820048, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AJbT8l9WrMMXFvOzw2nevlOjksRr31qrks5qxCbrgaJpZM4J-buj.

———————————

John R. Peterson Assoc. Professor of Physics and Astronomy Department of Physics and Astronomy Purdue University 525 Northwestern Ave. West Lafayette, IN 47906 (765) 494-5193

[SimonKrughoff]

no, that’s not correct. the throughput loss from dust on the detector in v3.5 is 0.25%. there are losses on the other surfaces too including condensation, but its not as easy to see in the flats.

@johnrpeterson can you please explain more. Looking at the flats, it appears that there is dust/debris that impacts virtually every pixel. The thing is that they don't look realistic to me and most people I've shown them to agree. Where did your requirements for the implementation of this come from?

If I were to make room available at NCSA for the flats, would that work for you? How much space would you need?

[johnrpeterson]

On Oct 6, 2016, at 10:16 AM, SimonKrughoff notifications@github.com<mailto:notifications@github.com> wrote:

no, that’s not correct. the throughput loss from dust on the detector in v3.5 is 0.25%. there are losses on the other surfaces too including condensation, but its not as easy to see in the flats.

@johnrpetersonhttps://github.com/johnrpeterson can you please explain more. Looking at the flats, it appears that there is dust/debris that impacts virtually every pixel. The thing is that they don't look realistic to me and most people I've shown them to agree. Where did your requirements for the implementation of this come from?

as i said, the morphology of what you are seeing in v3.5 is not perfectly realistic. we have some fixes to the scattering & absorption coming in v3.6

but the total throughput is 0.25% for the detector and different for other surfaces. that number is the total loss of light not the fraction of pixels because the dust does not block the light entirely as it depends on the dust size.

i have seen tons of real flats with strange structures in them due to debris.

If I were to make room available at NCSA for the flats, would that work for you? How much space would you need?

have to check. give me a day to get the number.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/DarkEnergyScienceCollaboration/Twinkles/issues/315#issuecomment-251975013, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AJbT8ia8hDch4jdolcuFOYmI5XaNtfWDks5qxQKrgaJpZM4J-buj.

———————————

John R. Peterson Assoc. Professor of Physics and Astronomy Department of Physics and Astronomy Purdue University 525 Northwestern Ave. West Lafayette, IN 47906 (765) 494-5193

[SimonKrughoff]

i have seen tons of real flats with strange structures in them due to debris.

There's no argument that there will be features due to debris. I'm saying that v3.5 flats look crazy. Not only do they not look like any flats I've seen from other optical telescopes, they don't look like any of the flats I've seen from the various test stands with LSST engineering and science grade sensors.

Unless there are specific reasons to make the flats look this way, this is effectively a bug in the flats and should be fixed before we try to use them in my opinion.

[SimonKrughoff]

but the total throughput is 0.25% for the detector and different for other surfaces. that number is the total loss of light not the fraction of pixels because the dust does not block the light entirely as it depends on the dust size.

This is tangential but related. I did a simulation with v3.5 with contaminationmode set to the default and then with contaminationmode 0. The difference was ~7%. That's much bigger than the 0.25% I was expecting from previous statements, but I don't understand all the things turning contamination off could do.

[johnrpeterson]

On Oct 6, 2016, at 1:15 PM, SimonKrughoff notifications@github.com<mailto:notifications@github.com> wrote:

i have seen tons of real flats with strange structures in them due to debris.

There's no argument that there will be features due to debris. I'm saying that v3.5 flats look crazy. Not only do they not look like any flats I've seen from other optical telescopes, they don't look like any of the flats I've seen from the various test stands with LSST engineering and science grade sensors.

yes, i said the morphology isn’t right (twice) in v3.5, due to scattering/absorption being incomplete (better in v3.6). but what you said about the level of light loss level is wrong. its within expectation.

Unless there are specific reasons to make the flats look this way, this is effectively a bug in the flats and should be fixed before we try to use them in my opinion.

its not a bug, its incomplete physics detail. and its better in v3.6.

john

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/DarkEnergyScienceCollaboration/Twinkles/issues/315#issuecomment-252028802, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AJbT8pK6zluh4f-SAOH5NgsTXHPix2dUks5qxSyxgaJpZM4J-buj.

———————————

John R. Peterson Assoc. Professor of Physics and Astronomy Department of Physics and Astronomy Purdue University 525 Northwestern Ave. West Lafayette, IN 47906 (765) 494-5193

[johnrpeterson]

that is the other surfaces (mirrors, lenses, etc.)

On Oct 6, 2016, at 1:18 PM, SimonKrughoff notifications@github.com<mailto:notifications@github.com> wrote:

but the total throughput is 0.25% for the detector and different for other surfaces. that number is the total loss of light not the fraction of pixels because the dust does not block the light entirely as it depends on the dust size.

This is tangential but related. I did a simulation with v3.5 with contaminationmode set to the default and then with contaminationmode 0. The difference was ~7%. That's much bigger than the 0.25% I was expecting from previous statements, but I don't understand all the things turning contamination off could do.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/DarkEnergyScienceCollaboration/Twinkles/issues/315#issuecomment-252029628, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AJbT8lVyLPCl9AokSU1NgHUOxid9hdb9ks5qxS1ugaJpZM4J-buj.

———————————

John R. Peterson Assoc. Professor of Physics and Astronomy Department of Physics and Astronomy Purdue University 525 Northwestern Ave. West Lafayette, IN 47906 (765) 494-5193

[SimonKrughoff]

its not a bug, its incomplete physics detail. and its better in v3.6.

O.K. so the recommendation is to wait until v3.6. Sounds good. I think v3.4 will serve for now.

[johnrpeterson]

but you can just use v3.5 and turn "contaminationmode 0” and that should be better than v3.4 for other reasons.

On Oct 6, 2016, at 1:38 PM, SimonKrughoff notifications@github.com<mailto:notifications@github.com> wrote:

its not a bug, its incomplete physics detail. and its better in v3.6.

O.K. so the recommendation is to wait until v3.6. Sounds good. I think v3.4 will serve for now.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/DarkEnergyScienceCollaboration/Twinkles/issues/315#issuecomment-252034843, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AJbT8hszCTXsfCjEGH19eG5uYebIIeEFks5qxTIGgaJpZM4J-buj.

———————————

John R. Peterson Assoc. Professor of Physics and Astronomy Department of Physics and Astronomy Purdue University 525 Northwestern Ave. West Lafayette, IN 47906 (765) 494-5193

[SimonKrughoff]

but you can just use v3.5 and turn "contaminationmode 0” and that should be better than v3.4 for other reasons.

Can you enumerate the reasons? Given the amount of time it takes to get to know each new version of phosim, I'm hesitant to move on from a version we understand.

[johnrpeterson]

Simon-

the choice of what version to you is the same as whether you update your operating system on your laptop. obviously newer is better 99% of the time but sometimes don’t want to introduce a new variable when you are busy.

but few points:

-new versions (X.Y) of phosim typically adds a few hundred revisions of corrections, validation, fixes, or features. you can see those crudely in the release notes. obviously depends on the application of whether any of that matters.

-the Twinkles team (including yourself) have been particularly active in filing phosim tickets. it is a little odd not to go to the new version, which addresses many of those.

-the future NERSC runs will be using version v3.6 or higher. so twinkles is getting rather behind here if the twinkles decides to stick with v3.4. but again its ok.

john

On Oct 6, 2016, at 1:58 PM, SimonKrughoff notifications@github.com<mailto:notifications@github.com> wrote:

but you can just use v3.5 and turn "contaminationmode 0” and that should be better than v3.4 for other reasons.

Can you enumerate the reasons? Given the amount of time it takes to get to know each new version of phosim, I'm hesitant to move on from a version we understand.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/DarkEnergyScienceCollaboration/Twinkles/issues/315#issuecomment-252040406, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AJbT8iKq1Zwhboni5ihALMcGuTKtVbxHks5qxTbbgaJpZM4J-buj.

———————————

John R. Peterson Assoc. Professor of Physics and Astronomy Department of Physics and Astronomy Purdue University 525 Northwestern Ave. West Lafayette, IN 47906 (765) 494-5193

[SimonKrughoff]

The problem is that new versions don't actually fix the things we expected them to fix. Specifically, v3.5 doesn't fix the cloud model or the checkpointing that we hoped it would. The sky brightness model is a little better, but the runtime enhancements were not as advertised (as far as I can tell). In addition, the flats are much worse (in terms of being unrealistic looking) than they were before.

Without a vetting process for the consumers of phosim to sign off on new versions, it's much harder to assess the benefit of moving to a new version than you make it out to be.

[johnrpeterson]

simon,

the sign off process is that we tag patches (vX.Y.Z) between the releases (vX.Y) to address specific concerns of the community. this is the vetting process. whatever concern you or anyone else has we can certainly address in patches at any point and turn around as fast as possible. please, please file tickets.

the cloud distribution is completely changed, so i don’t know what you mean by that. there is no open ticket on this.

i’m also not sure what you mean by checkpointing. we just started talking about how to do this at slac this summer at oxford, but the calculus on this is going to change for NERSC anyways with multi-threading. there is an open ticket about the centroid files about this, so maybe that is what you mean.

john

On Oct 7, 2016, at 11:10 AM, SimonKrughoff notifications@github.com<mailto:notifications@github.com> wrote:

The problem is that new versions don't actually fix the things we expected them to fix. Specifically, v3.5 doesn't fix the cloud model or the checkpointing that we hoped it would. The sky brightness model is a little better, but the runtime enhancements were not as advertised (as far as I can tell). In addition, the flats are much worse (in terms of being unrealistic looking) than they were before.

Without a vetting process for the consumers of phosim to sign off on new versions, it's much harder to assess the benefit of moving to a new version than you make it out to be.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/DarkEnergyScienceCollaboration/Twinkles/issues/315#issuecomment-252278360, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AJbT8qUHZICJUp-DW0hrQM6bhSKU5QBeks5qxmDigaJpZM4J-buj.

———————————

John R. Peterson Assoc. Professor of Physics and Astronomy Department of Physics and Astronomy Purdue University 525 Northwestern Ave. West Lafayette, IN 47906 (765) 494-5193

[TomGlanzman]

Hi John,

There are two outstanding bitbucket issues about the internal phoSim checkpointing:

11 - invalid centroid file

12 - the need avoid putting >90% of the processing time into a single checkpoint

• Tom

On 10/07/2016 08:28 AM, johnrpeterson wrote:

simon,

the sign off process is that we tag patches (vX.Y.Z) between the releases (vX.Y) to address specific concerns of the community. this is the vetting process. whatever concern you or anyone else has we can certainly address in patches at any point and turn around as fast as possible. please, please file tickets.

the cloud distribution is completely changed, so i don’t know what you mean by that. there is no open ticket on this.

i’m also not sure what you mean by checkpointing. we just started talking about how to do this at slac this summer at oxford, but the calculus on this is going to change for NERSC anyways with multi-threading. there is an open ticket about the centroid files about this, so maybe that is what you mean.

john

On Oct 7, 2016, at 11:10 AM, SimonKrughoff notifications@github.com<mailto:notifications@github.com> wrote:

The problem is that new versions don't actually fix the things we expected them to fix. Specifically, v3.5 doesn't fix the cloud model or the checkpointing that we hoped it would. The sky brightness model is a little better, but the runtime enhancements were not as advertised (as far as I can tell). In addition, the flats are much worse (in terms of being unrealistic looking) than they were before.

Without a vetting process for the consumers of phosim to sign off on new versions, it's much harder to assess the benefit of moving to a new version than you make it out to be.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/DarkEnergyScienceCollaboration/Twinkles/issues/315#issuecomment-252278360, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AJbT8qUHZICJUp-DW0hrQM6bhSKU5QBeks5qxmDigaJpZM4J-buj.

———————————

John R. Peterson Assoc. Professor of Physics and Astronomy Department of Physics and Astronomy Purdue University 525 Northwestern Ave. West Lafayette, IN 47906 (765) 494-5193

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/DarkEnergyScienceCollaboration/Twinkles/issues/315#issuecomment-252283056, or mute the thread https://github.com/notifications/unsubscribe-auth/AI_9RNYLgxUuNT7uHIgLfAULFYrhiN74ks5qxmUEgaJpZM4J-buj.

[SimonKrughoff]

the sign off process is that we tag patches (vX.Y.Z) between the releases (vX.Y) to address specific concerns of the community. this is the vetting process. whatever concern you or anyone else has we can certainly address in patches at any point and turn around as fast as possible. please, please file tickets.

O.K. Where would you prefer I file tickets? Should I do it in Jira or in bitbucket?

To be honest, my experience with getting tickets resolved has left a lot to be desired. I feel like there is no attempt by the phosim team to actually verify that the issue being ticketed is resolved to the satisfaction of the reporter. An example is: PHOSIM-11. I filed this ticket in 2014 and then in June of this year you closed the ticket without any comment other than saying you wouldn't support the feature I asked for.

Edit: To be fair, you did give a reason for not supporting it, but there was no dialog at all about how to resolve my fundamental issue that I want to be able to set the sky brightness normalization it the input file rather than the physics override file.

the cloud distribution is completely changed, so i don’t know what you mean by that. there is no open ticket on this.

This is another example of a case where there was no attempt to verify with the ticket reporter that the solution actually resolved the issue. It's true that you closed the ticket that was open, but there is no evidence that the fix improved the situation. In fact, in April I asked in PHOSIM-19 for some sort of verification that the cloud model was improved and that request was completely ignored.

i’m also not sure what you mean by checkpointing. we just started talking about how to do this at slac this summer at oxford, but the calculus on this is going to change for NERSC anyways with multi-threading. there is an open ticket about the centroid files about this, so maybe that is what you mean.

I was under the impression that checkpointing for HPC was a feature intended for v3.5, but according to issues filed by Tom on the bitbucket repo it seems like that isn't working yet. Apologies if I misinterpreted something and that the checkpointing improvements are scheduled for future release.

[johnrpeterson]

On Oct 7, 2016, at 1:10 PM, SimonKrughoff notifications@github.com<mailto:notifications@github.com> wrote:

O.K. Where would you prefer I file tickets? Should I do it in Jira or in bitbucket?

either is fine. please keep the content of the tickets technical/scientific as these are open to the world.

regarding your comments below:

• part of phosim-11 doesn’t make technical sense and fixing the variability was already implemented. we discussed this many times outside of Jira.
• phosim-19 is done and resolved
• checkpointing was already implemented years ago in v3.2 (we use it to make flats are purdue), so not related to v3.5. tom and i first talked about how slac should use it this July.

john

To be honest, my experience with getting tickets resolved has left a lot to be desired. I feel like there is no attempt by the phosim team to actually verify that the issue being ticketed is resolved to the satisfaction of the reporter. An example is: PHOSIM-11https://jira.lsstcorp.org/browse/PHOSIM-11?jql=project%20%3D%20PHOSIM. I filed this ticket in 2014 and then in June of this year you closed the ticket without any comment other than saying you wouldn't support the feature I asked for.

the cloud distribution is completely changed, so i don’t know what you mean by that. there is no open ticket on this.

This is another example of a case where there was no attempt to verify with the ticket reporter that the solution actually resolved the issue. It's true that you closed the ticket that was open, but there is no evidence that the fix improved the situation. In fact, in April I asked in PHOSIM-19https://jira.lsstcorp.org/browse/PHOSIM-19?jql=project%20%3D%20PHOSIM for some sort of verification that the cloud model was improved and that request was completely ignored.

i’m also not sure what you mean by checkpointing. we just started talking about how to do this at slac this summer at oxford, but the calculus on this is going to change for NERSC anyways with multi-threading. there is an open ticket about the centroid files about this, so maybe that is what you mean.

I was under the impression that checkpointing for HPC was a feature intended for v3.5, but according to issues filed by Tom on the bitbucket repo it seems like that isn't working yet. Apologies if I misinterpreted something and that the checkpointing improvements are scheduled for future release.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/DarkEnergyScienceCollaboration/Twinkles/issues/315#issuecomment-252308118, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AJbT8g3FQE4GgZtghQjh4mC_PQurATOnks5qxnzogaJpZM4J-buj.

———————————

John R. Peterson Assoc. Professor of Physics and Astronomy Department of Physics and Astronomy Purdue University 525 Northwestern Ave. West Lafayette, IN 47906 (765) 494-5193

[SimonKrughoff]

I think you are missing the point, since your response to my complaint is to simply reiterate the words that made me complain in the first place. I am telling you that I haven't been satisfied with the issue resolution process. Just telling me that something is done is not the same as producing a solution that I'm satisfied with.

In order to close the loop on issues, you need to confirm that the issue reporter agrees that the issue is resolved, or that they are satisfied with closing the issue without a resolution. That's one of the reasons the project does reviews of issues.

Specifically: In PHOSIM-11, I asked for a very specific technical functionality, and you said you wouldn't do it without any conversation whatsoever. That is not closing the loop.

In PHOSIM-19, you never responded to my request for verification that the cloud model is more correct than it used to be (and still haven't). Some tests I've done on my machine show that it is still very wrong.

Checkpointing still doesn't work for us. I don't know if you are continuing to work with Tom on this, but I hope so.

Your being dismissive of issues and ignoring comments and questions does not make the issue process seem very productive to me.