Recommend max star brightness in DC1 runs

[richardxdubois]

From the DC1 planning meeting (2016-09-22), the idea was to look at removing stars brighter than 10th (TBD) magnitude.

Email trail from Seth:

Not sure whether this belongs here, but I've started a Confluence page looking at the influence of bright stars on the CPU times for the Twinkles runs, based on Tom's current PhoSim-deep-pre3 runs, and phosim trimcat (i.e., input source list) files that Jim generated for the Opsim visit Tom is running (1668469). The giant instance catalog for the FOV gets divided up into 189 trimcat files, one per CCD. Evaluating total CPU times is tricky for these jobs with checkpointing (Tom can explain). So far I've looked only at the stellar content of the phosim jobs that have been running for weeks. Formerly such long times were not possible on the batch farm, which has a 120-CPU hour limit, but the checkpointing has effectively enabled it.

The bottom line so far is that the brightest few stars (two have magnitudes <7 in the instance catalog) have a huge influence on the CPU time requirements, not just at their locations but over a fairly large area.

Once we've worked out CPU time tallies, then we could consider out what magnitude limits might need to be imposed in the instance catalogs for the next Twinkles run.

[cwwalter]

BTW, would we remove the stars, or just reassign their magnitude to the maximum allowed?

[jchiang87]

The CatSim code can apply a magnitude constraint in the query for objects. See here and where we call it from Twinkles here and here. Using that sort of code, I would expect that the objects brighter than the constraint are simply removed. We could rewrite the magnitudes in the instance catalogs "by hand", but that would make it inconsistent with the true values in the CatSim object tables. @danielsf Any comments?

[fjaviersanchez]

Stars from the LSS point of view are quite important since, usually you throw out objects that are really close to super bright objects and you want to know how close to these stars you can push your mask before being able to detect some effect. On the other hand, relatively bright stars under poor seeing conditions (even with good seeing) can be misidentified as galaxies, and this is an important issue in LSS (I think there is no star/galaxy classifier in DM output?). My guess is that for DC1 if we don't have any realistic photo-z code we won't need these super bright objects, so I guess that from r>16-17 (this was the saturation limit right?) on we should be fine but we will definitely need them in the future. What we actually need is a realistic spatial distribution of stars, and this is already in CatSim. @slosar @egawiser What do you think?

[sethdigel]

Somehow I missed this issue when it was posted. Some relevant information on CPU times for simulating bright stars in the field of view is in the (wide-ranging) thread on Twinkles issue 315, and in Confluence postings Phosim CPU time in PhoSim_deep-pre2 runs and Bright Sources in Twinkles Fields. The bottom line is that for a realistic distribution of stars, the brightest one or two stars in a typical visit would be about magnitude 7, and these one or two stars would typically dominate the CPU time requirements for phosim to simulate the visit. In our simulation workflow, each CCD in the FOV is simulated with a separate run of phosim (in parallel on the batch farm). The big CPU time goes into simulating the tails of the response, typically one or two CCDs over from the one that has the actual star. These simulations take a long time because phosim cannot use various efficiencies that it can take advantage of for simulating the core. The number of CCDs that have the star included in their trimmed version of the instance catalog goes essentially as the square of the photon flux, so the overall CPU time gets out of hand fairly quickly for brighter stars.

The influence of bright stars stars on the overall CPU time starts to be noticeable at magnitude ~10 (see the plots at the 2nd link above), for visits for which the overall sky brightness is low. By magnitude 8, the stars start getting assigned to the trim catalogs for multiple CCDs, and the simulations for the 'outlier' CCDs go particularly slowly.

One potential alternative to removing bright stars would be to replace them with multiple fainter stars at the same location (like one 7th magnitude star could become 10 magnitude 9.5 stars). This would result in little impact on the overall CPU time for the simulation (relative to just omitting these stars). The bright stars would be realistically simulated in their 'home' CCDs. What would be missing is the tails of the response (multiple reflections and scattered light) more than ~1 CCD away from the bright stars, where the flux from the star (spread over the CCD) would typically be a small fraction of the sky background. It would be a tradeoff, but the simulated data would be closer to what we'd expect from realistic fields.

To get a sense of the number of bright stars we are talking about, I looked at the Bahcall & Soneira (1981) model. Based on V band magnitude, at the NGP the density of stars is 1.5 deg-2 for mag < 9 and 3.5 deg-2 for mag <10. So each ~10 deg^2 visit would have about 35 stars at magnitude 10 or brighter.

[jchiang87]

Note that replacing a bright star with multiple fainter ones would effectively disable the optimization in phosim to "bunch" photons from bright stars, since objects are simulated one at a time. I think the computational savings of this bunching can be large if it is known ahead of time that the star will saturate the charge wells of the pixels to which it contributes. I'm guessing that the halos from very bright stars cost more in overall cpu time than the tradeoff from disabling this bunching, but that tradeoff hasn't been assessed quantitatively, as far as I know.

[sethdigel]

We have some idea from Tom's PhoSim_deep-pre2 run about the benefits from bunching, and the costs of simulating the halos.

The plot below shows the CPU times for simulating individual CCDs vs the magnitude of the brightest star in the trimcat file for that CCD.

The brighter the star, the more trimcat files it ends up in, because phosim wants to simulate its tails in neighboring CCDs. The two brightest stars (one a little above mag 7 and one a little below) ended up in about 25 trimcat files. In the plot, the CPU times for CCDs containing these stars appear as vertical arrangements of dots (many piled up at the 120 CPU hour limit for the SLAC batch farm). The lowest points of those columns show the time needed to simulate the CCD that actually contained the star. Eyeballing an average (and subtracting the ~20 ks asymptotic time for simulating a field with only very faint stars), it would have taken about 80 ks to simulate a magnitude 7 star in the CCD that contained it. For a magnitude 9.5 star, 10 times fainter, the incremental CPU time was about 40 ks, only maybe 2 times faster. (Some of the scatter in the plots comes from the batch farm having hosts of different speeds.) The big savings of CPU time would come from the fainter stars not being assigned to the trim catalogs of many nearby CCDs. A magnitude 9.5 star would have a 'buffer' size of about 1000 pixels in phosim and could be assigned to at most 3 other CCDs, and typically fewer than that. Also, photon bunching is effectively disabled for simulating the tails, so the CPU time requirements for the outlier CCDs scale more like the numbers of photons, which would be the same. The bottom line is that the columns of 25 long-running jobs in the plot would be replaced with at most 4 points, one of them high up and the other few at the low end of the range.

For stars fainter than about magnitude 8.5 it doesn't look like there'd be much if any net CPU time benefit from subdividing them into fainter stars, and I'd probably vote for having all subdivided stars no fainter than about magnitude 9.5.

[dkirkby]

This was never going to be a high fidelity sample (e.g., because of the galaxy tiling) so I would be tend to be pragmatic about sacrificing realism in order to push all the jobs through. For example, the brightest stars could be removed from the simulation, but keeping track of their locations and magnitudes so that they can be added to the mask used for later LSS analysis. An appropriate mask could be derived from a special purpose PhoSim run(s) with only a single bright star.

[dkirkby]

On the subject of PhoSim timing, what would the plot above look like with bunching turned off? Are there any parameters to tune the bunching? I guess you would want to bunch more photons into a single ray for brighter stars, in order to level off the CPU time / star, but I don't see that happening in this plot.

Do you know what dominates the non-linear growth of CPU time with number of photons from a bright star? Are there any config options to turn off or speed up the relevant processes?

[johnrpeterson]

On Oct 7, 2016, at 10:15 AM, David Kirkby notifications@github.com<mailto:notifications@github.com> wrote:

On the subject of PhoSim timing, what would the plot above look like with bunching turned off? Are there any parameters to tune the bunching? I guess you would want to bunch more photons into a single ray for brighter stars, in order to level off the CPU time / star, but I don't see that happening in this plot.

Do you know what dominates the non-linear growth of CPU time with number of photons from a bright star? Are there any config options to turn off or speed up the relevant processes?

its not non-linear, its sub-linear. if you convert magnitudes into number of photons then you will find it is optimizing, and simulating the bright stars faster than all the faint object in a per photon sense.

curiously, most of the photons in the Universe come from ~10 mag objects so there are a lot of photons there.

as i’ve said on other threads too, we have simulated 0 magnitude stars with phosim, so this is all doable. it just depends on your cluster set up of whether this is a headache or not.

note that also, the slac running of these twinkles jobs was run in a rather strange way (the “-s looping”) causing the parallelization to only be partially working that i didn’t know about until recently. we are fixing this by expaining how to do this better for the future.

john

[fjaviersanchez]

I think that for LSS, at least for this first run, we want the jobs to finish and see whether we have the tools to analyze this or not. Maybe the best is to forget about these super-bright sources for the PhoSim runs, keep their location and magnitude, and if there is time/if it makes sense to do this, we could run GalSim on top to generate the images of these bright stars, even though they are not going to have most of the effects that we want to see, but at least we will have a sense of how big these objects are in the images, and how much area we lose.

[johnrpeterson]

all-

you can include bright stars to arbitrary magnitude in phosim. i think this point keeps getting lost on multiple threads.

here is a 0th magnitude through phosim at the link below. we're happy to run these at purdue if necessary.

https://www.dropbox.com/s/q94l6mc5eg8eiuq/PastedGraphic-1.pdf?dl=0

john

On Oct 7, 2016, at 2:03 PM, fjaviersanchez notifications@github.com<mailto:notifications@github.com> wrote:

I think that for LSS, at least for this first run, we want the jobs to finish and see whether we have the tools to analyze this or not. Maybe the best is to forget about these super-bright sources for the PhoSim runs, keep their location and magnitude, and if there is time/if it makes sense to do this, we could run GalSim on top to generate the images of these bright stars, even though they are not going to have most of the effects that we want to see, but at least we will have a sense of how big these objects are in the images, and how much area we lose.

[dkirkby]

That's a great image @johnrpeterson! Do you know roughly what fraction of the time is spent calculating the main contributions to this image, which I guess are:

• tracing primary photons through the atmosphere
• diffraction in the telescope pupil
• charge bleeding and other sensor effects
• tracing secondary reflected photons

[slosar]

My take on this is to assign every star brigher than 10th mag, 10th mag, but it would be nice to have some very small subarea with no compromises on stars just to come differential measurement of what happens after DM chews the shit.

[cwwalter]

Hi All,

It sounds like we a consensus that we at least need to record where the bright stars were so we can produce masks. But, there are two approaches: just removing them (which can be done in PhoSim with an option) or limiting them to a maximum brightness. Can we converge on what is needed on this in particular?

[slosar]

Limit them to maximum brightness, since given the plot above, they don't really matter until brighter than 10.... But again, a region around one bright bastard should be simulated with maxed to 10 and unmasked so we can calibrate what they really do.

[egawiser]

We discussed this on today's DESC-LSS telecon. Nobody raised any concerns about missing a bunch of bright stars; LSS will happily tolerate the correlated reduction in realism to make significant CPU savings. Our plan is: 1. Know positions of bright stars that were simulated at reduced brightness in DC1. 2. Request a separate small PhoSim run of a few such stars to measure their full impact in terms of incompleteness in the resulting galaxy catalog. 3. Use GalSim to model the impact of all bright stars in the DC1 field after calibrating GalSim output to match the PhoSim results in step 2. This will produce masks to apply at the eventual catalog level around the not-fully-simulated bright stars so that the impact on our galaxy catalog is realistic.

[cwwalter]

OK. Understood. Thanks team LSS!

So, this requires us to do something somewhat new and not achievable by a phosim option. Do we need a new issue to add limiting magnitudes and recording positions and original magnitudes in the pipeline?

[richardxdubois]

And it also means Seth's estimate of the CPU time savings will hold and we can assume 2.4M CPU-hrs to do DC1. I worked out we'd need 3.3k cores to turn it around in a month, so I'm suggesting we ask Debbie to secure 3k cores for two months.

[cwwalter]

Based on the decision in this issue, I am closing it and have created issue #22 to track the implementation of this feature. If the appropriate person could be assigned to that issue it would be great.