Characterize the issues with the PhoSim background.

[cwwalter]

At the SUNY meeting we discussed how feasible it would be to either try to fix the background level for at least some of the exposures by adding a flat background on top of the sky source retraced ones, or possibly to consider if the field could be used as a kind of DDF test. @egawiser also made some relevant comments on correlated noise (Eric could you mention that here?).

We decided that the first thing to do would be to characterize how bad the situation was. This means we need to understand:

• How much the depth varies over the entire target region (which is about four times the targeted area due to the rotation issue detailed in #36).

• Since the amount of background suppression depends on the local phoSim "altitude" line characterize how much variation there was in the background level.

@fjaviersanchez Did you say you were working on these?

[fjaviersanchez]

@cwwalter I am working on the first part. About the second part, I am not sure if I understand the second part well. Do you mean the background's variation in the coadds or the variation in the individual visits as a function of the altitude?

[cwwalter]

The second.

We don't know the function and we also don't know if it is mostly 1/8 or 1/2 etc etc..

[fjaviersanchez]

@cwwalter These are the plots with the sky-background that you were asking for:

Zooming in to see what's going on: It looks like it can be parametrized kind of easily and reverted to ~800 which is what it should be

[fjaviersanchez]

Also just for completeness the variation with azimuth (which looks to be none)

[fjaviersanchez]

Density in PhoSim coadds:

Depth in PhoSim coadds (it appears rotated because of HEALpix):

[cwwalter]

Is density number of visits? That seems to high...

[fjaviersanchez]

The density is just number of detected objects per bin in that histogram.

[fjaviersanchez]

Also some more plots:

Here I am making a 2D histogram of the difference of the measured-input magnitude as a function of the measured magnitude for all the stars (base_ClassificationExtendedness_values==0, detect_isPrimary==0) in 50 randomly selected (single) visits for PhoSim. I matched using a KDTree approach. I looked in circles of radius=2 arcsec around each detected source and selected the one that was the closest in magnitude (if there was any source available, if not then I flagged and rejected that source)

And above I am representing the difference in measured/true RA, DEC as a function of magnitude for the same stars as the previous plot.

[cwwalter]

Great. Thanks. Can you make the RA plot as the 1d histograms? It looks like PhoSim doesn't see this effect. This might be expected because if the way proper motions are put in and take out. @danielsf will need to comment.

[danielsf]

The PhoSim InstanceCatalogs should still include proper motion. We only meddle with precession, nutation, and aberration when transforming the PhoSim coordinate system . We leave proper motion in.

At least: that is how the code is supposed to work.

[danielsf]

How did we calculate the PhoSim truth catalogs?

[fjaviersanchez]

This is the 2D histogram (above I was using all stars that were matched within 1 pixel distance) here I am representing all of them:

[egawiser]

Going back to the background level, it would be helpful to plot background level versus moon distance and moon illumination fraction. I suspect that the excursions from the main locus are due to moon and that we would therefore want to keep those and just add noise to correct the "dark sky" background level that's below 800 up to ~800. That assumes that the additional moon part variation with altitude is correct, which is unclear, but you cannot subtract noise anyhow so this is probably all we could do.

[egawiser]

Now onto the topic of correlated noise: in real astronomical images, the background is non-uniform due to both to gradients in the true sky background level and contributions from the "wings" of the PSF of very bright stars and galaxies. Background subtraction removes most of this but leaves correlated noise. So modelling the noise as poisson i.e., uncorrelated from one CCD pixel to another, is inaccurate. For discussion of this, see S5.2 (noting Equation 4 and the Figure below) of the 2006 MUSYC paper along with references therein. For the figure below, note that N refers to the linear size of an aperture, which is why poisson noise scales with N and purely correlated noise scales with N^2; the actual behavior in a co-added image is intermediate between those two as shown by the circles.

[egawiser]

So what does this mean for correcting PhoSim simulations? We could measure how correlated the noise is, add uncorrelated background noise to bring it up to desired levels, and then re-measure how correlated the noise is, and note how unrealistic our fix seems to have made it. I have previously stated that I'm not worried about this for ImSim, where the initial background is probably less correlated than in real data, and we would only be adding a roughly equal amount of uncorrelated noise - I expect little change there in the noise correlation properties. But for PhoSim where our added noise would dominate in most single images, the change in noise correlation properties could be significant. I would think it still realistic enough for DC1 purposes, but it might be enough of a loss of realism that we'd need to note the effect in publications. Let me also point out that I'm not familiar with the specific background subtraction methods used by the DM Stack and have never tried to measure these noise correlations on simulated LSST images - that's an adventure awaiting us!

[egawiser]

A quick further note: I have IDL code written by Ryan Quadri that calculates the (sigma1,alpha, beta) values for sigma_N=sigma1 alpha N^beta for a given FITS image. You have to also provide a FITS segmentation map for that image, which it uses to avoid sampling the background level on detected objects. Something like this might already exist in the DM Stack, or it could easily be re-coded in Python etc.

[fjaviersanchez]

Following up on today's discussion I accumulated in an Nside=512 healpix map how many visits each pixel receives. The problem is that I only used the center of each visit and I didn't take into account the overlaps between visits but, this will give us a sparse version of the "correct" map:

The colorbar shows the number of visits per pixel. The central area has between 180 and 200 visits. I also produced the histogram showing the number of visits in each pixel:

As I said there's room for improvement and I'll ask how the stack tracks the overlaps/geometry of the visits for the coadding stage.

[fjaviersanchez]

We have extensively studied this for DC2 in https:/github.com/DC2_repo/issues/140 so I think this is solved.