[SPRINT] Embed CMNN Estimator in a PZ MAF

[MelissaGraham]

Embed CMNN Estimator in a PZ MAF

Attempt to make the CMNN Estimator code auto-run as part of a MAF and produce photo-z statistics for evaluating OpSim runs.

Contacts: Melissa Graham Day/Time: Main communication channel: Slack GitHub repo: https://github.com/dirac-institute/CMNN_Photoz_Estimator

Goals and deliverable

1. Set up the CMNN Estimator to automatically simulate galaxy catalogs and estimate photo-z results for a given OpSim run.
2. Redesign photo-z statistical outputs to provide easily understandable yet informative summary metrics.

Resources and skills needed

An environment with the OpSim and MAF framework installed (SciServer?). People familiar with OpSim and MAF, and with photo-z statistical quality metrics.

Detailed description

[MelissaGraham]

Started to work in SciServer with an old MAF that returns the 5sigma depth in all filters for all HEALpix that are: (1) observed in all 6 filters; (2) have E(B-V)<0.2; and (3) have i-band depth > a specified limit (depends on year of survey).

Having trouble with this old MAF and so am seeking to use a newer one as the starting point.

Once I have a working MAF that can return the median 5-sigma depths for all "extragalactic" fields, it will just take a few cells more to import the CMNN Estimator and have it run automatically and produce photo-z statistics.

The question about which kinds of statistics are most useful remains TBD though.

[yoachim]

Do you mean "coadded 5-sigma depth" rather than "median 5-sigma depths"? Note that we have the ExgalM5 metric which will give you the coadded depth and apply dust extinction.

I would encourage you to not make any cuts to run on only extragalactic area (we don't have "fields"). Ideally the metric would show things improve when we use a footprint that avoids dust.

[rhiannonlynne]

On the other hand - I think applying a cut to discount area which is not suitable (such as having dust extinction which is too high) might be reasonable, if the photo-z metric by itself cannot properly account for errors introduced by a higher error due to uncertain dust extinction.

Like Peter said - there is an ExgalM5 metric which does what you want. There is also an ExgalM5_with_cuts metric that will add cuts consistent with what DESC is using elsewhere.

Presumably there is some summary metric that takes into account the total area where photo-z is successful?

Lynne

On Wed, Dec 2, 2020 at 1:49 PM Peter Yoachim notifications@github.com wrote:

Do you mean "coadded 5-sigma depth" rather than "median 5-sigma depths"? Note that we have the ExgalM5 metric which will give you the coadded depth and apply dust extinction.

I would encourage you to not make any cuts to run on only extragalactic area (we don't have "fields"). Ideally the metric would show things improve when we use a footprint that avoids dust.

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/LSSTDESC/SprintWeek2020/issues/51#issuecomment-737516896, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAOFBX6HNKN6TSALJHJACLLSS2Y6BANCNFSM4T77I5KQ .

--

Dr R. Lynne Jones Research Scientist @ University of Washington / Dirac Institute / Rubin Observatory she/hers

[MelissaGraham]

I mean the average of the coadded 5-sigma depths in each filter, where the average is over all "extragalactic fields", where an "extragalactic field" means one that would be used for cosmology: has coverage in all 6 filters, has MW E(B-V)<0.2, and has an i-band depth fainter than some minimum.

Right now I'm just using the same one that you two wrote for me a ways back, but I got it set up in the DataLab. Then I added to that Jupyter Notebook functionaly to input the depths in ugrizy into the CMNN Photo-z Estimator and simulate test and training set catalogs, estimate their photo-z, and make some summary plots.

That's not the same as a real "photo-z metric" and I cannot make, for example, all-sky maps of "photo-z quality" like other metrics can make sky maps of "numbers of transients" etc. That would be computationally very very heavy to do a full photo-z simulation at every HEALpix. Thus for now this "photo-z MAF" is more of a summary statistic for the average photo-z quality in cosmological extragalactic fields...

Still to do: set things up nicer, as the connection and running of the CMNN Estimator is still kludgey and stuff.

[yoachim]

The CMNN is clearly doing a lot of work. Is there speedup to be had in running it for a single fiducial galaxy at a single redshift? I'm thinking train it once, then just get the errors for a single redshift at each healpix.

[MelissaGraham]

Using some kind of contrived, minimal test-set of galaxies is a good idea. I just need to think about (and talk with cosmologists about) what kind of galaxies are the best canaries. Like maybe Xc intrinsic colors at Xz representative true redshifts for a test set of just Xc*Xz test-set galaxies...

One problem with using such a small test set, though, is that the CMNN Estimator has a random-selection component, in that the "photo-z" is assigned by randomly choosing a "color-matched" training-set galaxy. This can be reset to choosing the best "color-matched" training set galaxy, but this method reduces the influence of the quality of the photometry from the photo-z estimate. So I think to robustly estimate the "photo-z quality" one needs to either use many test galaxies, or make repeated photo-z estimates for a given test galaxy...

(Sorry all that might be pretty opaque, maybe more of a note-to-self...)

[yoachim]

I like starting with a single test redshift because my naive intuition is that photo-z performance should be fairly correlated (if we do better at z=0.5, I'm betting it's a similar improvement for z=0.6). Rather than best color-matched galaxy, can you just pull the top N best matching? Then you could compute a weighted mean and weighted standard deviation for the photo-z residuals.

The cosmologists can tell me I'm wrong, but I'd do a redish galaxy and a blueish galaxy at z=0.2, 0.5, 1.2.

[rhiannonlynne]

Is the problem that the photo-z quality depends in large part not on how well a single galaxy does, but how much scatter there is overall and what the fraction of catastrophic outliers are? How come there isn't a grid of coadded depths in different bands -> photo-z performance lookup table? (or does it not just depend on depth?).

On Thu, Dec 10, 2020 at 10:22 AM Peter Yoachim notifications@github.com wrote:

I like starting with a single test redshift because my naive intuition is that photo-z performance should be fairly correlated (if we do better at z=0.5, I'm betting it's a similar improvement for z=0.6). Rather than best color-matched galaxy, can you just pull the top N best matching? Then you could compute a weighted mean and weighted standard deviation for the photo-z residuals.

The cosmologists can tell me I'm wrong, but I'd do a redish galaxy and a blueish galaxy at z=0.2, 0.5, 1.2.

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/LSSTDESC/SprintWeek2020/issues/51#issuecomment-742704985, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAOFBX3OSJ4MRWHPU2L3BQ3SUEGWXANCNFSM4T77I5KQ .

--

Dr R. Lynne Jones Research Scientist @ University of Washington / Dirac Institute / Rubin Observatory she/hers

[MelissaGraham]

Yes, right now under DESC's guidance we're doing a "low z" (0.8-1.2) and a "high z" (2.2-2.6 I think) bin for the results. But I get the impression that even that is too many... potentially unavoidable.

Yes, the CMNN Estimator could use the weighted mean of the color-matched subset, but this isn't different from using a weighted random draw. It already uses the standard dev. of the subset as the photo-z error.

The problem is indeed that a single galaxy just can't represent the total scatter overall, and can not give us the fraction of outliers measure which is needed.

I did spend many months running hundreds of simulations on a grid of coadded depths in the six bands, calculating the metrics for each of these runs, and attempting to build an interpolator that would let any user input an arbitrary set of depths and be returned the interpolated photo-z quality metrics. It doesn't work. The six dimensions from the six filters is just too many, and the correlations between filter depth and photo-z quality too complicated. I mean, maybe I was doing it all wrong and someone else could make it work. But for now I've completely abandoned an interpolator. :(