search

LSSTDESC / pz_blend

impact of blending on photo-zs using DC2 truth catalogs and image catalogs
BSD 3-Clause "New" or "Revised" License
2 stars 1 forks source link

stats #10

Open ih64 opened 4 years ago

ih64 commented 4 years ago

write something that will dump out all the statistics we could be interested in ks test for PIT values ks test for empirical specz and stacked pz others?

sschmidt23 commented 4 years ago

moments for the stacked p(z)?

CDE loss statistic? (see the PZDC1 paper)

ih64 commented 4 years ago

i've got the moments coming up in a different PR, I'll get started on a PR for the CDE loss statistic too

sschmidt23 commented 4 years ago

The CDE loss is actually fairly simple to compute, see here: https://github.com/LSSTDESC/PZDC1paper/blob/master/metric_scripts/cde_individual_metrics.py#L191

sschmidt23 commented 4 years ago

Also, while I constantly complain about the use of point estimates for science analysis, they can be handy for diagnostic purposes, so maybe we should add sigma_IQR, cat outlier fraction, and overall bias a la the PZDC1 paper. Again, there's already super simple code for this in the PZDC1paper repository, if I have time I'll try to port it in to this repo

ih64 commented 4 years ago

I like the idea of the sigma_iqr and catastrophic outlier fraction, it will allow us to quantitatively compare zz plots across the different matching cases

sschmidt23 commented 4 years ago

point estimates were covered in issue #21 and a new function was now merged.

enourbakhsh commented 4 years ago

I think you meant PR #21 which addresses issue #20. It might be useful to use this new function inside the plot_zz() method and draw two more lines on the zz plots.