search

astrodeepnet / debvader

Deblending astrophysical sources with variational autoencoders
BSD 3-Clause "New" or "Revised" License
6 stars 3 forks source link

Metric discussion #80

Open aguinot opened 2 years ago

aguinot commented 2 years ago

Let's discuss the metric we want to use to assess the quality of the deblending. What I have in mind:

How to do it?
We can use SEP.
For the flux: Equivalent of FLUX_AUTO from SExtractor (or APERRTURE?)
For the size: There is a lot of choice KRON_RADIUS, FLUX_RADIUS, FWHM For the SNR: Galsim definition or ratio flux/flux_err?
For the pixel error: I don't know what is the best, MSE? Do we include it?

roucelle commented 2 years ago

There are already deblending metrics we should check out (P.Melchior for ex) - some common discussed in the blending WG ?

aguinot commented 2 years ago

I 100% agree, the first point I made in #79 is to check the litterature and we should clearly start we that. We can make an issue to reference some papers.

b-biswas commented 2 years ago

I am jotting down a few points for the next meeting:

ideas from BTK:

Ideas from Scarlet

Shape measurement metric

BastienArcelin commented 2 years ago

There are already deblending metrics we should check out (P.Melchior for ex) - some common discussed in the blending WG ?

Yes but this metric is more about how much galaxies are blended, and not really to assess the quality of the deblending.

BastienArcelin commented 2 years ago

I am jotting down a few points for the next meeting:

ideas from BTK:

  • Segmentation metrics (pixels values predicted above the noise threshold): IOU [Intersection Over Union of ground truth vs prediction]
  • Mean square residual
  • Peak Signal-to-Noise Ratio (PSNR)
  • Structural Similarity (SSIM)

Ideas from Scarlet

  • Total flux
  • Correlation coefficient for SED and morphologies (the metric used in the paper does not include the noise term, I would like to incorporate that too in the metric)

Except the last one you mentioned I do not see any metric about shape recovery. I think you should have at least one metric about this. Also, a metric about the time to process a field image could be interesting considering the large amount of data that will be generated by future surveys. I just saw that you mentioned that in #81 .

aguinot commented 2 years ago

Since we have the ongoing project with Thomas on the shear recovery I thought we could just refer to this study here since it requires a very particular and complex setup to be tested properly. I guess we could include the ellipticity or maybe just the a,b parameters (major-/minor-axis). I would like to avoid the confusion with weak lensing applications.

b-biswas commented 2 years ago

In the last BTK hack session, Ismael pointed out some other morphological metrics used in Galsim-Hub. Namely: