Evaluate performance with different noise scenarios

[tobias-liaudat]

It would be interesting to evaluate the model estimation performance with different noise scenarios.

Provide some quantitative metrics about the performance.

Try:

• Different fixed SNRs. (ex. one fixed SNR for all the stars in the field)
• A field with stars with SNR in a certain range.

[tobias-liaudat]

This commit includes a Colab notebook that includes the basic estimation of a noiseless dataset.

It provides a nice starting point for the PSF model.

[tobias-liaudat]

Here I'm leaving some results on datasets with different SNR.

Description

The training of the model takes less than 4min in GPU. Right, now the OPD has dimension 256x256, the output PSF 64x64, the dataset has 140 training PSFs and 60 testing PSFs (these last PSFs are not using to estimate the model). We are using only 15 Zernike coefficients and the number of wavelength bins is set to 15.

I used the Adam optimiser (learning rate =1e-2), the loss function is only a l2 norm over the reconstructions. The batch size is of 16 and the number of epochs 20.

The notebook to reproduce the results is the one pushed in this commit.

Examples of the different PSFs looks like:

Result table

The result in terms of RMSE is presented in the following table. The second column describes the pixel RMSE computed on the training dataset while the third column on the testing dataset.

The first row shows the training without added noise. The varying SNR experience shows a dataset where the SNR of each observed star is drawn from a uniformly distributed random variable in the range [10, 70].

SNR	train RMSE	test RMSE
-	1.7000e-04	1.6895e-04
10	1.4269e-04	1.4094e-04
20	1.9035e-04	1.8169e-04
30	2.0591e-04	2.0303e-04
50	2.2996e-04	2.3486e-04
70	1.9754e-04	1.9153e-04
varying	1.6357e-04	1.5849e-04

Example for SNR 10

Here I leave some example images from the testing dataset of SNR 10.

Loss function

PSF recovery in sqrt() domain

Matrix coefficient estimation

Each line represents a Zernike coefficient order and each column represents one of the monomial coefficient for the polynomial variations.

[tobias-liaudat]

Varying SNR example

I'm adding here another example with some supplementary plots.

Seems that the Zernike coefficients are estimated pretty well in the noisy scenario.

Loss function

PSF recovery in sqrt() domain

Matrix Coefficient estimation

Each line represents a Zernike coefficient order and each column represents one of the monomial coefficient for the polynomial variations.

Zernike coefficient estimation

Each line represents a recovered PSF and each column an order of the Zernike Polynomial.

[tobias-liaudat]

I will close this issue since studies with different noise scenarios are already being made.