Comparison of finite and automatic differentiation procedures.

[andrevitorelli]

This is an investigation of the difference between doing automatic differentiation and finite differences for computing the response matrix. We will have plots like this below, from Yahia-Cherif et al 2021.

...where in x we will look at different finite step sizes, and y the resulting relative difference between methods.

[andrevitorelli]

Comparing finite vs automatic differentiation for simple 2 points in each direction (1p,1m,2p,2m) - and on a single image - we get this:

[andrevitorelli]

Nb is here: https://github.com/CosmoStat/autometacal/blob/e5a43752ebc5ca05d4c1d2e8aa2db6d5de5a00a9/notebooks/metacal_compare_auto_finite.ipynb

[EiffL]

Fantastic, very nice! Yes this is what we want :-) But I was thinking about a plot of the R matrix than ellipticity. This plot is still very interesting, and if I understand correctly, you apply the inverse R matrix obtained with either form of metacal and you compare the "calibrated" ellipticities. So it's nice because it tells us how an error on the response matrix might translate into a calibration error.

So, two things:

• Can you make the same plot but for the components of the response matrix itself? (like R11 R22)

• Can you add a small shear on your input galaxy object? like g1=0.01, so that we have a non-zero baseline for the ellipticity?

[andrevitorelli]

Here:

[andrevitorelli]

Nothing changes with a slightly sheared 'galaxy' (g1=0.05): (neither with a highly sheared one (either g1 or g2 = 0.5), not depicted)

[andrevitorelli]

With additional noise, off-diagonal elements rise.

[EiffL]

So that's pretty interesting! We see a non-zero bias at 1e-2 in this case with noise for the diagonal terms of the response matrix :-)

Alright, so I would suggest the following to make plots for the paper:

• [ ] We could plot on the y-axis the relative error on R components, to have all 4 components scaled approximately in the same way.

• [x] We could add a vertical line at the usual choice of 0.01 for the step, to guide the eye

• [x] We could make this plot for like a few galaxies of different SNR and size, just to see if we can see an difference in behaviour in different situations, and/or if we see an apparent effect, whether it's by chance or typical (I'm thinking for instance of this plot above with noise). Then for the paper we keep only a few typical examples.

• [ ] The next question we will need to ask is whether this effect is significant when computing the mean response over a large sample of galaxies.

In any case, I think we can start by adding a section on this comparison in the paper @andrevitorelli :-)

[andrevitorelli]

I'm not sure on plotting relative residuals at the moment, because there are important differences in magnitudes between diagonal and off diagonal elements in different scenarios that are washed away in relative residuals. I think this is actually due to implementation problems, and after these are resolved, a final plot with relative residuals would make sense, as we're only interested in overall behaviour.

[andrevitorelli]

I am still trying to understand why the derivative differences are so stable between different images. This was created from the median, and CL68 and 95 regions of absolute differences between auto and finite differentiation over 100 galaxy images.

[andrevitorelli]

Next tasks down the road to close this issue:

• [x] Batching the finite differences function
• [x] Plots with different noise levels
• [x] Plots with different pixel scales

Bonus mission?

• [ ] Porting the ngmix/moments.py to get_ellipticities() and rerun.

[andrevitorelli]

Some results with decreasing noise levels (by 1 order of magnitude each row)

[andrevitorelli]

• [x] Derivatives of the images regarding to shear
• [x] plotting shear response over a distribution of galaxies with different noise levels

[andrevitorelli]

• [x] check different noise levels with the plateaus (with ODD pixel-sized stamp)
• [x] plot with float64
• [x] cpu vs gpu

[EiffL]

ok, so, what happens with float 64 and cpu vs gpu?

[andrevitorelli]

@EiffL with even number of pixels, the autodiff vs finite differences go on getting closer and up to the step size of 1e-5 - instead of rising back again (the relative differences between methods become as low as 1e-7). With odd number of pixels nothing changes, the plateau is there.