Average of PSF's parameters !=0 across different exposures & GP interp

[PFLeget]

Hi,

I am working right now on atmospheric fit from DES Y1 images. By looking PSF’s parameters averages across 23 exposures, I notice (like @cpadavis ), that you can observed a systematic average, different from zeros and that vary across the FoV. Some of those average fluctuation are correlated to some CCD position. Other looks like rings pattern, which I guess is something known from SV and Y1 data and solved for Y3 with better astrometric solution.

Even if those problem disappear in Y3, I expect to have this same kind of problem (average across exposures !=0), and this is an issue for both Gaussian Process algorithms implemented within Piff. Indeed, the assumption of the GP is that the gaussian fluctuation are around an average function that can be a function of the coordinates (which is what I observed on Y1 data). So, not take into account this average will affect, the shape of the spatial correlation, the hyperparameters found and the quality of the interpolation.

This is not a big deal to treat that correctly, it just needs to subtract this average function determine across different exposure on a single exposure before to start the interpolation and to add it back when you get your GP interpolation.

I would like to implement this new features within both GP algorithm that are implemented within Piff. Basically to do that, I want to feed the GP class with a new argument that are a function of the coordinate and which return the average of the PSF’s parameters. However, I know that Piff should work also from a config files so my idea is to do something like that:

Using the dill package (which is pip installable and equivalent to pickle) you can wrap a function/class within a pickle files and after re-used the function by opening the pickle file. An example:

import dill 

class average:
    def __init__(self, coeff_x, coeff_y):
        self.coeff_x = coeff_x
        self.coeff_y = coeff_y

    def __call__(self, x, y):
        return self.coeff_x * x + self.coeff_y * y

mean = average(2, 3)
print mean(5, 2)
dill.dump(mean, open("test.pickle", mode="wb"))

and now from everywhere this is possible to re-used this function with coefficient 2 and 3:

import dill 
mean  = dill.load(open("test.pickle", mode="rb"))
print mean(5, 2)

I guess with this implementation, you just need to feed the config file with this pickle file and within the GP class, used directly the function implemented within the pickle. Do you think this is a good API to do it like that?

I am writing a new branch to implement this spatial average define in a pickle file. If you think, it should be done in an other way, comments are welcome. But I am convince at least, GP implementation should care about average across exposures because this is something needed to make GP interpolation work on real data.

Cheers, PF

[rmjarvis]

I'm confused why we need to write such a function to a file. Are you just concerned that the estimate of the average from a single ccd won't be accurate enough? Also, in the long run, we're probably going to want to use GP on full exposures, where the chip-to-chip variation will be handled by an optical component. Then the GP has the full exposure to estimate the average, which should be plenty accurate.

Anyway, something to discuss further on our Friday telecon.

[PFLeget]

I will present that at the Piff meeting tomorrow with plots, it will be more obvious about what I am worried.

But yes I am afraid (and this is what I saw on Y1 data from Chris) that the average on the CCD (even on the full exposure) is not enough to constraint average PSF’s parameters. After removing the optical part of the PSF, you can imagine that you still have systematic optics/electronic/astrometric residuals and that are within the atmospheric PSF’s parameters. In this case you should deal with it in the GP interpolation.

An analogy of that is when you are doing interpolation of SNIa light curve with gaussian process. If you are subtracting only the average flux on your light curve interpolation, your interpolation would be bad. The gaussian process is in this case around an average light curves, so you should remove this average light curves shape (which is time and wavelength dependent) during the gp interpolation in order to get a really optimal gp interpolation.

In this situation, the gaussian process of atmospheric PSF’s parameters is I guess around the systematic optics/electronic/astrometric residuals which are sky/pixel coordinate dependent. So it should be subtract during the gp interpolation (per ccd or on the full exposure).

[rmjarvis]

Closed via #70