Develop background subtraction model

[awhoward]

This code would measure background in inter-order regions of the 2D image and fit a 2D polynomial to it.

Background light is currently visible in the continuum of L1 ThAr spectra.

[awhoward]

He have HIRES code (in IDL) that might serve as inspiration.

[bjfultn]

@cwang2016 haven't we already done this? If so, please close this issue.

[cwang2016]

@bjfultn @awhoward We had done the implementation for background extraction as described in the issues of https://github.com/California-Planet-Search/KPF-Pipeline/issues/300 Please help check that issue (item 2) and confirm that is the same issue. Thanks.

The caluclation is mainly based on the method provided by Ryan and order trace result.

[awhoward]

@cwang2016 -- subtraction of a dark is different from background subtraction. The former comes from detector issues while the latter comes from scattered light inside the spectrometer/camera.

[awhoward]

@xdumusque commented in Slack:

Just to give more input regarding Andrew’s task on investigating background correction. In the attached figure, you can see that for serval orders in KPF (green), the TH flux between lines is not zero and have some low frequency pattern. The blue and orange Th spectrum on the back are for HARPS-N where it is much more flat. This is generally due to background contamination and can be corrected for by measuring between orders the contamination, and then fitting a 2D spline on the raw images before extraction. I know that such background can induce systematics in the wavelength solution, either with TH lines, but also with LFC.

[awhoward]

The first step in modeling this will be using the output of Order Trace to find the pixels in the 2D image that are more than N pixels above or below extracted orders. We want to set N large enough so that we're not detecting the wings of the cross-dispersion profile of each order. This will take a little fine-tuning.

[awhoward]

Some updates based on a discussion with @bjfultn, @rrubenza, @xdumusque, @almoulla.

Here are some plots from Xavier.

We also discussed how @rrubenza developed a module to do just what is described above! However, we're still seeing background in the L1 files. We discussed how perhaps the parameters of the background subtraction routine aren't optimized. For example, the scales in of the 2D spline may not be tuned.

[awhoward]

The next step is that @bjfultn is doing to look at running the parameters, staring first by turning off that module to see what happens.

[rrubenza]

@awhoward the code Cindy mentioned does do a 2D background removal. See the background_subtraction function within https://github.com/California-Planet-Search/KPF-Pipeline/blob/master/modules/image_processing/src/alg.py

[awhoward]

Further discussion yesterday reflected one some areas of improvement and challenge for the background subtraction code.

First, it currently does not have any buffer pixels around the order mask. Scattering from the nearest order (within a few pixels) is likely a big effect.

Second, the orders are very close together in the bluest order of each chip, about 5 pixels for the bluest order pair, but the separation increases in the red of each order pair. The orderlets are separated by only 4 pixels. This will be difficult to overcome. A possible solution is to have a parameter that increases the mask height (up and down in cross-dispersion).

Third, the background subtraction module is currently not tested. We should do that. Using exposures with single fibers illuminated might help with testing.

[awhoward]

I made a 2-page diagram to show the spacing between the spectral orders and orderlets (slices) within a order. This is for the green CCD, but the red CCD is very similar. See attached. Orderlet diagram - 2D.pdf

[bjfultn]

@cwang2016 assigning this to you now to tune the background subtraction algorithm. I think that the new ability to expand/contract the order mask will help a lot here.

[awhoward]

I'm writing to note that this is becoming a significant problem. The current version is not working, I'm sorry to say. As an example, see the attached L1 spectrum of SKY during a recent observation. The spectrum has significant negative excursions (~ -500 e-) that show that the background subtraction method is not accurate. We need to worry about errors in the L1 spectra that are that size, which is huge!

Solving this problem is a prerequisite for implementing a sky subtraction model, which is critical for nighttime observations of faint stars that we've been doing regularly now.