Arc wavelength solution

This is an issue submitted by Hanae Inami at NOAO. The submission was via email but I am copying it here so we that we can all help with providing an answer.

Hello,

I'm working on MOSFIRE data using MosfireDRP ver.1.1 downloaded from GitHub. The data were taken in K-band during April 12-14, 2014.

I have a question regarding the wavelength fitting and wavelength calibration using Ne and Ar. The attached plot shows RMS vs. spatial pixels for a mask observed during the run. The values are taken from the standard output from running Wavelength.fit_lambda, i.e., each data point on the x-axis is "pXXXX" and y-axis is "Y.YY rms" from:

resid ang S?? @ pXXXX: Y.YY rms ?.?? mad [shift ?].

The orange vertical lines and numbers indicate the pixel locations of the slits and slit numbers (i.e., "S??" above).

The different colors of the data points are from wavelength fittings for different sets of lines:

White: Sky emission lines
        Wavelength.fit_lambda(maskname, band, obsfiles, obsfiles, waveops)
Red: Ne+Ar arc lamp
        Wavelength.fit_lambda(maskname, band, 'Ne.txt', 'Ne.txt', waveops, wavenames2='Ar.txt')
Blue: Ar only
        Wavelength.fit_lambda(maskname, band, 'Ar.txt', 'Ar.txt', waveops)
Green: Ne only
        Wavelength.fit_lambda(maskname, band, 'Ne.txt', 'Ne.txt', waveops)

Regarding the fits for the arc lamps, it seems that the fits are better when only Ne lines (green) are used than when both of Ne and Ar are combined (red). I'm not sure how Wavelength.fit_lambda combines Ne and Ar for fitting, but this makes me think that if I use all of the sky+Ar+Ne lines (white+red) for the wavelength calibration (Wavelength.apply_lambda_sky_and_arc), the results would be worse than when I only use sky+Ne (white+green). In fact, I have a mask (not the mask of the attached plot) which fails in wavelength calibration when I use sky+Ar+Ne; the final rectified data include NaNs in parts of the 2D spectra. However, this is fixed (i.e., NaNs are gone) when I only use sky+Ne for the wavelength calibration. So I wonder if it's good to use Ar as well as Ne, or should I just use Ne for the wavelength calibration?

I thought that including Ar would improve the fits, but it doesn't seem to be that way. I don't know how the DRP treats Ar and Ne together, but maybe it could be because the counts for the Ar lines are on average smaller than Ne (about a factor or 2 or more)? We took 2-3 frames for both Ar and Ne each, and I combined all of them each (with Wavelength.imcombine).

I was about to upload some files at the NOAO FTP site in case it's useful, but it seems that it's currently down. As soon as it's back online, I'll upload the files that include the standard output from running Wavelength.fit_lambda which I used to make the attached plot, and the combined sky, Ne, and Ar images at:

ftp://ftp.noao.edu/pub/inami/tmp/mosfire/

Thank you, Hanae

out_fit_lambda

Hi Hanae,

I personally only have experience with skyline + Neon. This always worked well for me. What is the result if you do sky + Neon?

Also, when you say that there are NaNs in the wavelength solution, how often do you see them? Are they clustered at any wavelengths? Does it affect a row or a column? etc.

Thanks! ~N

On Mon, Feb 16, 2015 at 10:24 PM, MosfireDRP notifications@github.com wrote:

This is an issue submitted by Hanae Inami at NOAO. The submission was via email but I am copying it here so we that we can all help with providing an answer.

Hello,

I'm working on MOSFIRE data using MosfireDRP ver.1.1 downloaded from GitHub. The data were taken in K-band during April 12-14, 2014.

I have a question regarding the wavelength fitting and wavelength calibration using Ne and Ar. The attached plot shows RMS vs. spatial pixels for a mask observed during the run. The values are taken from the standard output from running Wavelength.fit_lambda, i.e., each data point on the x-axis is "pXXXX" and y-axis is "Y.YY rms" from:

resid ang S?? @ pXXXX: Y.YY rms ?.?? mad [shift ?].

The orange vertical lines and numbers indicate the pixel locations of the slits and slit numbers (i.e., "S??" above).

The different colors of the data points are from wavelength fittings for different sets of lines:

White: Sky emission lines Wavelength.fit_lambda(maskname, band, obsfiles, obsfiles, waveops) Red: Ne+Ar arc lamp Wavelength.fit_lambda(maskname, band, 'Ne.txt', 'Ne.txt', waveops, wavenames2='Ar.txt') Blue: Ar only Wavelength.fit_lambda(maskname, band, 'Ar.txt', 'Ar.txt', waveops) Green: Ne only Wavelength.fit_lambda(maskname, band, 'Ne.txt', 'Ne.txt', waveops)

Regarding the fits for the arc lamps, it seems that the fits are better when only Ne lines (green) are used than when both of Ne and Ar are combined (red). I'm not sure how Wavelength.fit_lambda combines Ne and Ar for fitting, but this makes me think that if I use all of the sky+Ar+Ne lines (white+red) for the wavelength calibration (Wavelength.apply_lambda_sky_and_arc), the results would be worse than when I only use sky+Ne (white+green). In fact, I have a mask (not the mask of the attached plot) which fails in wavelength calibration when I use sky+Ar+Ne; the final rectified data include NaNs in parts of the 2D spectra. However, this is fixed (i.e., NaNs are gone) when I only use sky+Ne for the wavelength calibration. So I wonder if it's good to use Ar as well as Ne, or should I just use Ne for the wavelength calibration?

I thought that including Ar would improve the fits, but it doesn't seem to be that way. I don't know how the DRP treats Ar and Ne together, but maybe it could be because the counts for the Ar lines are on average smaller than Ne (about a factor or 2 or more)? We took 2-3 frames for both Ar and Ne each, and I combined all of them each (with Wavelength.imcombine).

I was about to upload some files at the NOAO FTP site in case it's useful, but it seems that it's currently down. As soon as it's back online, I'll upload the files that include the standard output from running Wavelength.fit_lambda which I used to make the attached plot, and the combined sky, Ne, and Ar images at:

ftp://ftp.noao.edu/pub/inami/tmp/mosfire/

Thank you, Hanae

[image: out_fit_lambda] https://cloud.githubusercontent.com/assets/9344283/6224070/c1df6840-b619-11e4-9e39-68921138285b.png

Reply to this email directly or view it on GitHub https://github.com/Mosfire-DataReductionPipeline/MosfireDRP/issues/13.

+1 831 704 6425

Hi all, I suspect that there is something going wrong with the automatic fitting of the Argon K-band arc spectra, most likely because its linelist needs to be trimmed from the default in Wavelength.py (it probably includes too many weak features that are not being located successfully).

 Having said this, there is no reason to include Argon in getting a good wavelength

solution in K band, and I think people are being misled by the Kdriver.py file which appears to default to trying to use both.

  Neon has the role of filling in between 2.3 and 2.4 microns, where

the sky has few/no good lines; argon adds very little compared to Neon in that wavelength range. I would recommend that we remove argon from the distributed version of Kdriver.py (I had done this on my local version as soon as I saw it had been added, on the grounds that it is not necessary). It is actually unclear to me what is being done by the code in trying to use two different lamp solutions to supplement the sky lines (note that this was not part of Wavelength.py in the old google code distribution, and so must have been added fairly recently).

 Chuck

On Feb 16, 2015, at 10:24 PM, MosfireDRP notifications@github.com wrote:

This is an issue submitted by Hanae Inami at NOAO. The submission was via email but I am copying it here so we that we can all help with providing an answer.

Hello,

I'm working on MOSFIRE data using MosfireDRP ver.1.1 downloaded from GitHub. The data were taken in K-band during April 12-14, 2014.

I have a question regarding the wavelength fitting and wavelength calibration using Ne and Ar. The attached plot shows RMS vs. spatial pixels for a mask observed during the run. The values are taken from the standard output from running Wavelength.fit_lambda, i.e., each data point on the x-axis is "pXXXX" and y-axis is "Y.YY rms" from:

resid ang S?? @ pXXXX: Y.YY rms ?.?? mad [shift ?]. The orange vertical lines and numbers indicate the pixel locations of the slits and slit numbers (i.e., "S??" above).

The different colors of the data points are from wavelength fittings for different sets of lines:

White: Sky emission lines Wavelength.fit_lambda(maskname, band, obsfiles, obsfiles, waveops) Red: Ne+Ar arc lamp Wavelength.fit_lambda(maskname, band, 'Ne.txt', 'Ne.txt', waveops, wavenames2='Ar.txt') Blue: Ar only Wavelength.fit_lambda(maskname, band, 'Ar.txt', 'Ar.txt', waveops) Green: Ne only Wavelength.fit_lambda(maskname, band, 'Ne.txt', 'Ne.txt', waveops) Regarding the fits for the arc lamps, it seems that the fits are better when only Ne lines (green) are used than when both of Ne and Ar are combined (red). I'm not sure how Wavelength.fit_lambda combines Ne and Ar for fitting, but this makes me think that if I use all of the sky+Ar+Ne lines (white+red) for the wavelength calibration (Wavelength.apply_lambda_sky_and_arc), the results would be worse than when I only use sky+Ne (white+green). In fact, I have a mask (not the mask of the attached plot) which fails in wavelength calibration when I use sky+Ar+Ne; the final rectified data include NaNs in parts of the 2D spectra. However, this is fixed (i.e., NaNs are gone) when I only use sky+Ne for the wavelength calibration. So I wonder if it's good to use Ar as well as Ne, or should I just use Ne for the wavelength calibration?

I thought that including Ar would improve the fits, but it doesn't seem to be that way. I don't know how the DRP treats Ar and Ne together, but maybe it could be because the counts for the Ar lines are on average smaller than Ne (about a factor or 2 or more)? We took 2-3 frames for both Ar and Ne each, and I combined all of them each (with Wavelength.imcombine).

I was about to upload some files at the NOAO FTP site in case it's useful, but it seems that it's currently down. As soon as it's back online, I'll upload the files that include the standard output from running Wavelength.fit_lambda which I used to make the attached plot, and the combined sky, Ne, and Ar images at:

ftp://ftp.noao.edu/pub/inami/tmp/mosfire/ Thank you, Hanae

https://cloud.githubusercontent.com/assets/9344283/6224070/c1df6840-b619-11e4-9e39-68921138285b.png — Reply to this email directly or view it on GitHub https://github.com/Mosfire-DataReductionPipeline/MosfireDRP/issues/13.

Hi all,

I agree with Chuck. When I tried Ar lamp calibration with Wavelength.apply_interactive as written in the current K_driver.py, it returned large rms values.

Here is the output for Ne lamp calibration with Wavelength.apply_interactive(maskname, band, waveops, apply=obsfiles, to='Ne.txt', neon=True):

Loading: Offset_1.25.txt
Loading: Offset_-1.25.txt
Loading: Offset_1.5.txt
Loading: Offset_-1.5.txt
Loading: Ne.txt
Picking Neon's arc line list
Slit number =  1
slitno  1 STD: 0.03 MAD: 0.01
Slit number =  2
slitno  2 STD: 0.04 MAD: 0.03
Slit number =  3
slitno  3 STD: 0.03 MAD: 0.01
Slit number =  4
slitno  4 STD: 0.04 MAD: 0.03
Slit number =  5
slitno  5 STD: 0.02 MAD: 0.00
Slit number =  6
slitno  6 STD: 0.01 MAD: 0.00
Slit number =  7
slitno  7 STD: 0.02 MAD: 0.00
Slit number =  8
slitno  8 STD: 0.02 MAD: 0.01
Slit number =  9
slitno  9 STD: 0.00 MAD: 0.00
Slit number =  10
slitno 10 STD: 0.04 MAD: 0.03
Slit number =  11
slitno 11 STD: 0.00 MAD: 0.00
Slit number =  12
slitno 12 STD: 0.03 MAD: 0.01
Slit number =  13
slitno 13 STD: 0.03 MAD: 0.01
Slit number =  14
slitno 14 STD: 0.05 MAD: 0.03
Slit number =  15
slitno 15 STD: 0.00 MAD: 0.00
Slit number =  16
slitno 16 STD: 0.03 MAD: 0.00
Slit number =  17
slitno 17 STD: 0.22 MAD: 0.13
Slit number =  18
slitno 18 STD: 0.04 MAD: 0.04
Slit number =  19
slitno 19 STD: 0.04 MAD: 0.01
wave_stack_K_m150116_0606-0608.fits

It looks fine except for slit 17. On the other hand, Wavelength.apply_interactive(maskname, band, waveops, apply=obsfiles, to='Ar.txt', argon=True) returned the following.

Loading: Offset_1.25.txt
Loading: Offset_-1.25.txt
Loading: Offset_1.5.txt
Loading: Offset_-1.5.txt
Loading: Ar.txt
Picking Argon's arc line list
Slit number =  1
slitno  1 STD: 0.06 MAD: 0.07
Slit number =  2
slitno  2 STD: 0.65 MAD: 0.20
Slit number =  3
slitno  3 STD: 2.08 MAD: 0.85
Slit number =  4
slitno  4 STD: 0.29 MAD: 0.13
Slit number =  5
slitno  5 STD: 0.74 MAD: 0.44
Slit number =  6
slitno  6 STD: 2.07 MAD: 1.42
Slit number =  7
slitno  7 STD: 0.36 MAD: 0.11
Slit number =  8
slitno  8 STD: 0.06 MAD: 0.04
Slit number =  9
slitno  9 STD: 0.34 MAD: 0.11
Slit number =  10
slitno 10 STD: 0.30 MAD: 0.10
Slit number =  11
slitno 11 STD: 0.26 MAD: 0.06
Slit number =  12
slitno 12 STD: 0.33 MAD: 0.15
Slit number =  13
slitno 13 STD: 0.21 MAD: 0.11
Slit number =  14
slitno 14 STD: 0.26 MAD: 0.11
Slit number =  15
slitno 15 STD: 0.27 MAD: 0.16
Slit number =  16
slitno 16 STD: 0.27 MAD: 0.27
Slit number =  17
slitno 17 STD: 0.21 MAD: 0.10
Slit number =  18
slitno 18 STD: 0.35 MAD: 0.11
Slit number =  19
slitno 19 STD: 0.53 MAD: 0.10
wave_stack_K_m150116_0609-0611.fits

Since the STDs and MADs do not looks good, I've manually fit Ar lamps by Wavelength.fit_lambda_interactively(maskname, band, 'Ar.txt', waveops, bypass=False, argon=True) and found several lines which is too weak for the automatic fitting procedure to locate correctly. When I've removed these weak Ar lines from the fitting, it worked well, but still Wavelength.fit_lambda could fail to locate those used for the initial wavelength calibration at different Y positions.

Masato Onodera

Thank you All

I have forwarded all your comments to the original user.

I'll upload here any answer

Luca

On Feb 17, 2015, at 7:12 AM, monodera notifications@github.com wrote:

Hi all,

I agree with Chuck. When I tried Ar lamp calibration with Wavelength.apply_interactive as written in the current K_driver.py, it returned large rms values.

Here is the output for Ne lamp calibration with Wavelength.apply_interactive(maskname, band, waveops, apply=obsfiles, to='Ne.txt', neon=True):

Loading: Offset1.25.txt Loading: Offset-1.25.txt Loading: Offset1.5.txt Loading: Offset-1.5.txt Loading: Ne.txt Picking Neon's arc line list Slit number = 1 slitno 1 STD: 0.03 MAD: 0.01 Slit number = 2 slitno 2 STD: 0.04 MAD: 0.03 Slit number = 3 slitno 3 STD: 0.03 MAD: 0.01 Slit number = 4 slitno 4 STD: 0.04 MAD: 0.03 Slit number = 5 slitno 5 STD: 0.02 MAD: 0.00 Slit number = 6 slitno 6 STD: 0.01 MAD: 0.00 Slit number = 7 slitno 7 STD: 0.02 MAD: 0.00 Slit number = 8 slitno 8 STD: 0.02 MAD: 0.01 Slit number = 9 slitno 9 STD: 0.00 MAD: 0.00 Slit number = 10 slitno 10 STD: 0.04 MAD: 0.03 Slit number = 11 slitno 11 STD: 0.00 MAD: 0.00 Slit number = 12 slitno 12 STD: 0.03 MAD: 0.01 Slit number = 13 slitno 13 STD: 0.03 MAD: 0.01 Slit number = 14 slitno 14 STD: 0.05 MAD: 0.03 Slit number = 15 slitno 15 STD: 0.00 MAD: 0.00 Slit number = 16 slitno 16 STD: 0.03 MAD: 0.00 Slit number = 17 slitno 17 STD: 0.22 MAD: 0.13 Slit number = 18 slitno 18 STD: 0.04 MAD: 0.04 Slit number = 19 slitno 19 STD: 0.04 MAD: 0.01 wave_stack_K_m150116_0606-0608.fits It looks fine except for slit 17. On the other hand, Wavelength.apply_interactive(maskname, band, waveops, apply=obsfiles, to='Ar.txt', argon=True) returned the following.

Loading: Offset1.25.txt Loading: Offset-1.25.txt Loading: Offset1.5.txt Loading: Offset-1.5.txt Loading: Ar.txt Picking Argon's arc line list Slit number = 1 slitno 1 STD: 0.06 MAD: 0.07 Slit number = 2 slitno 2 STD: 0.65 MAD: 0.20 Slit number = 3 slitno 3 STD: 2.08 MAD: 0.85 Slit number = 4 slitno 4 STD: 0.29 MAD: 0.13 Slit number = 5 slitno 5 STD: 0.74 MAD: 0.44 Slit number = 6 slitno 6 STD: 2.07 MAD: 1.42 Slit number = 7 slitno 7 STD: 0.36 MAD: 0.11 Slit number = 8 slitno 8 STD: 0.06 MAD: 0.04 Slit number = 9 slitno 9 STD: 0.34 MAD: 0.11 Slit number = 10 slitno 10 STD: 0.30 MAD: 0.10 Slit number = 11 slitno 11 STD: 0.26 MAD: 0.06 Slit number = 12 slitno 12 STD: 0.33 MAD: 0.15 Slit number = 13 slitno 13 STD: 0.21 MAD: 0.11 Slit number = 14 slitno 14 STD: 0.26 MAD: 0.11 Slit number = 15 slitno 15 STD: 0.27 MAD: 0.16 Slit number = 16 slitno 16 STD: 0.27 MAD: 0.27 Slit number = 17 slitno 17 STD: 0.21 MAD: 0.10 Slit number = 18 slitno 18 STD: 0.35 MAD: 0.11 Slit number = 19 slitno 19 STD: 0.53 MAD: 0.10 wave_stack_K_m150116_0609-0611.fits Since the STDs and MADs do not looks good, I've manually fit Ar lamps by Wavelength.fit_lambda_interactively(maskname, band, 'Ar.txt', waveops, bypass=False, argon=True) and found several lines which is too weak for the automatic fitting procedure to locate correctly. When I've removed these weak Ar lines from the fitting, it worked well, but still Wavelength.fit_lambda could fail to locate those used for the initial wavelength calibration at different Y positions.

Masato Onodera

— Reply to this email directly or view it on GitHub.

Dear all,

Luca, thanks for posting this on GitHub. I just made an account, so I'll be able to write on it directly. Nick, Chuck, and Masato, thank you for your answers and comments.

Okay, I'll not use the Ar option, but just use Ne for wavelength solution. Indeed, I started to use the Ar option because it is included in the K-band driver filer in the DRP ver.1-1.

I also noticed that the identification of the arc lines using night sky solution (Wavelength.apply_interactive) is significantly worse for Ar than Ne, as Masato pointed out.

Nick, NaNs in the final rectified data only appears in one mask among four masks that we observed, and it only happened when I included Ar for wavelength calibration. When I only use sky+Ne, the rectified data look fine (no NaNs). In this mask, NaNs only appear in one of the slits. I don't know if the following is the reason that NaNs only appear in this mask, but this is the mask that I only have 2 frames (4 sec in total) for the arc lamp. For all the other masks that we observed, we have 3 frames (6 sec in total). It's likely that Ar automatic fitting for this mask is worse than the others, which caused failure in wavelength calibration using sky+Ne+Ar (instead of just sky+Ne). I've attached screenshots of EPS FITS files of the mask and slit with the issue. The NaNs are at the both of the blue and red sides of the 2D spectrum, but they don't happen across the entire columns or rows. It's easer to see the slit screenshot that the wavelength calibration doesn't look right for the entire slit. Please let me know if there is anything that I can provide to help solving this problem.

Thanks again, Hanae

nan_mask

nan_slit

We too ran into the same issue. Several of our K band slits showed these features which can be removed by not using the Ar arcs in the wavelength solution.

It seems to me that this is more related to the background model than issues with the wavelength calibration. If it is actually related to wavelength solution, perhaps Marc's attempt to integrate the Ar solution (https://github.com/Keck-DataReductionPipelines/MosfireDRP/issues/37) may help fix this?

Keck-DataReductionPipelines / MosfireDRP

Arc wavelength solution #13