DRP fails to reduce K band data

[hebeling]

I ran into the following fatal problem when running Driver.py on K-band data taken in April:

2019-04-28 18:04:25,605 - Wavelength.apply_lambda_sky_and_arc - INFO: Shifted arc by an average of -0.83 pixels 2019-04-28 18:04:27,335 - Wavelength.apply_lambda_sky_and_arc - INFO: 2d wavelengths: Slit 15/20 2019-04-28 18:04:28,030 - Wavelength.apply_lambda_sky_and_arc - INFO: Shifted arc by an average of -0.82 pixels 2019-04-28 18:04:29,612 - Wavelength.apply_lambda_sky_and_arc - INFO: 2d wavelengths: Slit 16/20 /Users/ebeling/anaconda3/envs/mospy_2018_macos/lib/python3.6/site-packages/numpy/core/fromnumeric.py:2909: RuntimeWarning: Mean of empty slice. out=out, kwargs) 2019-04-28 18:04:29,612 - Wavelength.apply_lambda_sky_and_arc - INFO: Shifted arc by an average of nan pixels /Users/ebeling/anaconda3/envs/mospy_2018_macos/lib/python3.6/site-packages/numpy/core/fromnumeric.py:2909: RuntimeWarning: Mean of empty slice. out=out, kwargs) 2019-04-28 18:04:29,613 - Wavelength.apply_lambda_sky_and_arc - INFO: 2d wavelengths: Slit 17/20 2019-04-28 18:04:31,379 - Wavelength.apply_lambda_sky_and_arc - INFO: Shifted arc by an average of -0.79 pixels 2019-04-28 18:04:33,021 - Wavelength.apply_lambda_sky_and_arc - INFO: 2d wavelengths: Slit 18/20 2019-04-28 18:04:34,030 - Wavelength.apply_lambda_sky_and_arc - INFO: Shifted arc by an average of -0.75 pixels 2019-04-28 18:04:35,672 - Wavelength.apply_lambda_sky_and_arc - INFO: 2d wavelengths: Slit 19/20 2019-04-28 18:04:38,999 - Wavelength.apply_lambda_sky_and_arc - INFO: Shifted arc by an average of -0.76 pixels 2019-04-28 18:04:40,786 - Wavelength.apply_lambda_sky_and_arc - INFO: 2d wavelengths: Slit 20/20 2019-04-28 18:04:45,385 - Wavelength.apply_lambda_sky_and_arc - INFO: Shifted arc by an average of -0.74 pixels 2019-04-28 18:04:47,226 - Wavelength.apply_lambda_sky_and_arc - INFO: e1248: writing lambda 2019-04-28 18:04:47,233 - IO. writefits - INFO: Wrote to 'merged_lambda_solution_wave_stack_K_m190426_0240-0250_and_wave_stack_K_m190426_0068-0070.fits' 2019-04-28 18:04:47,798 - Wavelength.apply_lambda_sky_and_arc - INFO: e1248: rectifying Traceback (most recent call last): File "Driver.py", line 45, in Wavelength.apply_lambda_sky_and_arc(maskname, band, obsfiles, 'Ne.txt', LROIs, waveops) File "/Users/ebeling/anaconda3/envs/mospy_2018_macos/lib/python3.6/site-packages/MOSFIRE-1.0.dev0-py3.6.egg/MOSFIRE/Wavelength.py", line 838, in apply_lambda_sky_and_arc ll_fid = np.arange(hpp[0], hpp[1], dlam) ValueError: Maximum allowed size exceeded

The DRP had no problem reducing J- and H-band data for the same mask. No problems were noted during the line identification (I am not using arcs). My python skills are not sufficiently developed to even understand the error message - exactly what has exceeded its maximal allowable size?

Thanks in advance for all help!

Aloha,

Harald

[joshwalawender]

Hi Harald, can you point me to the data, so I can run it here and see what might be going on?

-Josh

[csteidel]

This sounds like the kind of thing that happens when a slit edge is not identified successfully, usually due to two adjacent slits that are almost, but not quite, covering the same wavelength range... there are workarounds if that is the case...

On May 6, 2019, at 1:05 PM, Josh Walawender notifications@github.com wrote:

Hi Harald, can you point me to the data, so I can run it here and see what might be going on?

-Josh

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

[hebeling]

Apologies for the late reply - I was out of email contact for a few days. The data were taken on Apr 26, 2019 (UT). I think the contents of the Offset* files contain the relevant file names:

[marvin:apr19/e1248/K] ebeling% more Offset_*

Created by 'ebeling' on Fri Apr 26 04:37:52 2019

/d1/keck/mosfire/raw/apr19 # Abs. path to files [optional]

m190426_0241.fits # 178.93917 s

m190426_0242.fits # 178.93917 s

m190426_0245.fits # 178.93917 s

m190426_0246.fits # 178.93917 s

m190426_0249.fits # 178.93917 s

m190426_0250.fits # 178.93917 s

...skipping...

Created by 'ebeling' on Fri Apr 26 04:37:52 2019

/d1/keck/mosfire/raw/apr19 # Abs. path to files [optional]

m190426_0240.fits # 178.93917 s

m190426_0243.fits # 178.93917 s

m190426_0244.fits # 178.93917 s

m190426_0247.fits # 178.93917 s

m190426_0248.fits # 178.93917 s

m190426_0251.fits # 178.93917 s

Thanks for helping!

Aloha,

Harald

On Mon, May 6, 2019 at 10:21 AM csteidel notifications@github.com wrote:

This sounds like the kind of thing that happens when a slit edge is not identified successfully, usually due to two adjacent slits that are almost, but not quite, covering the same wavelength range... there are workarounds if that is the case...

On May 6, 2019, at 1:05 PM, Josh Walawender notifications@github.com wrote:

Hi Harald, can you point me to the data, so I can run it here and see what might be going on?

-Josh

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/Keck-DataReductionPipelines/MosfireDRP/issues/138#issuecomment-489761740, or mute the thread https://github.com/notifications/unsubscribe-auth/AHRYWWCLBUS2X2FOEKIZF5TPUCHIDANCNFSM4HLCVP3A .

[hebeling]

Aloha MosfireDRP experts,

Did you find time to check whether the problem I encountered is indeed caused by two adjacent slits that are only a tad off in their wavelength coverage, as you suspected? If so, how do I work around the ensuing DRP crash?

Mahalo!

Harald

On Fri, May 10, 2019 at 5:41 PM Harald Ebeling ebeling@ifa.hawaii.edu wrote:

Apologies for the late reply - I was out of email contact for a few days. The data were taken on Apr 26, 2019 (UT). I think the contents of the Offset* files contain the relevant file names:

[marvin:apr19/e1248/K] ebeling% more Offset_*

Created by 'ebeling' on Fri Apr 26 04:37:52 2019

/d1/keck/mosfire/raw/apr19 # Abs. path to files [optional]

m190426_0241.fits # 178.93917 s

m190426_0242.fits # 178.93917 s

m190426_0245.fits # 178.93917 s

m190426_0246.fits # 178.93917 s

m190426_0249.fits # 178.93917 s

m190426_0250.fits # 178.93917 s

...skipping...

Created by 'ebeling' on Fri Apr 26 04:37:52 2019

/d1/keck/mosfire/raw/apr19 # Abs. path to files [optional]

m190426_0240.fits # 178.93917 s

m190426_0243.fits # 178.93917 s

m190426_0244.fits # 178.93917 s

m190426_0247.fits # 178.93917 s

m190426_0248.fits # 178.93917 s

m190426_0251.fits # 178.93917 s

Thanks for helping!

Aloha,

Harald

On Mon, May 6, 2019 at 10:21 AM csteidel notifications@github.com wrote:

This sounds like the kind of thing that happens when a slit edge is not identified successfully, usually due to two adjacent slits that are almost, but not quite, covering the same wavelength range... there are workarounds if that is the case...

On May 6, 2019, at 1:05 PM, Josh Walawender notifications@github.com wrote:

Hi Harald, can you point me to the data, so I can run it here and see what might be going on?

-Josh

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/Keck-DataReductionPipelines/MosfireDRP/issues/138#issuecomment-489761740, or mute the thread https://github.com/notifications/unsubscribe-auth/AHRYWWCLBUS2X2FOEKIZF5TPUCHIDANCNFSM4HLCVP3A .

[lucarizzi]

I am downloading your data now. by the way, you are actually using arcs see this line in your Driver file: Wavelength.apply_lambda_sky_and_arc(maskname, band, obsfiles, 'Ne.txt', LROIs, waveops)

But they are used as secondary paired with the sky lines

[lucarizzi]

So the problem is related to using both the sky lines and the arc lines. I am not 100% sure I can pinpoint the exact cause, but if you disable the arc lines, you can get through the reduction. Here is a driver file that skips the arc lines:

import matplotlib matplotlib.use('TkAgg') # Force TkAgg backend for interactivity. This is

critical to bypass a bug in the MacOSX backend.

import os import time import logging logger = logging.getLogger(name) import sys

import MOSFIRE from MOSFIRE import Background, Combine, Detector, Flats, IO, Options, Rectify, Wavelength, Extract from MOSFIRE.MosfireDrpLog import info, debug, warning, error

import numpy as np np.seterr(all='ignore')

flatops = Options.flat waveops = Options.wavelength

Driver file automatically generated on Fri May 17 09:24:58 2019

If you have questions, please submit a ticket on the github issue page:

https://github.com/Keck-DataReductionPipelines/MosfireDRP/issues

Alternatively, email the developers at mosfiredrp@gmail.com

maskname = 'e1248' band = 'K'

Set noninteractive to True to autofit wavelenth solution instead of manually fitting.

noninteractiveflag=False obsfiles=['Offset1.5.txt','Offset-1.5.txt']

Flats.handle_flats('Flat.txt', maskname, band, flatops,lampOffList='FlatThermal.txt')

Wavelength.imcombine(obsfiles, maskname, band, waveops) Wavelength.fit_lambda_interactively(maskname, band, obsfiles,waveops, noninteractive=noninteractiveflag) Wavelength.fit_lambda(maskname, band, obsfiles, obsfiles,waveops) Wavelength.apply_lambda_simple(maskname, band, obsfiles, waveops)

Wavelength_file = 'lambda_solution_wave_stack_K_m190426_0240-0250.fits'

Background.handle_background(obsfiles,Wavelength_file,maskname,band,waveops)

redfiles = ["eps_" + file + ".fits" for file in obsfiles] Rectify.handle_rectification(maskname, redfiles,Wavelength_file,band,obsfiles,waveops)

Extract.extract_spectra(maskname, band, width=10, interactive=(not noninteractiveflag))

[hebeling]

Great, Luca, thank you! I don't think I need the arclines - the dispersion solution based on just the sky lines looked perfectly fine to me.

Mahalo,

Harald

On Thu, May 16, 2019 at 9:52 PM lucarizzi notifications@github.com wrote:

So the problem is related to using both the sky lines and the arc lines. I am not 100% sure I can pinpoint the exact cause, but if you disable the arc lines, you can get through the reduction. Here is a driver file that skips the arc lines:

import matplotlib matplotlib.use('TkAgg') # Force TkAgg backend for interactivity. This is

critical to bypass a bug in the MacOSX backend.

import os import time import logging logger = logging.getLogger(name) import sys

import MOSFIRE from MOSFIRE import Background, Combine, Detector, Flats, IO, Options, Rectify, Wavelength, Extract from MOSFIRE.MosfireDrpLog import info, debug, warning, error

import numpy as np np.seterr(all='ignore')

flatops = Options.flat waveops = Options.wavelength Driver file automatically generated on Fri May 17 09:24:58 2019 If you have questions, please submit a ticket on the github issue page: https://github.com/Keck-DataReductionPipelines/MosfireDRP/issues Alternatively, email the developers at mosfiredrp@gmail.com

maskname = 'e1248' band = 'K'

Set noninteractive to True to autofit wavelenth solution instead of

manually fitting. noninteractiveflag=False obsfiles=['Offset1.5.txt','Offset-1.5.txt']

Flats.handle_flats('Flat.txt', maskname, band, flatops,lampOffList='FlatThermal.txt')

Wavelength.imcombine(obsfiles, maskname, band, waveops) Wavelength.fit_lambda_interactively(maskname, band, obsfiles,waveops, noninteractive=noninteractiveflag) Wavelength.fit_lambda(maskname, band, obsfiles, obsfiles,waveops) Wavelength.apply_lambda_simple(maskname, band, obsfiles, waveops)

Wavelength_file = 'lambda_solution_wave_stack_K_m190426_0240-0250.fits'

Background.handle_background(obsfiles,Wavelength_file,maskname,band,waveops)

redfiles = ["eps_" + file + ".fits" for file in obsfiles] Rectify.handle_rectification(maskname, redfiles,Wavelength_file,band,obsfiles,waveops)

Extract.extract_spectra(maskname, band, width=10, interactive=(not

noninteractiveflag))

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/Keck-DataReductionPipelines/MosfireDRP/issues/138?email_source=notifications&email_token=AHRYWWBS5TSMPFSEHSHAMT3PVZPW3A5CNFSM4HLCVP3KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODVUA2WI#issuecomment-493358425, or mute the thread https://github.com/notifications/unsubscribe-auth/AHRYWWAUC7YMBIF2E4YXZ6LPVZPW3ANCNFSM4HLCVP3A .

[lucarizzi]

Just for completeness, and if you want to compare, using only Neon lines but no Argon lines, works as well:

import matplotlib matplotlib.use('TkAgg') # Force TkAgg backend for interactivity. This is

critical to bypass a bug in the MacOSX backend.

import os import time import logging logger = logging.getLogger(name)

import MOSFIRE from MOSFIRE import Background, Combine, Detector, Flats, IO, Options, Rectify, Wavelength, Extract from MOSFIRE.MosfireDrpLog import info, debug, warning, error

import numpy as np np.seterr(all='ignore')

flatops = Options.flat waveops = Options.wavelength

maskname = 'e1248' band = 'K'

Set noninteractive to True to autofit wavelenth solution instead of manually fitting.

noninteractiveflag=False obsfiles=['Offset1.5.txt','Offset-1.5.txt']

Flats.handle_flats('Flat.txt', maskname, band, flatops,lampOffList='FlatThermal.txt')

Wavelength.imcombine(obsfiles, maskname, band, waveops) Wavelength.imcombine('Ne.txt', maskname, band, waveops) Wavelength.fit_lambda_interactively(maskname, band, obsfiles,waveops, noninteractive=noninteractiveflag) Wavelength.apply_interactive(maskname, band, waveops, apply=obsfiles, to='Ne.txt', neon=True) Wavelength.fit_lambda(maskname, band, obsfiles, obsfiles,waveops) Wavelength.fit_lambda(maskname, band, 'Ne.txt', 'Ne.txt',waveops) LROI = [[21000,22800]]*1 LROIs = Wavelength.check_wavelength_roi(maskname, band, obsfiles, 'Ne.txt', LROI, waveops) Wavelength.apply_lambda_simple(maskname, band, obsfiles, waveops) Wavelength.apply_lambda_sky_and_arc(maskname, band, obsfiles, 'Ne.txt', LROIs, waveops)

Wavelength_file = 'merged_lambda_solution_wave_stack_K_m190426_0240-0250_and_wave_stack_K_m190426_0068-0070.fits'

Background.handle_background(obsfiles,Wavelength_file,maskname,band,waveops)

redfiles = ["eps_" + file + ".fits" for file in obsfiles] Rectify.handle_rectification(maskname, redfiles,Wavelength_file,band,obsfiles,waveops)

Extract.extract_spectra(maskname, band, width=10, interactive=(not noninteractiveflag))