search

kevin218 / Eureka

Eureka! is a data reduction and analysis pipeline intended for time-series observations with JWST.
https://eurekadocs.readthedocs.io/
MIT License
58 stars 45 forks source link

[Bug]: Too short wavelength range from NIRSpec G395H data #554

Closed gmorello closed 1 year ago

gmorello commented 1 year ago

FAQ check

Instrument

NIRSpec (Stages 1-3)

What happened?

I am trying to analyse JWST NIRSpec G395H NRS1 data. The expected wavelength range is 2.86 - 3.72 micron, but I only got up to 3.098 micron (no errors appeared). This was setting waverange_start and waverange_end to None in the S2_nirspec.ecf.

Following other reported issues, I tried to change to: waverange_start 6e-8 waverange_end 6e-8 This time getting an error when processing the last segment.

Error traceback output

s3_spec, s3_meta = s3.reduce(eventlabel, ecf_path=ecf_path, s2_meta=s2_meta)

File "/home/gmorello/Eureka/src/eureka/S3_data_reduction/s3_reduce.py", line 364, in reduce data, meta = straighten.straighten_trace(data, meta, File "/home/gmorello/Eureka/src/eureka/S3_data_reduction/straighten.py", line 149, in straighten_trace data.wave_2d.values = roll_columns(wave_data, single_shift)[0] File "/home/gmorello/Eureka/src/eureka/S3_data_reduction/straighten.py", line 96, in roll_columns result = arr[tuple(all_idcs)] IndexError: index -917 is out of bounds for axis 1 with size 26

What operating system are you using?

Ubuntu 22.04

What version of Python are you running?

Python 3.9.7

What Python packages do you have installed?

_libgcc_mutex 0.1 main _openmp_mutex 5.1 1_gnu alabaster 0.7.13 pypi_0 pypi asciitree 0.3.3 pypi_0 pypi asdf 2.15.0 pypi_0 pypi asdf-astropy 0.4.0 pypi_0 pypi asdf-coordinates-schemas 0.2.0 pypi_0 pypi asdf-standard 1.0.3 pypi_0 pypi asdf-transform-schemas 0.3.0 pypi_0 pypi asdf-unit-schemas 0.1.0 pypi_0 pypi asdf-wcs-schemas 0.1.1 pypi_0 pypi asteval 0.9.30 pypi_0 pypi astraeus 0.3 pypi_0 pypi astropy 5.3 pypi_0 pypi astropy-healpix 0.7 pypi_0 pypi astroquery 0.4.6 pypi_0 pypi astroscrappy 1.1.0 pypi_0 pypi asttokens 2.2.1 pypi_0 pypi attrs 23.1.0 pypi_0 pypi babel 2.12.1 pypi_0 pypi backcall 0.2.0 pypi_0 pypi batman-package 2.4.9 pypi_0 pypi bayesicfitting 3.1.1 pypi_0 pypi beautifulsoup4 4.12.2 pypi_0 pypi bokeh 2.4.3 pypi_0 pypi bottleneck 1.3.7 pypi_0 pypi ca-certificates 2023.05.30 h06a4308_0 ccdproc 2.4.1 pypi_0 pypi celerite 0.4.2 pypi_0 pypi certifi 2022.5.18.1 pypi_0 pypi cffi 1.15.1 pypi_0 pypi cftime 1.6.2 pypi_0 pypi charset-normalizer 3.1.0 pypi_0 pypi cloudpickle 2.2.1 pypi_0 pypi contourpy 1.1.0 pypi_0 pypi corner 2.2.2 pypi_0 pypi crds 11.17.2 pypi_0 pypi cryptography 41.0.1 pypi_0 pypi cycler 0.11.0 pypi_0 pypi cython 0.29.35 pypi_0 pypi dask 2022.6.0 pypi_0 pypi decorator 5.1.1 pypi_0 pypi docutils 0.20.1 pypi_0 pypi drizzle 1.13.7 pypi_0 pypi dynesty 2.1.2 pypi_0 pypi emcee 3.1.4 pypi_0 pypi entrypoints 0.4 pypi_0 pypi eureka 0.9 pypi_0 pypi exceptiongroup 1.1.2 pypi_0 pypi executing 1.2.0 pypi_0 pypi exotic-ld 3.0.0 pypi_0 pypi fasteners 0.18 pypi_0 pypi filelock 3.12.2 pypi_0 pypi fonttools 4.40.0 pypi_0 pypi fsspec 2023.6.0 pypi_0 pypi future 0.18.3 pypi_0 pypi george 0.4.0 pypi_0 pypi gwcs 0.18.3 pypi_0 pypi h5netcdf 1.2.0 pypi_0 pypi h5py 3.1.0 pypi_0 pypi html5lib 1.1 pypi_0 pypi idna 3.4 pypi_0 pypi imageio 2.31.1 pypi_0 pypi imagesize 1.4.1 pypi_0 pypi importlib-metadata 6.7.0 pypi_0 pypi importlib-resources 5.12.0 pypi_0 pypi iniconfig 2.0.0 pypi_0 pypi ipython 8.14.0 pypi_0 pypi jaraco-classes 3.2.3 pypi_0 pypi jedi 0.18.2 pypi_0 pypi jeepney 0.8.0 pypi_0 pypi jinja2 3.1.2 pypi_0 pypi jmespath 1.0.1 pypi_0 pypi jsonschema 4.9.1 pypi_0 pypi jwst 1.8.2 pypi_0 pypi keyring 24.2.0 pypi_0 pypi kiwisolver 1.4.4 pypi_0 pypi lazy-loader 0.3 pypi_0 pypi ld_impl_linux-64 2.38 h1181459_1 libffi 3.3 he6710b0_2 libgcc-ng 11.2.0 h1234567_1 libgomp 11.2.0 h1234567_1 libstdcxx-ng 11.2.0 h1234567_1 lmfit 1.2.1 pypi_0 pypi locket 1.0.0 pypi_0 pypi markupsafe 2.1.3 pypi_0 pypi matplotlib 3.7.1 pypi_0 pypi matplotlib-inline 0.1.6 pypi_0 pypi more-itertools 9.1.0 pypi_0 pypi ncurses 6.4 h6a678d5_0 netcdf4 1.6.4 pypi_0 pypi networkx 3.1 pypi_0 pypi numcodecs 0.11.0 pypi_0 pypi numpy 1.22.0 pypi_0 pypi numpydoc 1.5.0 pypi_0 pypi openssl 1.1.1u h7f8727e_0 packaging 23.1 pypi_0 pypi pandas 2.0.3 pypi_0 pypi parsley 1.3 pypi_0 pypi parso 0.8.3 pypi_0 pypi partd 1.4.0 pypi_0 pypi pexpect 4.8.0 pypi_0 pypi photutils 1.8.0 pypi_0 pypi pickleshare 0.7.5 pypi_0 pypi pillow 10.0.0 pypi_0 pypi pip 23.1.2 py39h06a4308_0 pluggy 1.2.0 pypi_0 pypi poppy 1.1.1 pypi_0 pypi prompt-toolkit 3.0.38 pypi_0 pypi psutil 5.9.5 pypi_0 pypi ptyprocess 0.7.0 pypi_0 pypi pure-eval 0.2.2 pypi_0 pypi pycparser 2.21 pypi_0 pypi pyerfa 2.0.0.3 pypi_0 pypi pygments 2.15.1 pypi_0 pypi pyparsing 3.1.0 pypi_0 pypi pyrsistent 0.19.3 pypi_0 pypi pysynphot 2.0.0 pypi_0 pypi pytest 7.4.0 pypi_0 pypi python 3.9.7 h12debd9_1 python-dateutil 2.8.2 pypi_0 pypi pytz 2023.3 pypi_0 pypi pyvo 1.4.1 pypi_0 pypi pywavelets 1.4.1 pypi_0 pypi pyyaml 6.0 pypi_0 pypi readline 8.2 h5eee18b_0 reproject 0.11.0 pypi_0 pypi requests 2.31.0 pypi_0 pypi scikit-image 0.21.0 pypi_0 pypi scipy 1.11.1 pypi_0 pypi secretstorage 3.3.3 pypi_0 pypi semantic-version 2.10.0 pypi_0 pypi setuptools 67.8.0 py39h06a4308_0 setuptools-scm 7.1.0 pypi_0 pypi six 1.16.0 pypi_0 pypi snowballstemmer 2.2.0 pypi_0 pypi soupsieve 2.4.1 pypi_0 pypi spherical-geometry 1.2.23 pypi_0 pypi sphinx 7.0.1 pypi_0 pypi sphinxcontrib-applehelp 1.0.4 pypi_0 pypi sphinxcontrib-devhelp 1.0.2 pypi_0 pypi sphinxcontrib-htmlhelp 2.0.1 pypi_0 pypi sphinxcontrib-jsmath 1.0.1 pypi_0 pypi sphinxcontrib-qthelp 1.0.3 pypi_0 pypi sphinxcontrib-serializinghtml 1.1.5 pypi_0 pypi sqlite 3.41.2 h5eee18b_0 stack-data 0.6.2 pypi_0 pypi stcal 1.2.2 pypi_0 pypi stdatamodels 0.4.5 pypi_0 pypi stpipe 0.5.0 pypi_0 pypi stsci-image 2.3.5 pypi_0 pypi stsci-imagestats 1.6.3 pypi_0 pypi stsci-stimage 0.2.6 pypi_0 pypi svo-filters 0.4.4 pypi_0 pypi tifffile 2023.4.12 pypi_0 pypi tk 8.6.12 h1ccaba5_0 tomli 2.0.1 pypi_0 pypi toolz 0.12.0 pypi_0 pypi tornado 6.3.2 pypi_0 pypi tqdm 4.65.0 pypi_0 pypi traitlets 5.9.0 pypi_0 pypi tweakwcs 0.8.2 pypi_0 pypi typing-extensions 4.7.1 pypi_0 pypi tzdata 2023.3 pypi_0 pypi uncertainties 3.1.7 pypi_0 pypi urllib3 2.0.3 pypi_0 pypi wcwidth 0.2.6 pypi_0 pypi webencodings 0.5.1 pypi_0 pypi wheel 0.38.4 py39h06a4308_0 wiimatch 0.3.1 pypi_0 pypi xarray 2023.6.0 pypi_0 pypi xz 5.4.2 h5eee18b_0 zarr 2.15.0 pypi_0 pypi zipp 3.15.0 pypi_0 pypi zlib 1.2.13 h5eee18b_0

Code of Conduct

AarynnCarter commented 1 year ago

Hi @gmorello I won't be able to look at this properly until at least next week. When you look at the output Stage 2 data products, is the full array there or has it been trimmed?

I am also not personally familiar with the functions related to the trace straightening. @jbrande, @erinmmay have you seen errors like this during ERS / GO work?

erinmmay commented 1 year ago

For the wavelength issue: what is your xwindow in your stage 3 ecf?

For that error, this has been fixed in newer versions of Eureka. Can you tell me when you last installed/updated Eureka? I also can see from that error that it looks like your ywindow in your stage 3 or your slit_y_low and slit_y_high in stage 2 is not the full array [0,32]. I recommend using the full array for G395H because the trace does get rather close to the edges of the detector due to the curvature.

gmorello commented 1 year ago

Hi, thanks both for your answers. I have just updated Eureka! In stage2, I am using (copied from demos/JWST/S2_nirspec_fs_template.ecf) slit_y_low -1 slit_y_high 50 I now try to change to 0 and 32, but perhaps the issue is in stage3 (copied from demos/JWST/S3_nirspec_fs_template.ecf): ywindow [2,28] # Vertical axis as seen in DS9 xwindow [60,410] # Horizontal axis as seen in DS9 @erinmmay What should I use instead? And how to understand it for different instrument configurations?

gmorello commented 1 year ago

Oh, maybe it was a stupid question. I can see the trace in ds9.

erinmmay commented 1 year ago

Your slit_y_low and slit_y_high are fine (as long as they're outside of 0 and 32 it won't trim anything), however for the ywindow in stage 3 I recommend 0 to 32 just so you don't lose any data along the edges.

The xwindow explains the wavelength region. If you open the stage 2 outputs in DS9 you can see where the trace is and use that information to determine which values to put in these boxes. The G395H disperser uses the full width of the NRS1 detector, which is 2048 pixels. If you look at the NRS1 file in DS9 you'll notice that the trace starts around pixel ~600, so a good xwindow range for NRS1 is [600,2047]. In general you'll want to open your stage 2 fits file in DS9 to get this information

gmorello commented 1 year ago

Hi, now it seems to work somehow, but even after updating Eureka!, I get the same error for the seg003 if setting waverange_start 6e-8 and waverange_end 6e-6. It works if I set both to None.

However, I also tried excluding seg003, and obtained different wavelength ranges from the same windows. If I set waverange_start 6e-8 and waverange_end 6e-6, I get 2.471-3.149 micron. If I set waverange_start None and waverange_end None, I get 2.957-3.632 micron.

erinmmay commented 1 year ago

I am unable to reproduce this wavelength error on other G395H data. waverange_start = 6e-8 and waverange_end = 6e-6 outputs the full detector in Stage 2 for me. It's also not clear to me why this is only impacting seg003. I'm downloading this data now to look into it more.

In the meantime, please check your stage 2 output fits files in ds9 - each integration should have the shape 32x2048. If all have that shape, then the issue with wavelengths is still with your xwindow in stage 3, the wavelengths will depend on what you put there. If they are not both 32x2048, please share your stage 1 and stage 2 ecfs so I can confirm if this is a bug with the jwst pipeline or Eureka!. But I'll note that Eureka! does not have any custom steps in Stage 2 and only serves as a wrapper for the jwst pipeline.

I did just take a look at the Stage 3 code that's in the main branch of Eureka! and see that the fix to that original error is there. Please double check is that you correctly updated Eureka! within the conda environment you're using (if you're using an environment). Instructions for updating are here: https://eurekadocs.readthedocs.io/en/latest/installation.html . Just remember to activate your conda environment before updating -- this is a common thing to forget to do!

Specifically, that error popped up when the code was unable to find the center of light in a given column because there was no light in that column. This possibility is now accounted for in the code that finds the center of the light/trace, but can also be side stepped by setting an xwindow that is guaranteed to have the trace within it. This is primarily an issue for NRS1 since the trace doesn't fall on the first ~25% of the detector. But if you use an xwindow of [1000,2047], for example, the error should not appear. I also recommend testing this while I look into why your issues are only impacting seg003.

erinmmay commented 1 year ago

After running some test on the WASP-39b G395H data (I believe this is what you're using based on a different issue you opened) I still cannot reproduce your error.

I did notice that if you use "None" for the waverange_start and waverange_end in stage 2 it actually does pre-trim your array (mine has shape [32x1333]), so your xwindow in stage 3 will be different depending on what settings you select in stage 2. (again, this is why you always should look at the stage 2 outputs to select that window).

Otherwise I am unable to reproduce any problems with segment 3, and my stage 3 runs without a problem as long as I select an xwindow based on the stage 2 outputs. In both waverange_start/end cases my stage 3 outputs the full wavelength range that is expected.

gmorello commented 1 year ago

Hi all, You are right, as there was a problem with my last update. The error now disappeared. I checked that the results obtained with None or specifying a large wavelength range are identical in the common wavelength range. Thanks for your help!