search

kevin218 / Eureka

Eureka! is a data reduction and analysis pipeline intended for time-series observations with JWST.
https://eurekadocs.readthedocs.io/
MIT License
60 stars 47 forks source link

[Bug]: stuck at S3 with real NIRSPEC data - index out of bond #402

Closed trappist1i closed 2 years ago

trappist1i commented 2 years ago

FAQ check

Instrument

NIRSpec (Stages 1-3)

What happened?

Playing with some of the first real NIRSPEC/Prism transit data, I am stuck at the beginning of stage 3 with the following error message. Apparently, the pipeline finds the calint.fits files but crashes on some array indexation. This error does not occur with the simulated (long and tiny) wasp-39b data.

Error traceback output

Starting Stage 3 Reduction

Input directory: /Users/clavainl/JWST_REDU/data_analysis/Trappist1/Nirspec_2420/Transit1/Stage2/S2_2022-07-18_trappist1c_run1/ Output directory: /Users/clavainl/JWST_REDU/data_analysis/Trappist1/Nirspec_2420/Transit1/Stage3/S3_2022-07-18_trappist1c_run2/ap6_bg7/ Using ap=6, bg=7 Copying S3 control file

Found 3 data file(s) ending in calints.fits WARNING: Are you using real JWST data? If so, you should edit the flag_bg() function in nirspec.py and look at Issue #193 on Github! Starting file 1 of 3 Traceback (most recent call last): File "/Users/clavainl/JWST_REDU/data_analysis/Trappist1/Nirspec_2420/Transit1/run_eureka.py", line 27, in s3_spec, s3_meta = s3.reduce(eventlabel, ecf_path=ecf_path, File "/Users/clavainl/src/eureka/src/eureka/S3_data_reduction/s3_reduce.py", line 263, in reduce data, meta = util.trim(data, meta) File "/Users/clavainl/src/eureka/src/eureka/lib/util.py", line 69, in trim subdata = data.isel(y=np.arange(meta.ywindow[0], meta.ywindow[1]), File "/Users/clavainl/anaconda2/envs/eureka/lib/python3.9/site-packages/xarray/core/dataset.py", line 2366, in isel var_value = var_value.isel(var_indexers) File "/Users/clavainl/anaconda2/envs/eureka/lib/python3.9/site-packages/xarray/core/variable.py", line 1186, in isel return self[key] File "/Users/clavainl/anaconda2/envs/eureka/lib/python3.9/site-packages/xarray/core/variable.py", line 779, in getitem data = as_indexable(self._data)[indexer] File "/Users/clavainl/anaconda2/envs/eureka/lib/python3.9/site-packages/xarray/core/indexing.py", line 1324, in getitem result = self.array[key] File "/Users/clavainl/anaconda2/envs/eureka/lib/python3.9/site-packages/pandas/core/indexes/base.py", line 5055, in getitem result = getitem(key) IndexError: index 16 is out of bounds for axis 0 with size 16

What operating system are you using?

Mac OS 10.14.6

What version of Python are you running?

Python 3.9.7

What Python packages do you have installed?

packages in environment at /Users/clavainl/anaconda2/envs/eureka:

#

Name Version Build Channel

alabaster 0.7.12 pypi_0 pypi appnope 0.1.3 pypi_0 pypi asdf 2.12.0 pypi_0 pypi asdf-astropy 0.2.1 pypi_0 pypi asdf-coordinates-schemas 0.1.0 pypi_0 pypi asdf-standard 1.0.2 pypi_0 pypi asdf-transform-schemas 0.2.2 pypi_0 pypi asdf-wcs-schemas 0.1.1 pypi_0 pypi asteval 0.9.27 pypi_0 pypi astraeus 0.2 pypi_0 pypi astropy 5.1 pypi_0 pypi astropy-healpix 0.6 pypi_0 pypi astroquery 0.4.6 pypi_0 pypi astroscrappy 1.1.0 pypi_0 pypi asttokens 2.0.5 pypi_0 pypi attrs 21.4.0 pypi_0 pypi babel 2.10.3 pypi_0 pypi backcall 0.2.0 pypi_0 pypi batman-package 2.4.9 pypi_0 pypi bayesicfitting 3.0.1 pypi_0 pypi beautifulsoup4 4.11.1 pypi_0 pypi bokeh 2.4.3 pypi_0 pypi bottleneck 1.3.5 pypi_0 pypi ca-certificates 2022.4.26 hecd8cb5_0
ccdproc 2.3.1 pypi_0 pypi celerite 0.4.2 pypi_0 pypi certifi 2022.6.15 py39hecd8cb5_0
cftime 1.6.1 pypi_0 pypi charset-normalizer 2.1.0 pypi_0 pypi cloudpickle 2.1.0 pypi_0 pypi corner 2.2.1 pypi_0 pypi crds 11.16.5 pypi_0 pypi cycler 0.11.0 pypi_0 pypi cython 0.29.30 pypi_0 pypi dask 2022.6.0 pypi_0 pypi decorator 5.1.1 pypi_0 pypi docutils 0.18.1 pypi_0 pypi drizzle 1.13.4 pypi_0 pypi dynesty 1.2.3 pypi_0 pypi emcee 3.1.2 pypi_0 pypi eureka 0.2 dev_0 executing 0.8.3 pypi_0 pypi exotic-ld 2.0.0 pypi_0 pypi filelock 3.7.1 pypi_0 pypi fonttools 4.34.4 pypi_0 pypi fsspec 2022.5.0 pypi_0 pypi future 0.18.2 pypi_0 pypi george 0.4.0 pypi_0 pypi gwcs 0.18.1 pypi_0 pypi h5netcdf 1.0.1 pypi_0 pypi h5py 3.1.0 pypi_0 pypi html5lib 1.1 pypi_0 pypi idna 3.3 pypi_0 pypi imageio 2.19.3 pypi_0 pypi imagesize 1.4.1 pypi_0 pypi importlib-metadata 4.12.0 pypi_0 pypi iniconfig 1.1.1 pypi_0 pypi ipython 8.4.0 pypi_0 pypi jedi 0.18.1 pypi_0 pypi jinja2 3.1.2 pypi_0 pypi jmespath 1.0.1 pypi_0 pypi jsonschema 4.7.2 pypi_0 pypi jwst 1.6.0 pypi_0 pypi keyring 23.7.0 pypi_0 pypi kiwisolver 1.4.4 pypi_0 pypi libcxx 12.0.0 h2f01273_0
libffi 3.3 hb1e8313_2
lmfit 1.0.3 pypi_0 pypi locket 1.0.0 pypi_0 pypi lxml 4.9.1 pypi_0 pypi markupsafe 2.1.1 pypi_0 pypi matplotlib 3.5.2 pypi_0 pypi matplotlib-inline 0.1.3 pypi_0 pypi ncurses 6.3 hca72f7f_3
netcdf4 1.6.0 pypi_0 pypi networkx 2.8.4 pypi_0 pypi numpy 1.23.1 pypi_0 pypi numpydoc 1.4.0 pypi_0 pypi openssl 1.1.1q hca72f7f_0
packaging 21.3 pypi_0 pypi pandas 1.4.3 pypi_0 pypi parsley 1.3 pypi_0 pypi parso 0.8.3 pypi_0 pypi partd 1.2.0 pypi_0 pypi pexpect 4.8.0 pypi_0 pypi photutils 1.5.0 pypi_0 pypi pickleshare 0.7.5 pypi_0 pypi pillow 9.2.0 pypi_0 pypi pip 22.1.2 py39hecd8cb5_0
pluggy 1.0.0 pypi_0 pypi poppy 1.0.3 pypi_0 pypi prompt-toolkit 3.0.30 pypi_0 pypi psutil 5.9.1 pypi_0 pypi ptyprocess 0.7.0 pypi_0 pypi pure-eval 0.2.2 pypi_0 pypi py 1.11.0 pypi_0 pypi pyerfa 2.0.0.1 pypi_0 pypi pygments 2.12.0 pypi_0 pypi pyparsing 3.0.9 pypi_0 pypi pyrsistent 0.18.1 pypi_0 pypi pysynphot 2.0.0 pypi_0 pypi pytest 7.1.2 pypi_0 pypi python 3.9.7 h88f2d9e_1
python-dateutil 2.8.2 pypi_0 pypi pytz 2022.1 pypi_0 pypi pyvo 1.3 pypi_0 pypi pywavelets 1.3.0 pypi_0 pypi pyyaml 6.0 pypi_0 pypi readline 8.1.2 hca72f7f_1
reproject 0.8 pypi_0 pypi requests 2.28.1 pypi_0 pypi scikit-image 0.19.3 pypi_0 pypi scipy 1.8.1 pypi_0 pypi semantic-version 2.10.0 pypi_0 pypi setuptools 61.2.0 py39hecd8cb5_0
six 1.16.0 pypi_0 pypi snowballstemmer 2.2.0 pypi_0 pypi soupsieve 2.3.2.post1 pypi_0 pypi spherical-geometry 1.2.22 pypi_0 pypi sphinx 5.0.2 pypi_0 pypi sphinxcontrib-applehelp 1.0.2 pypi_0 pypi sphinxcontrib-devhelp 1.0.2 pypi_0 pypi sphinxcontrib-htmlhelp 2.0.0 pypi_0 pypi sphinxcontrib-jsmath 1.0.1 pypi_0 pypi sphinxcontrib-qthelp 1.0.3 pypi_0 pypi sphinxcontrib-serializinghtml 1.1.5 pypi_0 pypi sqlite 3.38.5 h707629a_0
stack-data 0.3.0 pypi_0 pypi stcal 1.0.0 pypi_0 pypi stdatamodels 0.4.3 pypi_0 pypi stpipe 0.4.1 pypi_0 pypi stsci-image 2.3.5 pypi_0 pypi stsci-imagestats 1.6.3 pypi_0 pypi stsci-stimage 0.2.5 pypi_0 pypi svo-filters 0.4.3 pypi_0 pypi tifffile 2022.5.4 pypi_0 pypi tk 8.6.12 h5d9f67b_0
tomli 2.0.1 pypi_0 pypi toolz 0.12.0 pypi_0 pypi tornado 6.2 pypi_0 pypi tqdm 4.64.0 pypi_0 pypi traitlets 5.3.0 pypi_0 pypi tweakwcs 0.7.4 pypi_0 pypi typing-extensions 4.3.0 pypi_0 pypi tzdata 2022a hda174b7_0
uncertainties 3.1.7 pypi_0 pypi urllib3 1.26.10 pypi_0 pypi wcwidth 0.2.5 pypi_0 pypi webencodings 0.5.1 pypi_0 pypi wheel 0.37.1 pyhd3eb1b0_0
wiimatch 0.2.1 pypi_0 pypi xarray 2022.3.0 pypi_0 pypi xz 5.2.5 hca72f7f_1
zipp 3.8.1 pypi_0 pypi zlib 1.2.12 h4dc903c_2

Code of Conduct

taylorbell57 commented 2 years ago

Looking at your error message, it seems your ywindow value is outside of the range of the NIRSpec detector (my best guess is you used a NIRCam template or something). You should choose values that select the region you want for background subtraction and source extraction (look at our NIRSpec template as a starting point).

taylorbell57 commented 2 years ago

Actually, it could have also been an issue with your xwindow values as well now that I think about it

taylorbell57 commented 2 years ago

Also, if you're looking at real data, you'll almost certainly want to update Eureka to v0.3 (only just released) as there are a significant number of bugs we've found and patched in the first few days of looking at real data

trappist1i commented 2 years ago

Thanks. I upgraded to v0.3 and the problem is still there. I'm well using the proper NIRSpec template (attached).

In base.py, I added a line to write the variable "key" before the command 'result = getitem(key)' at line 5055 that crashes, and the result is the array [ 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27]. So if I get well the error message "IndexError: index 16 is out of bounds for axis 0 with size 16", this vector is too long.

S3__ecf_screenshot
taylorbell57 commented 2 years ago

Ah okay, thanks for the ECF and the extra information about which indices are trying to be used. From that, it is clear that the y-axis is the issue. The only way I can see this happening is if the something went wrong with Stage 2 that resulted in the y-axis of the image being cropped. Can you confirm that you used slit_y_low -1 and slit_y_high 50 in the Stage 2 ECF? I know you didn't use the waverange settings for NIRSpec, so it's possible you also didn't use the slit_y parameters for NIRSpec. Make sure you used the NIRSpec S2 template as the settings do significantly differ between instruments

trappist1i commented 2 years ago

Looking at the Stage 3 ecf file, I understood that the array mentionned in the error message is the ywindow of the ecf file, i.e. the y dimension of the pixel selection box. As my Prism spectra have 16 pixels along the y-dimension, leaving [2,28] makes the code crash, which is totally normal. Sorry for this stupid mistake and thanks for the help! After having set ywindow to [1:16], Stage 3 works well.

taylorbell57 commented 2 years ago

My point though is that your PRISM spectra shouldn't need to only be 16 pixels tall - that's why we have those slit_y parameters in the Stage 2 ECF - to give you more pixels in the spatial direction to get a better background subtraction. Relatedly, you typically don't want to use the first one or few pixels and the last one or few pixels as they are especially noisy. I recommend you look at your Stage 2 ECF, as you likely missed the slit_y settings from the NIRSpec ECF template (and potentially other relevant skip settings)

AarynnCarter commented 2 years ago

As a quick note, the PRISM could only be 16 pixels tall if using the SUB512S subarray. If you aren't, definitely adjust the slit_y settings.

taylorbell57 commented 2 years ago

Ah, good to know Aarynn - I'm not that familiar with all the subarrays yet

taylorbell57 commented 2 years ago

From what I can tell, this issue has been resolved, so I am closing this now. Please re-open the issue if it is still relevant