lakeishamrr commented 2 years ago

Instrument

NIRSpec (Stages 1-3)

What happened?

I'm trying to run Stage 3 using the Stage 2 output from box but it's giving me a value error. I installed Eureka! using the "Install with Git and conda" instructions and all the files I'm using are from the GitHub. I was told it could be an Eureka! version issue if I used a Stage 2 output from a previous version but I'm not sure how to verify that and how to possibly fix it.

Error traceback output

WARNING: Unable to find an output metadata file from Eureka!'s S2 step in the inputdir: "Data/Stage2_Output/"! Assuming this S2 data was produced by the JWST pipeline instead.

Starting Stage 3 Reduction

Input directory: /Users/lakeishamramosrosado/Research/Data/Stage2_Output/ Output directory: /Users/lakeishamramosrosado/Research/Data/Stage3_Output/S3_2022-04-20_nirspec_template_ap6_bg7_run7/ Using ap=6, bg=7 Copying S3 control file

Found 1 data file(s) ending in calints.fits WARNING: Are you using real JWST data? If so, you should edit the flag_bg() function in nirspec.py and look at Issue #193 on Github! Reading file 1 of 1 WARNING: Manually setting INTSTART to 1 and INTEND to NINTS WARNING: The timestamps for the simulated NIRSpec data are currently hardcoded because they are not in the .fits files themselves Source position on detector is row 8. Automatically getting reference files to convert units to electrons Converting from brightness units MJy to electrons Traceback (most recent call last): File "/Users/lakeishamramosrosado/eureka_analysis/JWST/run_eureka_demo.py", line 17, in s3_meta = s3.reduceJWST(eventlabel,ecf_path = ecf_path) File "/Users/lakeishamramosrosado/opt/anaconda3/envs/eureka/lib/python3.9/site-packages/eureka/S3_data_reduction/s3_reduce.py", line 223, in reduceJWST data, meta = b2f.convert_to_e(data, meta, log) File "/Users/lakeishamramosrosado/opt/anaconda3/envs/eureka/lib/python3.9/site-packages/eureka/S3_data_reduction/bright2flux.py", line 240, in convert_to_e data = dn2electrons(data, meta) File "/Users/lakeishamramosrosado/opt/anaconda3/envs/eureka/lib/python3.9/site-packages/eureka/S3_data_reduction/bright2flux.py", line 83, in dn2electrons data.subdata *= subgain ValueError: operands could not be broadcast together with shapes (8192,19,350) (24,350) (8192,19,350)

What operating system are you using?

Mac OS Monterey 12.3.1

What version of Python are you running?

Python 3.9.12

What Python packages do you have installed?

packages in environment at /Users/lakeishamramosrosado/opt/anaconda3/envs/eureka:

Name Version Build Channel

alabaster 0.7.12 py_0 conda-forge aom 3.3.0 h96cf925_1 conda-forge appnope 0.1.3 pyhd8ed1ab_0 conda-forge arviz 0.11.2 pyhd8ed1ab_0 conda-forge asdf 2.10.1 pyhd8ed1ab_1 conda-forge asdf-standard 1.0.2 pyhd8ed1ab_0 conda-forge asdf-transform-schemas 0.2.2 pyhd8ed1ab_0 conda-forge asteval 0.9.23 pyhd8ed1ab_0 conda-forge astropy 5.0.4 py39h86b5767_0 conda-forge astropy-healpix 0.6 py39hc89836e_1 conda-forge astroquery 0.4.6 pypi_0 pypi astroscrappy 1.1.0 py39hc89836e_0 conda-forge asttokens 2.0.5 pyhd8ed1ab_0 conda-forge attrs 21.4.0 pyhd8ed1ab_0 conda-forge babel 2.9.1 pyh44b312d_0 conda-forge backcall 0.2.0 pyh9f0ad1d_0 conda-forge backports 1.0 py_2 conda-forge backports.functools_lru_cache 1.6.4 pyhd8ed1ab_0 conda-forge batman-package 2.4.8 py39h63b48b0_2 conda-forge beautifulsoup4 4.11.1 pyha770c72_0 conda-forge bleach 5.0.0 pyhd8ed1ab_0 conda-forge blosc 1.21.0 he49afe7_0 conda-forge bokeh 2.4.2 py39h6e9494a_1 conda-forge bottleneck 1.3.4 py39h86b5767_1 conda-forge brotli 1.0.9 h5eb16cf_7 conda-forge brotli-bin 1.0.9 h5eb16cf_7 conda-forge brotlipy 0.7.0 py39h63b48b0_1004 conda-forge brunsli 0.1 h046ec9c_0 conda-forge bzip2 1.0.8 h0d85af4_4 conda-forge c-ares 1.18.1 h0d85af4_0 conda-forge c-blosc2 2.0.4 ha1a4663_1 conda-forge ca-certificates 2022.3.29 hecd8cb5_1
cached-property 1.5.2 hd8ed1ab_1 conda-forge cached_property 1.5.2 pyha770c72_1 conda-forge ccdproc 2.3.0 pyhd8ed1ab_0 conda-forge certifi 2021.10.8 py39hecd8cb5_2
cffi 1.15.0 py39hc55c11b_1
cfitsio 4.1.0 h2c97ad1_0 conda-forge cftime 1.6.0 py39h86b5767_1 conda-forge charls 2.3.4 he49afe7_0 conda-forge charset-normalizer 2.0.12 pyhd8ed1ab_0 conda-forge click 8.1.2 py39h6e9494a_0 conda-forge cloudpickle 2.0.0 pyhd8ed1ab_0 conda-forge colorama 0.4.4 pyh9f0ad1d_0 conda-forge commonmark 0.9.1 py_0 conda-forge corner 2.2.1 pyhd8ed1ab_0 conda-forge coverage 6.3.2 py39h63b48b0_2 conda-forge crds 11.12.1 pypi_0 pypi cryptography 36.0.2 py39h1644bb1_1 conda-forge curl 7.82.0 h9f20792_0 conda-forge cycler 0.11.0 pyhd8ed1ab_0 conda-forge cython 0.29.28 py39hfd1d529_2 conda-forge cytoolz 0.11.2 py39h63b48b0_2 conda-forge dask-core 2022.4.1 pyhd8ed1ab_0 conda-forge decorator 5.1.1 pyhd8ed1ab_0 conda-forge defusedxml 0.7.1 pyhd8ed1ab_0 conda-forge docutils 0.17.1 py39h6e9494a_1 conda-forge drizzle 1.13.4 pypi_0 pypi dynesty 1.2.2 pyhd8ed1ab_0 conda-forge emcee 3.1.1 pyh8a188c0_0 conda-forge entrypoints 0.4 pyhd8ed1ab_0 conda-forge eureka 0.0.1 pypi_0 pypi executing 0.8.3 pyhd8ed1ab_0 conda-forge filelock 3.6.0 pypi_0 pypi fonttools 4.32.0 py39h63b48b0_0 conda-forge freetype 2.10.4 h4cff582_1 conda-forge fsspec 2022.3.0 pyhd8ed1ab_0 conda-forge future 0.18.2 py39h6e9494a_5 conda-forge george 0.4.0 py39h4d6be9b_0 conda-forge giflib 5.2.1 hbcb3906_2 conda-forge gwcs 0.17.1 pyhd8ed1ab_0 conda-forge h5py 3.6.0 nompi_py39hbc6cb89_100 conda-forge hdf4 4.2.15 hefd3b78_3 conda-forge hdf5 1.12.1 nompi_ha60fbc9_104 conda-forge html5lib 1.1 pypi_0 pypi hypothesis 6.43.3 pyhd8ed1ab_0 conda-forge idna 3.3 pyhd8ed1ab_0 conda-forge imagecodecs 2022.2.22 py39h42efd52_4 conda-forge imageio 2.17.0 pyhcf75d05_0 conda-forge imagesize 1.3.0 pyhd8ed1ab_0 conda-forge importlib-metadata 4.11.3 py39h6e9494a_1 conda-forge importlib-resources 5.7.1 pyhd8ed1ab_0 conda-forge importlib_metadata 4.11.3 hd8ed1ab_1 conda-forge importlib_resources 5.7.1 pyhd8ed1ab_0 conda-forge iniconfig 1.1.1 pyh9f0ad1d_0 conda-forge ipython 8.2.0 py39h6e9494a_0 conda-forge jbig 2.1 h0d85af4_2003 conda-forge jedi 0.18.1 py39h6e9494a_1 conda-forge jinja2 3.0.3 pyhd3eb1b0_0
jmespath 1.0.0 pyhd8ed1ab_0 conda-forge joblib 1.1.0 pyhd8ed1ab_0 conda-forge jpeg 9e h5eb16cf_1 conda-forge jsonschema 3.2.0 pyhd8ed1ab_3 conda-forge jupyter_client 7.2.2 pyhd8ed1ab_1 conda-forge jupyter_core 4.9.2 py39h6e9494a_0 conda-forge jupyterlab_pygments 0.2.2 pyhd8ed1ab_0 conda-forge jwst 1.3.3 pypi_0 pypi jxrlib 1.1 h35c211d_2 conda-forge keyring 23.5.0 pypi_0 pypi kiwisolver 1.4.2 py39h7248d28_1 conda-forge krb5 1.19.3 hb49756b_0 conda-forge lcms2 2.12 h577c468_0 conda-forge lerc 3.0 he49afe7_0 conda-forge libaec 1.0.6 he49afe7_0 conda-forge libavif 0.10.1 h5eb16cf_0 conda-forge libblas 3.9.0 14_osx64_openblas conda-forge libbrotlicommon 1.0.9 h5eb16cf_7 conda-forge libbrotlidec 1.0.9 h5eb16cf_7 conda-forge libbrotlienc 1.0.9 h5eb16cf_7 conda-forge libcblas 3.9.0 14_osx64_openblas conda-forge libcurl 7.82.0 h9f20792_0 conda-forge libcxx 13.0.1 hc203e6f_0 conda-forge libdeflate 1.10 h0d85af4_0 conda-forge libedit 3.1.20191231 h0678c8f_2 conda-forge libev 4.33 haf1e3a3_1 conda-forge libffi 3.3 hb1e8313_2
libgfortran 5.0.0 9_3_0_h6c81a4c_23 conda-forge libgfortran5 9.3.0 h6c81a4c_23 conda-forge liblapack 3.9.0 14_osx64_openblas conda-forge libnetcdf 4.8.1 nompi_h6609ca0_101 conda-forge libnghttp2 1.47.0 h942079c_0 conda-forge libopenblas 0.3.20 openmp_hb3cd9ec_0 conda-forge libpng 1.6.37 h7cec526_2 conda-forge libsodium 1.0.18 hbcb3906_1 conda-forge libssh2 1.10.0 h52ee1ee_2 conda-forge libtiff 4.3.0 h17f2ce3_3 conda-forge libwebp 1.2.2 h28dabe5_0 conda-forge libwebp-base 1.2.2 h0d85af4_1 conda-forge libxcb 1.13 h0d85af4_1004 conda-forge libzip 1.8.0 h8b0c345_1 conda-forge libzlib 1.2.11 h6c3fc93_1014 conda-forge libzopfli 1.0.3 h046ec9c_0 conda-forge llvm-openmp 13.0.1 hcb1a161_1 conda-forge lmfit 1.0.3 pyhd8ed1ab_0 conda-forge locket 0.2.0 py_2 conda-forge lxml 4.8.0 pypi_0 pypi lz4-c 1.9.3 he49afe7_1 conda-forge markdown-it-py 2.1.0 pypi_0 pypi markupsafe 2.1.1 py39h63b48b0_1 conda-forge matplotlib 3.5.1 py39h6e9494a_0 conda-forge matplotlib-base 3.5.1 py39hb07454d_0 conda-forge matplotlib-inline 0.1.3 pyhd8ed1ab_0 conda-forge mdit-py-plugins 0.3.0 pypi_0 pypi mdurl 0.1.1 pypi_0 pypi mistune 0.8.4 py39h89e85a6_1005 conda-forge mpi 1.0 openmpi conda-forge mpi4py 3.1.3 py39hd70dddc_1 conda-forge munkres 1.1.4 pyh9f0ad1d_0 conda-forge myst-parser 0.17.2 pypi_0 pypi nbclient 0.6.0 pyhd8ed1ab_0 conda-forge nbconvert 6.5.0 pyhd8ed1ab_0 conda-forge nbconvert-core 6.5.0 pyhd8ed1ab_0 conda-forge nbconvert-pandoc 6.5.0 pyhd8ed1ab_0 conda-forge nbformat 5.3.0 pyhd8ed1ab_0 conda-forge nbsphinx 0.8.8 pyhd8ed1ab_0 conda-forge ncurses 6.3 h96cf925_1 conda-forge nest-asyncio 1.5.5 pyhd8ed1ab_0 conda-forge netcdf4 1.5.8 nompi_py39he7d1c46_101 conda-forge networkx 2.8 pyhd8ed1ab_0 conda-forge numpy 1.22.3 py39hf56e92f_2 conda-forge numpydoc 1.2.1 pypi_0 pypi openjpeg 2.4.0 h6e7aa92_1 conda-forge openmpi 4.1.3 hc1c689a_101 conda-forge openssl 1.1.1n hca72f7f_0
packaging 21.3 pyhd8ed1ab_0 conda-forge pandas 1.4.2 py39hbd61c47_1 conda-forge pandoc 2.18 h694c41f_0 conda-forge pandocfilters 1.5.0 pyhd8ed1ab_0 conda-forge parsley 1.3 pypi_0 pypi parso 0.8.3 pyhd8ed1ab_0 conda-forge partd 1.2.0 pyhd8ed1ab_0 conda-forge pexpect 4.8.0 pyh9f0ad1d_2 conda-forge photutils 1.1.0 pypi_0 pypi pickleshare 0.7.5 py_1003 conda-forge pillow 9.1.0 py39hd2c7aa1_2 conda-forge pip 21.2.4 py39hecd8cb5_0
pluggy 1.0.0 py39h6e9494a_3 conda-forge poppy 1.0.2 pypi_0 pypi prompt-toolkit 3.0.29 pyha770c72_0 conda-forge psutil 5.9.0 py39h63b48b0_1 conda-forge pthread-stubs 0.4 hc929b4f_1001 conda-forge ptyprocess 0.7.0 pyhd3deb0d_0 conda-forge pure_eval 0.2.2 pyhd8ed1ab_0 conda-forge py 1.11.0 pyh6c4a22f_0 conda-forge pycparser 2.21 pyhd8ed1ab_0 conda-forge pyerfa 2.0.0.1 py39h86b5767_2 conda-forge pygments 2.11.2 pyhd8ed1ab_0 conda-forge pyopenssl 22.0.0 pyhd8ed1ab_0 conda-forge pyparsing 3.0.8 pyhd8ed1ab_0 conda-forge pyrsistent 0.18.1 py39h63b48b0_1 conda-forge pysocks 1.7.1 py39h6e9494a_5 conda-forge pysynphot 2.0.0 pypi_0 pypi pytest 7.1.1 py39h6e9494a_1 conda-forge pytest-arraydiff 0.3 py_0 conda-forge pytest-astropy 0.10.0 pyhd8ed1ab_0 conda-forge pytest-astropy-header 0.1.2 py_0 conda-forge pytest-cov 3.0.0 pyhd8ed1ab_0 conda-forge pytest-doctestplus 0.12.0 pyhd8ed1ab_0 conda-forge pytest-filter-subpackage 0.1.1 py_0 conda-forge pytest-mock 3.7.0 pyhd8ed1ab_0 conda-forge pytest-openfiles 0.5.0 py_0 conda-forge pytest-remotedata 0.3.3 pyhd8ed1ab_0 conda-forge python 3.9.12 hdfd78df_0
python-dateutil 2.8.2 pyhd8ed1ab_0 conda-forge python-fastjsonschema 2.15.3 pyhd8ed1ab_0 conda-forge python_abi 3.9 2_cp39 conda-forge pytz 2022.1 pyhd8ed1ab_0 conda-forge pyvo 1.3 pypi_0 pypi pywavelets 1.3.0 py39h86b5767_1 conda-forge pyyaml 6.0 py39h63b48b0_4 conda-forge pyzmq 22.3.0 py39hc2dc7ec_2 conda-forge readline 8.1 h05e3726_0 conda-forge recommonmark 0.7.1 pyhd8ed1ab_0 conda-forge reproject 0.8 py39hc89836e_2 conda-forge requests 2.27.1 pyhd8ed1ab_0 conda-forge schwimmbad 0.3.2 py39h6e9494a_2 conda-forge scikit-image 0.19.2 py39h4d6be9b_0 conda-forge scikit-learn 1.0.2 py39hd4eea88_0 conda-forge scipy 1.8.0 py39h056f1c0_1 conda-forge semantic_version 2.9.0 pyhd8ed1ab_0 conda-forge setuptools 62.1.0 py39h6e9494a_0 conda-forge setuptools-scm 6.4.2 pyhd8ed1ab_0 conda-forge six 1.16.0 pyh6c4a22f_0 conda-forge snappy 1.1.8 hb1e8313_3 conda-forge snowballstemmer 2.2.0 pyhd8ed1ab_0 conda-forge sortedcontainers 2.4.0 pyhd8ed1ab_0 conda-forge soupsieve 2.3.1 pyhd8ed1ab_0 conda-forge spherical-geometry 1.2.22 pypi_0 pypi sphinx 4.5.0 pyh6c4a22f_0 conda-forge sphinx-rtd-theme 1.0.0 pypi_0 pypi sphinxcontrib-applehelp 1.0.2 py_0 conda-forge sphinxcontrib-devhelp 1.0.2 py_0 conda-forge sphinxcontrib-htmlhelp 2.0.0 pyhd8ed1ab_0 conda-forge sphinxcontrib-jsmath 1.0.1 py_0 conda-forge sphinxcontrib-qthelp 1.0.3 py_0 conda-forge sphinxcontrib-serializinghtml 1.1.5 pyhd8ed1ab_2 conda-forge sqlite 3.38.2 hb516253_0 conda-forge stack_data 0.2.0 pyhd8ed1ab_0 conda-forge stcal 0.6.2 pypi_0 pypi stdatamodels 0.2.4 pypi_0 pypi stpipe 0.3.3 pypi_0 pypi stsci-image 2.3.5 pypi_0 pypi stsci-imagestats 1.6.3 pypi_0 pypi stsci-stimage 0.2.5 pypi_0 pypi svo-filters 0.4.1 pypi_0 pypi threadpoolctl 3.1.0 pyh8a188c0_0 conda-forge tifffile 2022.4.8 pyhd8ed1ab_0 conda-forge tinycss2 1.1.1 pyhd8ed1ab_0 conda-forge tk 8.6.12 h5dbffcc_0 conda-forge toml 0.10.2 pyhd8ed1ab_0 conda-forge tomli 2.0.1 pyhd8ed1ab_0 conda-forge toolz 0.11.2 pyhd8ed1ab_0 conda-forge tornado 6.1 py39h63b48b0_3 conda-forge tqdm 4.64.0 pyhd8ed1ab_0 conda-forge traitlets 5.1.1 pyhd8ed1ab_0 conda-forge tweakwcs 0.7.4 pypi_0 pypi typing_extensions 4.2.0 pyha770c72_1 conda-forge tzdata 2022a h191b570_0 conda-forge uncertainties 3.1.6 pyhd8ed1ab_0 conda-forge unicodedata2 14.0.0 py39h63b48b0_1 conda-forge urllib3 1.26.9 pyhd8ed1ab_0 conda-forge wcwidth 0.2.5 pyh9f0ad1d_2 conda-forge webencodings 0.5.1 py_1 conda-forge wheel 0.37.1 pyhd8ed1ab_0 conda-forge xarray 2022.3.0 pyhd8ed1ab_0 conda-forge xorg-libxau 1.0.9 h35c211d_0 conda-forge xorg-libxdmcp 1.1.3 h35c211d_0 conda-forge xz 5.2.5 haf1e3a3_1 conda-forge yaml 0.2.5 h0d85af4_2 conda-forge zeromq 4.3.4 he49afe7_1 conda-forge zfp 0.5.5 h4a89273_8 conda-forge zipp 3.8.0 pyhd8ed1ab_0 conda-forge zlib 1.2.11 h6c3fc93_1014 conda-forge zstd 1.5.2 h582d3a0_0 conda-forge

Code of Conduct

[X] I agree to follow this project's Code of Conduct

AarynnCarter commented 2 years ago

Hi @lakeishamrr, sorry for taking a while to get back to you.

A few initial questions / thoughts:

Is this using the most recent NIRSpec data? (https://app.box.com/s/tj1jnivn9ekiyhecl5up7mkg8xrd1htl/folder/154382679630)
The Stage 2 data on the box is mostly for demonstration purposes, I would advise starting with the Stage 1 products (see the Box readme for more information).
It seems the following line is where the issue arises :
```
data.subdata *= subgain
ValueError: operands could not be broadcast together with shapes (8192,19,350) (24,350) (8192,19,350)
```
I would suggest taking a look at your Stage 3 .ecf to ensure that the ywindow, bg_hw, and spec_hw are suitable for the Stage 2 data you are using.
I would also suggest switching to the Tiny dataset while you are getting things running (it's much faster), and then switching to the full dataset at a later time.