search

kevin218 / Eureka

Eureka! is a data reduction and analysis pipeline intended for time-series observations with JWST.
https://eurekadocs.readthedocs.io/
MIT License
56 stars 43 forks source link

Stage 3 optical spectrum extraction and obtaining 2-D Spectrum plot with a linear wavelength x-axis for NIRSpec G395H #668

Open kohhei315 opened 1 week ago

kohhei315 commented 1 week ago

Instrument

NIRSpec (Stages 1-3)

What happened?

I am currently Running Stage 3 of Eureka! Pipeline, specifically in generating 2-D spectrum plot with linear wavelength x-axis. It seems that the normalized flux from integration number 0 to 155 only has a value in the wavelength around 2.8 μm (x-axis is from 2.8 ~ 3.8 μm) shown in the given png file. However, it must have some flux in all the region across x-axis (wavelength). I thought that the results from optimal spectrum extraction contributes to this error and so I would like to also provide my results for optimal spectrum extraction below.

fig3101-2D_LC fig3303_file0_int000_Profile

I will also provide some details of the code below during optimal spectrum extraction.

Performing optimal spectral extraction...
  0%|                                                   | 0/155 [00:00<?, ?it/s]
Figure(800x400)
...
  1%|▎                                        | 1/155 [00:32<1:22:12, 32.03s/it]Figure(800x400)
...
100%|█████████████████████████████████████████| 155/155 [20:02<00:00,  7.76s/it]
  Creating figures for optimal spectral extraction
  0%|                                                     | 0/5 [00:00<?, ?it/s]Figure(640x480)
 20%|█████████                                    | 1/5 [00:00<00:03,  1.16it/s]Figure(640x480)
 40%|██████████████████                           | 2/5 [00:01<00:02,  1.21it/s]Figure(640x480)
 60%|███████████████████████████                  | 3/5 [00:02<00:01,  1.22it/s]Figure(640x480)
 80%|████████████████████████████████████         | 4/5 [00:03<00:00,  1.16it/s]Figure(640x480)
100%|█████████████████████████████████████████████| 5/5 [00:04<00:00,  1.19it/s]
Figure(800x400)

Error traceback output

No response

What operating system are you using?

macOS Sonoma 14.5

What version of Python are you running?

Python 3.11.9

What Python packages do you have installed?

# Name                    Version                   Build  Channel
alabaster                 0.7.16                   pypi_0    pypi
anyio                     4.4.0              pyhd8ed1ab_0    conda-forge
appnope                   0.1.4              pyhd8ed1ab_0    conda-forge
argon2-cffi               23.1.0             pyhd8ed1ab_0    conda-forge
argon2-cffi-bindings      21.2.0          py311h2725bcf_4    conda-forge
arrow                     1.3.0              pyhd8ed1ab_0    conda-forge
asciitree                 0.3.3                    pypi_0    pypi
asdf                      3.2.0                    pypi_0    pypi
asdf-astropy              0.6.1                    pypi_0    pypi
asdf-coordinates-schemas  0.3.0                    pypi_0    pypi
asdf-standard             1.1.1                    pypi_0    pypi
asdf-transform-schemas    0.5.0                    pypi_0    pypi
asdf-wcs-schemas          0.4.0                    pypi_0    pypi
asteval                   0.9.33                   pypi_0    pypi
astraeus                  0.3                      pypi_0    pypi
astropy                   6.1.1                    pypi_0    pypi
astropy-healpix           1.0.3                    pypi_0    pypi
astropy-iers-data         0.2024.6.17.0.31.35          pypi_0    pypi
astroquery                0.4.7                    pypi_0    pypi
astroscrappy              1.2.0                    pypi_0    pypi
asttokens                 2.4.1              pyhd8ed1ab_0    conda-forge
async-lru                 2.0.4              pyhd8ed1ab_0    conda-forge
attrs                     23.2.0             pyh71513ae_0    conda-forge
babel                     2.14.0             pyhd8ed1ab_0    conda-forge
backports-tarfile         1.2.0                    pypi_0    pypi
batman-package            2.4.9                    pypi_0    pypi
bayesicfitting            3.2.1                    pypi_0    pypi
beautifulsoup4            4.12.3             pyha770c72_0    conda-forge
bleach                    6.1.0              pyhd8ed1ab_0    conda-forge
bokeh                     2.4.3                    pypi_0    pypi
bottleneck                1.4.0                    pypi_0    pypi
brotli-python             1.1.0           py311hdf8f085_1    conda-forge
bzip2                     1.0.8                h10d778d_5    conda-forge
ca-certificates           2024.6.2             h8857fd0_0    conda-forge
cached-property           1.5.2                hd8ed1ab_1    conda-forge
cached_property           1.5.2              pyha770c72_1    conda-forge
ccdproc                   2.4.2                    pypi_0    pypi
celerite                  0.4.2                    pypi_0    pypi
certifi                   2024.6.2           pyhd8ed1ab_0    conda-forge
cffi                      1.16.0          py311hc0b63fd_0    conda-forge
cftime                    1.6.4                    pypi_0    pypi
charset-normalizer        3.3.2              pyhd8ed1ab_0    conda-forge
click                     8.1.7                    pypi_0    pypi
cloudpickle               3.0.0                    pypi_0    pypi
comm                      0.2.2              pyhd8ed1ab_0    conda-forge
contourpy                 1.2.1                    pypi_0    pypi
corner                    2.2.2                    pypi_0    pypi
crds                      11.17.25                 pypi_0    pypi
cycler                    0.12.1                   pypi_0    pypi
dask                      2024.6.2                 pypi_0    pypi
debugpy                   1.8.1           py311hdd0406b_0    conda-forge
decorator                 5.1.1              pyhd8ed1ab_0    conda-forge
defusedxml                0.7.1              pyhd8ed1ab_0    conda-forge
dill                      0.3.8                    pypi_0    pypi
docutils                  0.21.2                   pypi_0    pypi
drizzle                   1.15.2                   pypi_0    pypi
dynesty                   2.1.3                    pypi_0    pypi
emcee                     3.1.6                    pypi_0    pypi
entrypoints               0.4                pyhd8ed1ab_0    conda-forge
eureka                    0.11.dev0+g696df7c8.d20240622          pypi_0    pypi
exceptiongroup            1.2.0              pyhd8ed1ab_2    conda-forge
executing                 2.0.1              pyhd8ed1ab_0    conda-forge
exotic-ld                 3.2.0                    pypi_0    pypi
fasteners                 0.19                     pypi_0    pypi
filelock                  3.15.3                   pypi_0    pypi
fonttools                 4.53.0                   pypi_0    pypi
fqdn                      1.5.1              pyhd8ed1ab_0    conda-forge
fsspec                    2024.6.0                 pypi_0    pypi
future                    1.0.0                    pypi_0    pypi
george                    0.4.1                    pypi_0    pypi
gwcs                      0.21.0                   pypi_0    pypi
h11                       0.14.0             pyhd8ed1ab_0    conda-forge
h2                        4.1.0              pyhd8ed1ab_0    conda-forge
h5netcdf                  1.3.0                    pypi_0    pypi
h5py                      3.11.0                   pypi_0    pypi
hpack                     4.0.0              pyh9f0ad1d_0    conda-forge
html5lib                  1.1                      pypi_0    pypi
httpcore                  1.0.5              pyhd8ed1ab_0    conda-forge
httpx                     0.27.0             pyhd8ed1ab_0    conda-forge
hyperframe                6.0.1              pyhd8ed1ab_0    conda-forge
idna                      3.7                pyhd8ed1ab_0    conda-forge
imageio                   2.34.1                   pypi_0    pypi
imagesize                 1.4.1                    pypi_0    pypi
importlib-metadata        7.2.0              pyha770c72_0    conda-forge
importlib_metadata        7.2.0                hd8ed1ab_0    conda-forge
importlib_resources       6.4.0              pyhd8ed1ab_0    conda-forge
iniconfig                 2.0.0                    pypi_0    pypi
ipykernel                 6.29.4             pyh57ce528_0    conda-forge
ipython                   8.25.0             pyh707e725_0    conda-forge
isoduration               20.11.0            pyhd8ed1ab_0    conda-forge
jaraco-classes            3.4.0                    pypi_0    pypi
jaraco-context            5.3.0                    pypi_0    pypi
jaraco-functools          4.0.1                    pypi_0    pypi
jedi                      0.19.1             pyhd8ed1ab_0    conda-forge
jinja2                    3.1.4              pyhd8ed1ab_0    conda-forge
jmespath                  1.0.1                    pypi_0    pypi
json5                     0.9.25             pyhd8ed1ab_0    conda-forge
jsonpointer               3.0.0           py311h6eed73b_0    conda-forge
jsonschema                4.22.0             pyhd8ed1ab_0    conda-forge
jsonschema-specifications 2023.12.1          pyhd8ed1ab_0    conda-forge
jsonschema-with-format-nongpl 4.22.0             pyhd8ed1ab_0    conda-forge
jupyter-lsp               2.2.5              pyhd8ed1ab_0    conda-forge
jupyter_client            8.6.2              pyhd8ed1ab_0    conda-forge
jupyter_core              5.7.2           py311h6eed73b_0    conda-forge
jupyter_events            0.10.0             pyhd8ed1ab_0    conda-forge
jupyter_server            2.14.1             pyhd8ed1ab_0    conda-forge
jupyter_server_terminals  0.5.3              pyhd8ed1ab_0    conda-forge
jupyterlab                4.2.2              pyhd8ed1ab_0    conda-forge
jupyterlab_pygments       0.3.0              pyhd8ed1ab_1    conda-forge
jupyterlab_server         2.27.2             pyhd8ed1ab_0    conda-forge
jwst                      1.11.4                   pypi_0    pypi
keyring                   25.2.1                   pypi_0    pypi
kiwisolver                1.4.5                    pypi_0    pypi
krb5                      1.21.2               hb884880_0    conda-forge
lazy-loader               0.4                      pypi_0    pypi
libcxx                    17.0.6               h88467a6_0    conda-forge
libedit                   3.1.20191231         h0678c8f_2    conda-forge
libexpat                  2.6.2                h73e2aa4_0    conda-forge
libffi                    3.4.2                h0d85af4_5    conda-forge
libsodium                 1.0.18               hbcb3906_1    conda-forge
libsqlite                 3.46.0               h1b8f9f3_0    conda-forge
libzlib                   1.3.1                h87427d6_1    conda-forge
lmfit                     1.3.1                    pypi_0    pypi
locket                    1.0.0                    pypi_0    pypi
markupsafe                2.1.5           py311he705e18_0    conda-forge
matplotlib                3.9.0                    pypi_0    pypi
matplotlib-inline         0.1.7              pyhd8ed1ab_0    conda-forge
mistune                   3.0.2              pyhd8ed1ab_0    conda-forge
more-itertools            10.3.0                   pypi_0    pypi
nbclient                  0.10.0             pyhd8ed1ab_0    conda-forge
nbconvert-core            7.16.4             pyhd8ed1ab_1    conda-forge
nbformat                  5.10.4             pyhd8ed1ab_0    conda-forge
ncurses                   6.5                  h5846eda_0    conda-forge
nest-asyncio              1.6.0              pyhd8ed1ab_0    conda-forge
netcdf4                   1.7.1                    pypi_0    pypi
networkx                  3.3                      pypi_0    pypi
notebook                  7.2.1              pyhd8ed1ab_0    conda-forge
notebook-shim             0.2.4              pyhd8ed1ab_0    conda-forge
numcodecs                 0.12.1                   pypi_0    pypi
numpy                     1.24.4                   pypi_0    pypi
numpydoc                  1.7.0                    pypi_0    pypi
opencv-python-headless    4.10.0.84                pypi_0    pypi
openssl                   3.3.1                h87427d6_0    conda-forge
overrides                 7.7.0              pyhd8ed1ab_0    conda-forge
packaging                 24.1               pyhd8ed1ab_0    conda-forge
pandas                    2.2.2                    pypi_0    pypi
pandocfilters             1.5.0              pyhd8ed1ab_0    conda-forge
parsley                   1.3                      pypi_0    pypi
parso                     0.8.4              pyhd8ed1ab_0    conda-forge
partd                     1.4.2                    pypi_0    pypi
pexpect                   4.9.0              pyhd8ed1ab_0    conda-forge
photutils                 1.12.0                   pypi_0    pypi
pickleshare               0.7.5                   py_1003    conda-forge
pillow                    10.3.0                   pypi_0    pypi
pip                       24.0               pyhd8ed1ab_0    conda-forge
pkgutil-resolve-name      1.3.10             pyhd8ed1ab_1    conda-forge
platformdirs              4.2.2              pyhd8ed1ab_0    conda-forge
pluggy                    1.5.0                    pypi_0    pypi
poppy                     1.1.1                    pypi_0    pypi
prometheus_client         0.20.0             pyhd8ed1ab_0    conda-forge
prompt-toolkit            3.0.47             pyha770c72_0    conda-forge
psutil                    5.9.8           py311he705e18_0    conda-forge
ptyprocess                0.7.0              pyhd3deb0d_0    conda-forge
pure_eval                 0.2.2              pyhd8ed1ab_0    conda-forge
pycparser                 2.22               pyhd8ed1ab_0    conda-forge
pyerfa                    2.0.1.4                  pypi_0    pypi
pygments                  2.18.0             pyhd8ed1ab_0    conda-forge
pyobjc-core               10.3.1          py311h9d23797_0    conda-forge
pyobjc-framework-cocoa    10.3.1          py311h9d23797_0    conda-forge
pyparsing                 3.1.2                    pypi_0    pypi
pysocks                   1.7.1              pyha2e5f31_6    conda-forge
pysynphot                 2.0.0                    pypi_0    pypi
pytest                    8.2.2                    pypi_0    pypi
python                    3.11.9          h657bba9_0_cpython    conda-forge
python-dateutil           2.9.0              pyhd8ed1ab_0    conda-forge
python-fastjsonschema     2.20.0             pyhd8ed1ab_0    conda-forge
python-json-logger        2.0.7              pyhd8ed1ab_0    conda-forge
python_abi                3.11                    4_cp311    conda-forge
pytz                      2024.1             pyhd8ed1ab_0    conda-forge
pyvo                      1.5.2                    pypi_0    pypi
pyyaml                    6.0.1           py311h2725bcf_1    conda-forge
pyzmq                     26.0.3          py311h89e2aaa_0    conda-forge
readline                  8.2                  h9e318b2_1    conda-forge
referencing               0.35.1             pyhd8ed1ab_0    conda-forge
reproject                 0.13.1                   pypi_0    pypi
requests                  2.32.3             pyhd8ed1ab_0    conda-forge
rfc3339-validator         0.1.4              pyhd8ed1ab_0    conda-forge
rfc3986-validator         0.1.1              pyh9f0ad1d_0    conda-forge
rpds-py                   0.18.1          py311h295b1db_0    conda-forge
scikit-image              0.24.0                   pypi_0    pypi
scipy                     1.9.3                    pypi_0    pypi
semantic-version          2.10.0                   pypi_0    pypi
send2trash                1.8.3              pyh31c8845_0    conda-forge
setuptools                70.1.0             pyhd8ed1ab_0    conda-forge
setuptools-scm            8.1.0                    pypi_0    pypi
six                       1.16.0             pyh6c4a22f_0    conda-forge
sniffio                   1.3.1              pyhd8ed1ab_0    conda-forge
snowballstemmer           2.2.0                    pypi_0    pypi
soupsieve                 2.5                pyhd8ed1ab_1    conda-forge
spherical-geometry        1.3.2                    pypi_0    pypi
sphinx                    7.3.7                    pypi_0    pypi
sphinxcontrib-applehelp   1.0.8                    pypi_0    pypi
sphinxcontrib-devhelp     1.0.6                    pypi_0    pypi
sphinxcontrib-htmlhelp    2.0.5                    pypi_0    pypi
sphinxcontrib-jsmath      1.0.1                    pypi_0    pypi
sphinxcontrib-qthelp      1.0.7                    pypi_0    pypi
sphinxcontrib-serializinghtml 1.1.10                   pypi_0    pypi
stack_data                0.6.2              pyhd8ed1ab_0    conda-forge
stcal                     1.4.4                    pypi_0    pypi
stdatamodels              1.7.2                    pypi_0    pypi
stpipe                    0.5.2                    pypi_0    pypi
stsci-image               2.3.9                    pypi_0    pypi
stsci-imagestats          1.8.3                    pypi_0    pypi
stsci-stimage             0.2.9                    pypi_0    pypi
svo-filters               0.4.4                    pypi_0    pypi
tabulate                  0.9.0                    pypi_0    pypi
terminado                 0.18.1             pyh31c8845_0    conda-forge
tifffile                  2024.6.18                pypi_0    pypi
tinycss2                  1.3.0              pyhd8ed1ab_0    conda-forge
tk                        8.6.13               h1abcd95_1    conda-forge
tomli                     2.0.1              pyhd8ed1ab_0    conda-forge
toolz                     0.12.1                   pypi_0    pypi
tornado                   6.4.1           py311h72ae277_0    conda-forge
tqdm                      4.66.4                   pypi_0    pypi
traitlets                 5.14.3             pyhd8ed1ab_0    conda-forge
tweakwcs                  0.8.7                    pypi_0    pypi
types-python-dateutil     2.9.0.20240316     pyhd8ed1ab_0    conda-forge
typing-extensions         4.12.2               hd8ed1ab_0    conda-forge
typing_extensions         4.12.2             pyha770c72_0    conda-forge
typing_utils              0.1.0              pyhd8ed1ab_0    conda-forge
tzdata                    2024.1                   pypi_0    pypi
uncertainties             3.2.1                    pypi_0    pypi
uri-template              1.3.0              pyhd8ed1ab_0    conda-forge
urllib3                   2.2.2              pyhd8ed1ab_0    conda-forge
wcwidth                   0.2.13             pyhd8ed1ab_0    conda-forge
webcolors                 24.6.0             pyhd8ed1ab_0    conda-forge
webencodings              0.5.1              pyhd8ed1ab_2    conda-forge
websocket-client          1.8.0              pyhd8ed1ab_0    conda-forge
wheel                     0.43.0             pyhd8ed1ab_1    conda-forge
wiimatch                  0.3.2                    pypi_0    pypi
xarray                    2024.6.0                 pypi_0    pypi
xz                        5.2.6                h775f41a_0    conda-forge
yaml                      0.2.5                h0d85af4_2    conda-forge
zarr                      2.18.2                   pypi_0    pypi
zeromq                    4.3.5                hde137ed_4    conda-forge
zipp                      3.19.2             pyhd8ed1ab_0    conda-forge

Code of Conduct

kevin218 commented 1 week ago

It looks like most of your pixels are getting flagged as bad. You can start by adjusting the settings in your Stage 3 file to get cleaner results.

taylorbell57 commented 1 week ago

I agree with Kevin that this is likely not a bug with Eureka but more likely an issue with how you have setup you ECF. I recommend you look carefully at all of your plots and ECF settings to try to troubleshoot the issue yourself. I recommend you set isplots_S3 to 5 and nplots to 5 so that you get several useful troubleshooting plots. And to help you understand the ECF settings, please take a look at our documentation at https://eurekadocs.readthedocs.io/en/latest/ecf.html

If you're still unable to resolve the issue yourself after carefully looking at all the plots and ECF settings, can copy-paste the contents of your Stage 3 ECF; that'll make it easier for us to help you troubleshoot

kohhei315 commented 1 week ago

Thank you for your suggestions. I will change some settings in stage3 ecf files and redo the process. I will let send the details about my defined parameters in ecf if I have more troubles in the output results.