search

kevin218 / Eureka

Eureka! is a data reduction and analysis pipeline intended for time-series observations with JWST.
https://eurekadocs.readthedocs.io/
MIT License
58 stars 45 forks source link

[Bug]: Unable to find any "rampfitstep.fits" files #525

Closed Witchblade101 closed 1 year ago

Witchblade101 commented 1 year ago

FAQ check

Instrument

NIRISS (Stages 1-3)

What happened?

Stage 2 crashed with an error that it couldn't find any rampfitstep.fits files. In my Stage1 output directory it looks like they are now rate.fits and rateint.fits files, same as the JWST pipeline outputs.

Error traceback output

(eureka) minmei:w17b dlong$ python run_eureka.py

Starting Stage 1 Processing Input directory: /Users/dlong/DataAnalysis/JWST/eureka/w17b/test/ Output directory: /Users/dlong/DataAnalysis/JWST/eureka/w17b/Stage1/S1_2023-04-24_wasp17b_run1/ Copying S1 control file

Found 1 data file(s) ending in uncal.fits Starting file 1 of 1: jw01353101001_04101_00001-seg001_nis_uncal.fits 2023-04-24 17:32:27,750 - stpipe.EurekaS1Pipeline - INFO - EurekaS1Pipeline instance created. 2023-04-24 17:32:27,751 - stpipe.EurekaS1Pipeline.group_scale - INFO - GroupScaleStep instance created. 2023-04-24 17:32:27,752 - stpipe.EurekaS1Pipeline.dq_init - INFO - DQInitStep instance created. 2023-04-24 17:32:27,753 - stpipe.EurekaS1Pipeline.saturation - INFO - SaturationStep instance created. 2023-04-24 17:32:27,754 - stpipe.EurekaS1Pipeline.ipc - INFO - IPCStep instance created. 2023-04-24 17:32:27,755 - stpipe.EurekaS1Pipeline.superbias - INFO - SuperBiasStep instance created. 2023-04-24 17:32:27,756 - stpipe.EurekaS1Pipeline.refpix - INFO - RefPixStep instance created. 2023-04-24 17:32:27,757 - stpipe.EurekaS1Pipeline.rscd - INFO - RscdStep instance created. 2023-04-24 17:32:27,758 - stpipe.EurekaS1Pipeline.firstframe - INFO - FirstFrameStep instance created. 2023-04-24 17:32:27,759 - stpipe.EurekaS1Pipeline.lastframe - INFO - LastFrameStep instance created. 2023-04-24 17:32:27,759 - stpipe.EurekaS1Pipeline.linearity - INFO - LinearityStep instance created. 2023-04-24 17:32:27,761 - stpipe.EurekaS1Pipeline.dark_current - INFO - DarkCurrentStep instance created. 2023-04-24 17:32:27,762 - stpipe.EurekaS1Pipeline.reset - INFO - ResetStep instance created. 2023-04-24 17:32:27,763 - stpipe.EurekaS1Pipeline.persistence - INFO - PersistenceStep instance created. 2023-04-24 17:32:27,764 - stpipe.EurekaS1Pipeline.jump - INFO - JumpStep instance created. 2023-04-24 17:32:27,765 - stpipe.EurekaS1Pipeline.ramp_fit - INFO - RampFitStep instance created. 2023-04-24 17:32:27,766 - stpipe.EurekaS1Pipeline.gain_scale - INFO - GainScaleStep instance created. 2023-04-24 17:32:27,769 - stpipe.Eureka_SuperBiasStep - INFO - Eureka_SuperBiasStep instance created. 2023-04-24 17:32:27,770 - stpipe.Eureka_RampFitStep - INFO - Eureka_RampFitStep instance created. 2023-04-24 17:32:27,863 - stpipe.EurekaS1Pipeline - INFO - Step EurekaS1Pipeline running with args ('/Users/dlong/DataAnalysis/JWST/eureka/w17b/test/jw01353101001_04101_00001-seg001_nis_uncal.fits',).

2023-04-24 17:44:36,101 - stpipe.EurekaS1Pipeline.gain_scale - INFO - GAINFACT not found in gain reference file 2023-04-24 17:44:36,102 - stpipe.EurekaS1Pipeline.gain_scale - INFO - Step will be skipped 2023-04-24 17:44:36,109 - stpipe.EurekaS1Pipeline.gain_scale - INFO - Step gain_scale done 2023-04-24 17:44:40,077 - stpipe.EurekaS1Pipeline - INFO - Saved model in /Users/dlong/DataAnalysis/JWST/eureka/w17b/Stage1/S1_2023-04-24_wasp17b_run1/jw01353101001_04101_00001-seg001_nis_rateints.fits 2023-04-24 17:44:40,077 - stpipe.EurekaS1Pipeline - INFO - ... ending calwebb_detector1 2023-04-24 17:44:40,077 - stpipe.EurekaS1Pipeline - INFO - Results used CRDS context: jwst_1078.pmap 2023-04-24 17:44:40,192 - stpipe.EurekaS1Pipeline - INFO - Saved model in /Users/dlong/DataAnalysis/JWST/eureka/w17b/Stage1/S1_2023-04-24_wasp17b_run1/jw01353101001_04101_00001-seg001_nis_rate.fits 2023-04-24 17:44:40,192 - stpipe.EurekaS1Pipeline - INFO - Step EurekaS1Pipeline done Total time (min): 12.21 Saving Metadata Starting Stage 2 Reduction Input directory: /Users/dlong/DataAnalysis/JWST/eureka/w17b/Stage1/S1_2023-04-24_wasp17b_run1/ Output directory: /Users/dlong/DataAnalysis/JWST/eureka/w17b/Stage2/S2_2023-04-24_wasp17b_run1/ Copying S2 control file Traceback (most recent call last): File "/Users/dlong/DataAnalysis/JWST/eureka/w17b/run_eureka.py", line 25, in s2_meta = s2.calibrateJWST(eventlabel, ecf_path=ecf_path) File "/Users/dlong/Eureka/src/eureka/S2_calibrations/s2_calibrate.py", line 115, in calibrateJWST meta = util.readfiles(meta, log) File "/Users/dlong/Eureka/src/eureka/lib/util.py", line 66, in readfiles raise AssertionError(f'Unable to find any "{meta.suffix}.fits" files ' AssertionError: Unable to find any "rampfitstep.fits" files in the inputdir: "/Users/dlong/DataAnalysis/JWST/eureka/w17b/Stage1/S1_2023-04-24_wasp17b_run1/"! You likely need to change the inputdir in S2_wasp17b.ecf to point to the folder containing the "rampfitstep.fits" files. (eureka) minmei:w17b dlong$ /Users/dlong/miniconda3/envs/eureka/lib/python3.9/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 32 leaked semaphore objects to clean up at shutdown warnings.warn('resource_tracker: There appear to be %d ' ### What operating system are you using? MacOS 12.6.5 ### What version of Python are you running? Python 3.9.7 ### What Python packages do you have installed? # packages in environment at /Users/dlong/miniconda3/envs/eureka: # # Name Version Build Channel aesara-theano-fallback 0.1.0 pypi_0 pypi alabaster 0.7.13 pypi_0 pypi anyio 3.6.2 pypi_0 pypi appnope 0.1.3 pypi_0 pypi argon2-cffi 21.3.0 pypi_0 pypi argon2-cffi-bindings 21.2.0 pypi_0 pypi arrow 1.2.3 pypi_0 pypi arviz 0.12.1 pypi_0 pypi asdf 2.15.0 pypi_0 pypi asdf-astropy 0.4.0 pypi_0 pypi asdf-coordinates-schemas 0.2.0 pypi_0 pypi asdf-standard 1.0.3 pypi_0 pypi asdf-transform-schemas 0.3.0 pypi_0 pypi asdf-unit-schemas 0.1.0 pypi_0 pypi asdf-wcs-schemas 0.1.1 pypi_0 pypi asteval 0.9.29 pypi_0 pypi astraeus 0.3 pypi_0 pypi astropy 5.2.2 pypi_0 pypi astropy-healpix 0.7 pypi_0 pypi astroquery 0.4.6 pypi_0 pypi astroscrappy 1.1.0 pypi_0 pypi asttokens 2.2.1 pypi_0 pypi attrs 23.1.0 pypi_0 pypi babel 2.12.1 pypi_0 pypi backcall 0.2.0 pypi_0 pypi batman-package 2.4.9 pypi_0 pypi bayesicfitting 3.1.1 pypi_0 pypi beautifulsoup4 4.12.2 pypi_0 pypi bleach 6.0.0 pypi_0 pypi bokeh 2.4.3 pypi_0 pypi bottleneck 1.3.7 pypi_0 pypi ca-certificates 2022.12.7 h033912b_0 conda-forge cachetools 5.3.0 pypi_0 pypi ccdproc 2.4.0 pypi_0 pypi celerite 0.4.2 pypi_0 pypi certifi 2022.5.18.1 pypi_0 pypi cffi 1.15.1 pypi_0 pypi cftime 1.6.2 pypi_0 pypi charset-normalizer 3.1.0 pypi_0 pypi cloudpickle 2.2.1 pypi_0 pypi comm 0.1.3 pypi_0 pypi contourpy 1.0.7 pypi_0 pypi corner 2.2.2 pypi_0 pypi coverage 7.2.3 pypi_0 pypi crds 11.17.0 pypi_0 pypi cycler 0.11.0 pypi_0 pypi cython 0.29.34 pypi_0 pypi dask 2022.6.0 pypi_0 pypi debugpy 1.6.7 pypi_0 pypi decorator 5.1.1 pypi_0 pypi defusedxml 0.7.1 pypi_0 pypi deprecat 2.1.1 pypi_0 pypi dill 0.3.6 pypi_0 pypi docutils 0.18.1 pypi_0 pypi drizzle 1.13.7 pypi_0 pypi dynesty 2.1.1 pypi_0 pypi emcee 3.1.4 pypi_0 pypi eureka 0.1.dev2596+g3c10926 pypi_0 pypi exceptiongroup 1.1.1 pypi_0 pypi executing 1.2.0 pypi_0 pypi exoplanet 0.5.3 pypi_0 pypi exoplanet-core 0.1.2 pypi_0 pypi exotic-ld 3.0.0 pypi_0 pypi fastjsonschema 2.16.3 pypi_0 pypi fastprogress 1.0.3 pypi_0 pypi filelock 3.12.0 pypi_0 pypi flake8 6.0.0 pypi_0 pypi fonttools 4.39.3 pypi_0 pypi fqdn 1.5.1 pypi_0 pypi fsspec 2023.4.0 pypi_0 pypi future 0.18.3 pypi_0 pypi george 0.4.0 pypi_0 pypi gwcs 0.18.3 pypi_0 pypi h5netcdf 1.1.0 pypi_0 pypi h5py 3.1.0 pypi_0 pypi html5lib 1.1 pypi_0 pypi idna 3.4 pypi_0 pypi imageio 2.28.0 pypi_0 pypi imagesize 1.4.1 pypi_0 pypi importlib-metadata 6.6.0 pypi_0 pypi importlib-resources 5.12.0 pypi_0 pypi iniconfig 2.0.0 pypi_0 pypi ipykernel 6.22.0 pypi_0 pypi ipython 8.12.0 pypi_0 pypi ipython-genutils 0.2.0 pypi_0 pypi ipywidgets 8.0.6 pypi_0 pypi isoduration 20.11.0 pypi_0 pypi jaraco-classes 3.2.3 pypi_0 pypi jedi 0.18.2 pypi_0 pypi jinja2 3.1.2 pypi_0 pypi jmespath 1.0.1 pypi_0 pypi jsonpointer 2.3 pypi_0 pypi jsonschema 4.9.1 pypi_0 pypi jupyter 1.0.0 pypi_0 pypi jupyter-client 8.2.0 pypi_0 pypi jupyter-console 6.6.3 pypi_0 pypi jupyter-core 5.3.0 pypi_0 pypi jupyter-events 0.6.3 pypi_0 pypi jupyter-server 2.5.0 pypi_0 pypi jupyter-server-terminals 0.4.4 pypi_0 pypi jupyterlab-pygments 0.2.2 pypi_0 pypi jupyterlab-widgets 3.0.7 pypi_0 pypi jwst 1.8.2 pypi_0 pypi keyring 23.13.1 pypi_0 pypi kiwisolver 1.4.4 pypi_0 pypi libcxx 16.0.2 hd57cbcb_0 conda-forge libffi 3.4.2 h0d85af4_5 conda-forge libsqlite 3.40.0 ha978bb4_1 conda-forge libzlib 1.2.13 hfd90126_4 conda-forge llvm-openmp 16.0.2 hff08bdf_0 conda-forge lmfit 1.2.0 pypi_0 pypi locket 1.0.0 pypi_0 pypi markdown-it-py 2.2.0 pypi_0 pypi markupsafe 2.1.2 pypi_0 pypi matplotlib 3.7.1 pypi_0 pypi matplotlib-inline 0.1.6 pypi_0 pypi mccabe 0.7.0 pypi_0 pypi mdit-py-plugins 0.3.5 pypi_0 pypi mdurl 0.1.2 pypi_0 pypi mistune 2.0.5 pypi_0 pypi mkl 2022.2.1 h44ed08c_16952 conda-forge mkl-service 2.4.0 py39hcf0e074_0 conda-forge more-itertools 9.1.0 pypi_0 pypi myst-parser 1.0.0 pypi_0 pypi nbclassic 0.5.5 pypi_0 pypi nbclient 0.7.3 pypi_0 pypi nbconvert 7.3.1 pypi_0 pypi nbformat 5.8.0 pypi_0 pypi nbsphinx 0.9.1 pypi_0 pypi ncurses 6.3 h96cf925_1 conda-forge nest-asyncio 1.5.6 pypi_0 pypi netcdf4 1.6.3 pypi_0 pypi networkx 3.1 pypi_0 pypi notebook 6.5.4 pypi_0 pypi notebook-shim 0.2.3 pypi_0 pypi numpy 1.21.6 pypi_0 pypi numpydoc 1.5.0 pypi_0 pypi opencv-python 4.7.0.72 pypi_0 pypi openssl 3.1.0 h8a1eda9_1 conda-forge packaging 23.1 pypi_0 pypi pandas 2.0.1 pypi_0 pypi pandocfilters 1.5.0 pypi_0 pypi parsley 1.3 pypi_0 pypi parso 0.8.3 pypi_0 pypi partd 1.4.0 pypi_0 pypi patsy 0.5.3 pypi_0 pypi pexpect 4.8.0 pypi_0 pypi photutils 1.7.0 pypi_0 pypi pickleshare 0.7.5 pypi_0 pypi pillow 9.5.0 pypi_0 pypi pip 23.1.1 pyhd8ed1ab_0 conda-forge platformdirs 3.2.0 pypi_0 pypi pluggy 1.0.0 pypi_0 pypi poppy 1.0.3 pypi_0 pypi prometheus-client 0.16.0 pypi_0 pypi prompt-toolkit 3.0.38 pypi_0 pypi psutil 5.9.5 pypi_0 pypi ptyprocess 0.7.0 pypi_0 pypi pure-eval 0.2.2 pypi_0 pypi pycodestyle 2.10.0 pypi_0 pypi pycparser 2.21 pypi_0 pypi pyerfa 2.0.0.3 pypi_0 pypi pyflakes 3.0.1 pypi_0 pypi pygments 2.15.1 pypi_0 pypi pymc3 3.11.5 pypi_0 pypi pymc3-ext 0.1.1 pypi_0 pypi pyparsing 3.0.9 pypi_0 pypi pyrsistent 0.19.3 pypi_0 pypi pysynphot 2.0.0 pypi_0 pypi pytest 7.3.1 pypi_0 pypi pytest-cov 4.0.0 pypi_0 pypi pytest-doctestplus 0.12.1 pypi_0 pypi python 3.9.7 h38b4d05_3_cpython conda-forge python-dateutil 2.8.2 pypi_0 pypi python-json-logger 2.0.7 pypi_0 pypi python_abi 3.9 3_cp39 conda-forge pytz 2023.3 pypi_0 pypi pyvo 1.4.1 pypi_0 pypi pywavelets 1.4.1 pypi_0 pypi pyyaml 6.0 pypi_0 pypi pyzmq 25.0.2 pypi_0 pypi qtconsole 5.4.2 pypi_0 pypi qtpy 2.3.1 pypi_0 pypi readline 8.2 h9e318b2_1 conda-forge reproject 0.10.0 pypi_0 pypi requests 2.28.2 pypi_0 pypi rfc3339-validator 0.1.4 pypi_0 pypi rfc3986-validator 0.1.1 pypi_0 pypi scikit-image 0.19.3 pypi_0 pypi scipy 1.7.3 pypi_0 pypi semantic-version 2.10.0 pypi_0 pypi semver 3.0.0 pypi_0 pypi send2trash 1.8.0 pypi_0 pypi setuptools 67.7.2 pyhd8ed1ab_0 conda-forge setuptools-scm 7.1.0 pypi_0 pypi six 1.16.0 pyh6c4a22f_0 conda-forge sniffio 1.3.0 pypi_0 pypi snowballstemmer 2.2.0 pypi_0 pypi soupsieve 2.4.1 pypi_0 pypi spherical-geometry 1.2.23 pypi_0 pypi sphinx 6.2.0 pypi_0 pypi sphinx-automodapi 0.15.0 pypi_0 pypi sphinx-rtd-theme 1.2.0 pypi_0 pypi sphinxcontrib-applehelp 1.0.4 pypi_0 pypi sphinxcontrib-devhelp 1.0.2 pypi_0 pypi sphinxcontrib-htmlhelp 2.0.1 pypi_0 pypi sphinxcontrib-jquery 4.1 pypi_0 pypi sphinxcontrib-jsmath 1.0.1 pypi_0 pypi sphinxcontrib-qthelp 1.0.3 pypi_0 pypi sphinxcontrib-serializinghtml 1.1.5 pypi_0 pypi sqlite 3.40.0 h9ae0607_1 conda-forge stack-data 0.6.2 pypi_0 pypi starry 1.2.0 pypi_0 pypi stcal 1.2.2 pypi_0 pypi stdatamodels 0.4.5 pypi_0 pypi stpipe 0.5.0 pypi_0 pypi stsci-image 2.3.5 pypi_0 pypi stsci-imagestats 1.6.3 pypi_0 pypi stsci-stimage 0.2.6 pypi_0 pypi svo-filters 0.4.4 pypi_0 pypi tbb 2021.9.0 hb8565cd_0 conda-forge terminado 0.17.1 pypi_0 pypi theano-pymc 1.1.2 pypi_0 pypi tifffile 2023.4.12 pypi_0 pypi tinycss2 1.2.1 pypi_0 pypi tk 8.6.12 h5dbffcc_0 conda-forge tomli 2.0.1 pypi_0 pypi toolz 0.12.0 pypi_0 pypi tornado 6.3.1 pypi_0 pypi tqdm 4.65.0 pypi_0 pypi traitlets 5.9.0 pypi_0 pypi tweakwcs 0.8.2 pypi_0 pypi typing-extensions 4.5.0 pypi_0 pypi tzdata 2023.3 pypi_0 pypi uncertainties 3.1.7 pypi_0 pypi uri-template 1.2.0 pypi_0 pypi urllib3 1.26.15 pypi_0 pypi wcwidth 0.2.6 pypi_0 pypi webcolors 1.13 pypi_0 pypi webencodings 0.5.1 pypi_0 pypi websocket-client 1.5.1 pypi_0 pypi wheel 0.40.0 pyhd8ed1ab_0 conda-forge widgetsnbextension 4.0.7 pypi_0 pypi wiimatch 0.3.1 pypi_0 pypi wrapt 1.15.0 pypi_0 pypi xarray 2023.4.2 pypi_0 pypi xarray-einstats 0.5.1 pypi_0 pypi xz 5.2.6 h775f41a_0 conda-forge zipp 3.15.0 pypi_0 pypi zlib 1.2.13 hfd90126_4 conda-forge ### Code of Conduct - [X] I agree to follow this project's Code of Conduct
Witchblade101 commented 1 year ago

Nevermind! That's set in the S2 ecf file, and I accidentally used an old one!

taylorbell57 commented 1 year ago

Note however that we still do not support NIRISS data reductions - we hope to get there sometime, but it'll take some significant work