search

kevin218 / Eureka

Eureka! is a data reduction and analysis pipeline intended for time-series observations with JWST.
https://eurekadocs.readthedocs.io/
MIT License
60 stars 47 forks source link

Crashing in stage 1 #516

Closed amitmazz closed 1 year ago

amitmazz commented 1 year ago

Instrument

NIRSpec (Stages 1-3)

What happened?

I am running stage 1 of Eureka pipeline for the source Wasp39-b. I am using the latest version of the pipeline. It appears that it is crashing midway after halting for a long time period (more than 1 hour). The very last line I am seeing is appended below. I am not able to go any further.

Error traceback output

2023-04-04 14:40:49,549 - stpipe.Eureka_RampFitStep - WARNING - /home/amit/anaconda3/envs/eureka/lib/python3.9/site-packages/stdatamodels/validate.py:38: ValidationWarning: While validating meta.exposure.p_readpatt the following error occurred: 'NRS|NRSRAPID' does not match '^((ACQ1|ACQ2|BRIGHT1|BRIGHT2|DEEP2|DEEP8|FAST|FASTGRPAVG|FASTGRPAVG8|FASTGRPAVG16|FASTGRPAVG32|FASTGRPAVG64|FASTR1|FASTR100|FGS|FGS60|FGS8370|FGS840|FGSRAPID|FINEGUIDE|ID|MEDIUM2|MEDIUM8|NIS|NISRAPID|NRS|NRSIRS2|NRSN16R4|NRSN32R8|NRSN8R2|NRSRAPID|NRSIRS2RAPID|NRSRAPIDD1|NRSRAPIDD2|NRSRAPIDD6|NRSSLOW|RAPID|SHALLOW2|SHALLOW4|SLOW|SLOWR1|TRACK|ANY|N/A)\s\|\s)+$'

Failed validating 'pattern' in schema: OrderedDict([('title', 'Readout pattern'), ('type', 'string'), ('pattern', '^((ACQ1|ACQ2|BRIGHT1|BRIGHT2|DEEP2|DEEP8|FAST|FASTGRPAVG|FASTGRPAVG8|FASTGRPAVG16|FASTGRPAVG32|FASTGRPAVG64|FASTR1|FASTR100|FGS|FGS60|FGS8370|FGS840|FGSRAPID|FINEGUIDE|ID|MEDIUM2|MEDIUM8|NIS|NISRAPID|NRS|NRSIRS2|NRSN16R4|NRSN32R8|NRSN8R2|NRSRAPID|NRSIRS2RAPID|NRSRAPIDD1|NRSRAPIDD2|NRSRAPIDD6|NRSSLOW|RAPID|SHALLOW2|SHALLOW4|SLOW|SLOWR1|TRACK|ANY|N/A)\s\|\s)+$'), ('fits_keyword', 'P_READPA')])

On instance: 'NRS|NRSRAPID' warnings.warn(errmsg, ValidationWarning)

2023-04-04 14:40:49,580 - stpipe.Eureka_RampFitStep - INFO - Extracting gain subarray to match science data 2023-04-04 14:40:49,580 - stpipe.Eureka_RampFitStep - INFO - Extracting readnoise subarray to match science data 2023-04-04 14:40:49,580 - stpipe.Eureka_RampFitStep - INFO - Using algorithm = default 2023-04-04 14:40:49,581 - stpipe.Eureka_RampFitStep - INFO - Using weighting = default

What operating system are you using?

Linux-Ubuntu

What version of Python are you running?

python 3.9.16

What Python packages do you have installed?

Name Version Build Channel

_libgcc_mutex 0.1 main
_openmp_mutex 5.1 1_gnu
aesara-theano-fallback 0.1.0 pypi_0 pypi alabaster 0.7.13 pypi_0 pypi anyio 3.6.2 pypi_0 pypi argon2-cffi 21.3.0 pypi_0 pypi argon2-cffi-bindings 21.2.0 pypi_0 pypi arrow 1.2.3 pypi_0 pypi arviz 0.12.1 pypi_0 pypi asdf 2.14.4 pypi_0 pypi asdf-astropy 0.4.0 pypi_0 pypi asdf-coordinates-schemas 0.2.0 pypi_0 pypi asdf-standard 1.0.3 pypi_0 pypi asdf-transform-schemas 0.3.0 pypi_0 pypi asdf-unit-schemas 0.1.0 pypi_0 pypi asdf-wcs-schemas 0.1.1 pypi_0 pypi asteval 0.9.29 pypi_0 pypi astraeus 0.3 pypi_0 pypi astropy 5.2.2 pypi_0 pypi astropy-healpix 0.7 pypi_0 pypi astroquery 0.4.6 pypi_0 pypi astroscrappy 1.1.0 pypi_0 pypi asttokens 2.2.1 pypi_0 pypi attrs 22.2.0 pypi_0 pypi babel 2.12.1 pypi_0 pypi backcall 0.2.0 pypi_0 pypi batman-package 2.4.9 pypi_0 pypi bayesicfitting 3.1.1 pypi_0 pypi beautifulsoup4 4.12.0 pypi_0 pypi bleach 6.0.0 pypi_0 pypi bokeh 2.4.3 pypi_0 pypi bottleneck 1.3.7 pypi_0 pypi ca-certificates 2023.01.10 h06a4308_0
cachetools 5.3.0 pypi_0 pypi ccdproc 2.4.0 pypi_0 pypi celerite 0.4.2 pypi_0 pypi certifi 2022.5.18.1 pypi_0 pypi cffi 1.15.1 pypi_0 pypi cftime 1.6.2 pypi_0 pypi charset-normalizer 3.1.0 pypi_0 pypi cloudpickle 2.2.1 pypi_0 pypi codecov 2.1.12 pypi_0 pypi comm 0.1.3 pypi_0 pypi contourpy 1.0.7 pypi_0 pypi corner 2.2.1 pypi_0 pypi coverage 7.2.2 pypi_0 pypi crds 11.16.21 pypi_0 pypi cryptography 40.0.1 pypi_0 pypi cycler 0.11.0 pypi_0 pypi cython 0.29.34 pypi_0 pypi dask 2022.6.0 pypi_0 pypi debugpy 1.6.6 pypi_0 pypi decorator 5.1.1 pypi_0 pypi defusedxml 0.7.1 pypi_0 pypi deprecat 2.1.1 pypi_0 pypi dill 0.3.6 pypi_0 pypi docutils 0.18.1 pypi_0 pypi drizzle 1.13.7 pypi_0 pypi dynesty 2.1.0 pypi_0 pypi emcee 3.1.4 pypi_0 pypi eureka 0.9.dev155+ga3ca120 pypi_0 pypi exceptiongroup 1.1.1 pypi_0 pypi executing 1.2.0 pypi_0 pypi exoplanet 0.5.3 pypi_0 pypi exoplanet-core 0.1.2 pypi_0 pypi exotic-ld 3.0.0 pypi_0 pypi fastjsonschema 2.16.3 pypi_0 pypi fastprogress 1.0.3 pypi_0 pypi filelock 3.10.7 pypi_0 pypi flake8 6.0.0 pypi_0 pypi fonttools 4.39.3 pypi_0 pypi fqdn 1.5.1 pypi_0 pypi fsspec 2023.3.0 pypi_0 pypi future 0.18.3 pypi_0 pypi george 0.4.0 pypi_0 pypi gwcs 0.18.3 pypi_0 pypi h5netcdf 1.1.0 pypi_0 pypi h5py 3.1.0 pypi_0 pypi html5lib 1.1 pypi_0 pypi idna 3.4 pypi_0 pypi image-registration 0.2.7.dev16+g24e5018 pypi_0 pypi imageio 2.27.0 pypi_0 pypi imagesize 1.4.1 pypi_0 pypi importlib-metadata 6.1.0 pypi_0 pypi importlib-resources 5.12.0 pypi_0 pypi iniconfig 2.0.0 pypi_0 pypi intel-openmp 2021.4.0 h06a4308_3561
ipykernel 6.22.0 pypi_0 pypi ipython 8.12.0 pypi_0 pypi ipython-genutils 0.2.0 pypi_0 pypi ipywidgets 8.0.6 pypi_0 pypi isoduration 20.11.0 pypi_0 pypi jaraco-classes 3.2.3 pypi_0 pypi jedi 0.18.2 pypi_0 pypi jeepney 0.8.0 pypi_0 pypi jinja2 3.1.2 pypi_0 pypi jmespath 1.0.1 pypi_0 pypi jsonpointer 2.3 pypi_0 pypi jsonschema 4.9.1 pypi_0 pypi jupyter 1.0.0 pypi_0 pypi jupyter-client 8.1.0 pypi_0 pypi jupyter-console 6.6.3 pypi_0 pypi jupyter-core 5.3.0 pypi_0 pypi jupyter-events 0.6.3 pypi_0 pypi jupyter-server 2.5.0 pypi_0 pypi jupyter-server-terminals 0.4.4 pypi_0 pypi jupyterlab-pygments 0.2.2 pypi_0 pypi jupyterlab-widgets 3.0.7 pypi_0 pypi jwst 1.8.0 pypi_0 pypi keyring 23.13.1 pypi_0 pypi kiwisolver 1.4.4 pypi_0 pypi lazy-loader 0.2 pypi_0 pypi ld_impl_linux-64 2.38 h1181459_1
libffi 3.4.2 h6a678d5_6
libgcc-ng 11.2.0 h1234567_1
libgomp 11.2.0 h1234567_1
libstdcxx-ng 11.2.0 h1234567_1
lmfit 1.1.0 pypi_0 pypi locket 1.0.0 pypi_0 pypi lxml 4.9.2 pypi_0 pypi markdown-it-py 2.2.0 pypi_0 pypi markupsafe 2.1.2 pypi_0 pypi matplotlib 3.7.1 pypi_0 pypi matplotlib-inline 0.1.6 pypi_0 pypi mccabe 0.7.0 pypi_0 pypi mdit-py-plugins 0.3.5 pypi_0 pypi mdurl 0.1.2 pypi_0 pypi mistune 2.0.5 pypi_0 pypi mkl 2021.4.0 h06a4308_640
mkl-service 2.4.0 py39h7f8727e_0
more-itertools 9.1.0 pypi_0 pypi myst-parser 1.0.0 pypi_0 pypi nbclassic 0.5.4 pypi_0 pypi nbclient 0.7.3 pypi_0 pypi nbconvert 7.3.0 pypi_0 pypi nbformat 5.8.0 pypi_0 pypi nbsphinx 0.9.1 pypi_0 pypi ncurses 6.4 h6a678d5_0
nest-asyncio 1.5.6 pypi_0 pypi netcdf4 1.6.3 pypi_0 pypi networkx 3.0 pypi_0 pypi notebook 6.5.3 pypi_0 pypi notebook-shim 0.2.2 pypi_0 pypi numpy 1.21.6 pypi_0 pypi numpydoc 1.5.0 pypi_0 pypi opencv-python 4.7.0.72 pypi_0 pypi openssl 1.1.1t h7f8727e_0
packaging 23.0 pypi_0 pypi pandas 1.5.3 pypi_0 pypi pandocfilters 1.5.0 pypi_0 pypi parsley 1.3 pypi_0 pypi parso 0.8.3 pypi_0 pypi partd 1.3.0 pypi_0 pypi patsy 0.5.3 pypi_0 pypi pexpect 4.8.0 pypi_0 pypi photutils 1.6.0 pypi_0 pypi pickleshare 0.7.5 pypi_0 pypi pillow 9.5.0 pypi_0 pypi pip 23.0.1 py39h06a4308_0
platformdirs 3.2.0 pypi_0 pypi pluggy 1.0.0 pypi_0 pypi poppy 1.0.3 pypi_0 pypi prometheus-client 0.16.0 pypi_0 pypi prompt-toolkit 3.0.38 pypi_0 pypi psutil 5.9.4 pypi_0 pypi ptyprocess 0.7.0 pypi_0 pypi pure-eval 0.2.2 pypi_0 pypi pycodestyle 2.10.0 pypi_0 pypi pycparser 2.21 pypi_0 pypi pyerfa 2.0.0.3 pypi_0 pypi pyflakes 3.0.1 pypi_0 pypi pygments 2.14.0 pypi_0 pypi pymc3 3.11.5 pypi_0 pypi pymc3-ext 0.1.1 pypi_0 pypi pyparsing 3.0.9 pypi_0 pypi pyrsistent 0.19.3 pypi_0 pypi pysynphot 2.0.0 pypi_0 pypi pytest 7.2.2 pypi_0 pypi pytest-cov 4.0.0 pypi_0 pypi pytest-doctestplus 0.12.1 pypi_0 pypi python 3.9.16 h7a1cb2a_2
python-dateutil 2.8.2 pypi_0 pypi python-json-logger 2.0.7 pypi_0 pypi pytz 2023.3 pypi_0 pypi pyvo 1.4.1 pypi_0 pypi pywavelets 1.4.1 pypi_0 pypi pyyaml 6.0 pypi_0 pypi pyzmq 25.0.2 pypi_0 pypi qtconsole 5.4.2 pypi_0 pypi qtpy 2.3.1 pypi_0 pypi readline 8.2 h5eee18b_0
reproject 0.10.0 pypi_0 pypi requests 2.28.2 pypi_0 pypi rfc3339-validator 0.1.4 pypi_0 pypi rfc3986-validator 0.1.1 pypi_0 pypi scikit-image 0.19.3 pypi_0 pypi scipy 1.7.3 pypi_0 pypi secretstorage 3.3.3 pypi_0 pypi semantic-version 2.10.0 pypi_0 pypi semver 3.0.0 pypi_0 pypi send2trash 1.8.0 pypi_0 pypi setuptools 65.6.3 py39h06a4308_0
setuptools-scm 7.1.0 pypi_0 pypi six 1.16.0 pyhd3eb1b0_1
sniffio 1.3.0 pypi_0 pypi snowballstemmer 2.2.0 pypi_0 pypi soupsieve 2.4 pypi_0 pypi spherical-geometry 1.2.23 pypi_0 pypi sphinx 6.1.3 pypi_0 pypi sphinx-automodapi 0.15.0 pypi_0 pypi sphinx-rtd-theme 1.2.0 pypi_0 pypi sphinxcontrib-applehelp 1.0.4 pypi_0 pypi sphinxcontrib-devhelp 1.0.2 pypi_0 pypi sphinxcontrib-htmlhelp 2.0.1 pypi_0 pypi sphinxcontrib-jquery 4.1 pypi_0 pypi sphinxcontrib-jsmath 1.0.1 pypi_0 pypi sphinxcontrib-qthelp 1.0.3 pypi_0 pypi sphinxcontrib-serializinghtml 1.1.5 pypi_0 pypi sqlite 3.41.1 h5eee18b_0
stack-data 0.6.2 pypi_0 pypi starry 1.2.0 pypi_0 pypi stcal 1.2.2 pypi_0 pypi stdatamodels 0.4.5 pypi_0 pypi stpipe 0.4.6 pypi_0 pypi stsci-image 2.3.5 pypi_0 pypi stsci-imagestats 1.6.3 pypi_0 pypi stsci-stimage 0.2.6 pypi_0 pypi svo-filters 0.4.4 pypi_0 pypi terminado 0.17.1 pypi_0 pypi theano-pymc 1.1.2 pypi_0 pypi tifffile 2023.3.21 pypi_0 pypi tinycss2 1.2.1 pypi_0 pypi tk 8.6.12 h1ccaba5_0
tomli 2.0.1 pypi_0 pypi toolz 0.12.0 pypi_0 pypi tornado 6.2 pypi_0 pypi tqdm 4.65.0 pypi_0 pypi traitlets 5.9.0 pypi_0 pypi tweakwcs 0.8.1 pypi_0 pypi typing-extensions 4.5.0 pypi_0 pypi tzdata 2023c h04d1e81_0
uncertainties 3.1.7 pypi_0 pypi uri-template 1.2.0 pypi_0 pypi urllib3 1.26.15 pypi_0 pypi wcwidth 0.2.6 pypi_0 pypi webcolors 1.13 pypi_0 pypi webencodings 0.5.1 pypi_0 pypi websocket-client 1.5.1 pypi_0 pypi wheel 0.38.4 py39h06a4308_0
widgetsnbextension 4.0.7 pypi_0 pypi wiimatch 0.3.1 pypi_0 pypi wrapt 1.15.0 pypi_0 pypi xarray 2023.3.0 pypi_0 pypi xarray-einstats 0.5.1 pypi_0 pypi xz 5.2.10 h5eee18b_1
zipp 3.15.0 pypi_0 pypi zlib 1.2.13 h5eee18b_0

Code of Conduct

amitmazz commented 1 year ago

I am also attaching the stage 1 input parameters.

Eureka! Control File for Stage 1: Detector Processing

Stage 1 Documentation: https://eurekadocs.readthedocs.io/en/latest/ecf.html#stage-1

suffix uncal

Control ramp fitting method

ramp_fit_algorithm 'default' #Options are 'default', 'mean', or 'differenced' ramp_fit_max_cores 'none' #Options are 'none', quarter', 'half','all'

Pipeline stages

skip_group_scale False skip_dq_init False skip_saturation False skip_ipc True #Skipped by default for all instruments skip_superbias False skip_refpix False skip_linearity False skip_persistence True #Skipped by default for Near-IR TSO skip_dark_current False skip_jump True skip_ramp_fitting False skip_gain_scale False

Pipeline stages parameters

jump_rejection_threshold 4.0 #float, default is 4.0, CR sigma rejection threshold

Custom linearity reference file

custom_linearity False linearity_file /path/to/custom/linearity/fits/file

Custom bias

bias_correction None # Bias correction options: [mean, group_level, smooth, None] bias_group 1 # Group number options: [1, 2, ..., each] bias_smooth_length 201 # Window length when using 'smooth' bias correction custom_bias False superbias_file /path/to/custom/superbias/fits/file

Saturation

update_sat_flags False # Wheter to update the saturation flags more aggressively expand_prev_group False # Expand saturation flags to previous group dq_sat_mode percentile # Options: [percentile, min, defined] dq_sat_percentile 50 # Percentile of the entire time series to use to define the saturation mask (50=median) dq_sat_columns [[0, 0], [0,0], [0,0], [0,0], [0,0]] #for dq_sat_mode = defined, user defined saturated columns

Background subtraction

grouplevel_bg True ncpu 40 bg_y1 6 bg_y2 26 bg_deg 0 p3thresh 3 verbose True isplots_S1 3 nplots 5 hide_plots True

Mask curved traces

masktrace True window_len 11 expand_mask 8 ignore_low None ignore_hi None

Manual reference pixel correction for NIRSpec PRISM

refpix_corr False npix_top 8 npix_bot 8

Project directory

topdir /home/amit/Eureka_files/JWST/

Directories relative to topdir

inputdir uncalibrated outputdir Stage1

Diagnostics

testing_S1 False

"Default" ramp fitting settings

default_ramp_fit_weighting default #Options are "default", "fixed", "interpolated", "flat", or "custom" default_ramp_fit_fixed_exponent 10 #Only used for "fixed" weighting default_ramp_fit_custom_snr_bounds [5,10,20,50,100] # Only used for "custom" weighting, array no spaces default_ramp_fit_custom_exponents [0.4,1,3,6,10] # Only used for "custom" weighting, array no spaces

taylorbell57 commented 1 year ago

Are you certain that you are only processing the FITS files that you intend to work on? I believe you said in your previously submitted issue that you are working on NRS1 observations - if so, are the only FITS files in your input directory NRS1 files, or are there also NRS2 files or other reference files? If there are other non-science data files in your input directory, the pipeline doesn't know which files you are trying to analyze and tries to process them all.

To resolve this, either:

  1. Delete or move any non-science files from your input directory, or
  2. Change your suffix setting in the Stage 1 ECF to be nrs1_uncal (assuming you're trying to work on NRS1 data)
amitmazz commented 1 year ago

I have an input directory named 'uncalibrated' which contains only NRS1 files:

  1. jw01366004001_04101_00001-seg001_nrs1_uncal.fits
  2. jw01366004001_04101_00001-seg002_nrs1_uncal.fits
  3. jw01366004001_04101_00001-seg003_nrs1_uncal.fits
  4. jw01366004001_04101_00001-seg004_nrs1_uncal.fits

The above issue is taking place while processing the first NRS1 file and the terminal is crashing after a long halt. I rechecked my input directory and also changed the suffix. But the problem persists still i.e. I am still getting that warning. But it hasn't crashed after changing the suffix and moved onto the next NRS1 file. As it appears, when I checked the readout in the fits header it says 'NRSRAPID'. I am not sure whether it is relevant in this case

taylorbell57 commented 1 year ago

Ah, so I just realized you're looking at the ERS team's WASP-39b PRISM data (I'd assumed you were looking at G395H data for some reason). For the PRISM data, the science observations are indeed in the NRS1 files, so you've got the right files. I've recently run the G395H data myself, but haven't run the PRISM data before. I'll try running it locally to see what's up.

My best guess though is that you're running into an issue that numerous folks have encountered, where Stages 1 or 2 of the jwst pipeline (which is what Eureka's Stages 1 and 2 are built from) end up exceeding your RAM limit and your computer then kills the code to avoid your computer crashing. How much RAM do you have on the computer you're using?

taylorbell57 commented 1 year ago

Hmm, so I don't get the "Failed validating 'pattern' in schema:" warning - not sure what could be causing that issue, but perhaps it's because I downloaded those PRISM data a month ago (perhaps they updated the uncal files since then? Unlikely but not impossible). My jwst version is 1.8.3 while yours is 1.8.0 (perhaps the source of the issue), and my stdatamodels version is 0.4.3 while yours is 0.4.5 (potentially the source of the difference as well). My CRDS version is also 11.16.16 while yours is 11.16.21 (not likely to be the cause, but who knows). As long as you get sensible outputs from your reduction and fit though, I think you're probably safe to ignore this warning.

And either way, the code runs without crashing on my laptop (64 GB of RAM) so I suspect it might be a RAM issue that caused your crash earlier. NIRSpec data files are pretty large, and 8 GB of RAM would probably be too small (especially if you're doing other things like using a browser) - 12 GB may or may not be fine, 16 GB is likely enough, and 32 GB really should be safe (my 64 GB is certainly overkill for these data). And the fact that it crashed once and didn't another time suggests that your other RAM-intensive tasks probably changed between the runs (e.g. fewer Chrome tabs). If you can't get it to run the whole way through and you can't upgrade your computer's RAM or use a machine with more RAM, you could look into using cloud computing resources like Google Colab (which has ~13 GB of RAM but only 1 fairly slow CPU thread but is free), but I can't support you in doing so.

amitmazz commented 1 year ago

I do not know if the crashing issue was due to the RAM because I was running on a different server which has quite high memory. However I was able to run the whole thing. The outputs are however not good (maybe and issue with the inputs I have given) which is a separate issue I tried upgrading the version of jwst by running 'pip install --upgrade git+https://github.com/kevin218/Eureka.git@main#egg=eureka[jwst]'. Now the version of JWST is 1.8.2. I hope this will resolve some of the issues I was having previously (such as 'Failed validating 'pattern' in schema')

taylorbell57 commented 1 year ago

Hmm strange, well seeing as you can't reproduce the crash I'm just going to mark this issue as closed for now. If it crashes again, please do reopen the issue. I myself haven't tried reproducing the ERS's PRISM reduction yet (I've only done G395H and haven't yet checked how close a match it is), but I'll add a note to my to-do list to publicly post ECFs that approximately reproduce all of the ERS team's results when I can find the time