Closed Saugata500 closed 1 year ago
AarynnCarter
commented
1 year ago Hi, how much RAM do you have available on the machine you are using? The JWST data files can sometimes be quite large, and lead to crashes if your machine doesn't have enough memory to allocate them to.
@taylorbell57, does this look like a memory issue to you?
taylorbell57
commented
1 year ago Yeah, I'm quite confident it's a limitation with your computer's RAM (we've gotten several similar bug reports in the past, and the "Killed" message has been a common theme when there's RAM limitations in Stages 1-2). You could try using Google Colab or some other cloud computing service (non-trivial, but doable) to run Stages 1-2 (Stages 3+ don't use much RAM), or you could try manually segmenting the files into smaller pieces (making sure to update all of the relevant FITS headers; definitely non-trivial, but I've done it before).
There's really not anything else we can do to help though, so I'm going to close this issue now
Instrument
NIRSpec (Stages 1-3)
What happened?
When I am running Stage 1 or Stage 2, it seems to run through fine for one of the smaller segments of my data (~150) integrations. But when i try to run S1 on a larger segment (~1500 integrations), the code abruptly exits by printing 'Killed'.
When i run the Stage 2 on a bunch of rateints files (10 of them), the code runs through for the first three segments but exits while running the fourth file file and prints 'Killed'
I am not exactly sure where this issue is coming up from. How should I run multiple uncal files or rateints files?
Error traceback output
2023-02-21 16:31:47,268 - stpipe.EurekaSpec2Pipeline.master_background_mos - INFO - Step skipped. 2023-02-21 16:31:47,269 - stpipe.EurekaSpec2Pipeline.master_background_mos - INFO - Results used CRDS context: jwst_1046.pmap 2023-02-21 16:31:47,269 - stpipe.EurekaSpec2Pipeline.master_background_mos - INFO - Step master_background_mos done 2023-02-21 16:31:47,409 - stpipe.EurekaSpec2Pipeline.wavecorr - INFO - Step wavecorr running with args (<SlitModel(1519, 32, 2048) from jw02488001001_04102_00001-seg004_nrs1_rateints.fits>,). 2023-02-21 16:31:47,410 - stpipe.EurekaSpec2Pipeline.wavecorr - INFO - Step wavecorr parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.fits', 'output_use_model': False, 'output_use_index': True, 'save_results': False, 'skip': False, 'suffix': 'wavecorr', 'search_output_file': True, 'input_dir': '/data2/saugata/PhD/HD806_project/JWST/Stage1/Compiled_stage1_rateints'} 2023-02-21 16:31:47,738 - stpipe.EurekaSpec2Pipeline.wavecorr - INFO - Using WAVECORR reference file /data2/saugata/PhD/crds_cache/references/jwst/nirspec/jwst_nirspec_wavecorr_0003.asdf Killed
What operating system are you using?
Ubuntu 18.04.5 LTS
What version of Python are you running?
Python 3.9
What Python packages do you have installed?
Name Version Build Channel
_libgcc_mutex 0.1 main
_openmp_mutex 5.1 1_gnu
alabaster 0.7.12 pypi_0 pypi arrow 1.2.3 pypi_0 pypi asdf 2.13.0 pypi_0 pypi asdf-astropy 0.2.2 pypi_0 pypi asdf-coordinates-schemas 0.1.0 pypi_0 pypi asdf-standard 1.0.3 pypi_0 pypi asdf-transform-schemas 0.3.0 pypi_0 pypi asdf-wcs-schemas 0.1.1 pypi_0 pypi asteval 0.9.27 pypi_0 pypi astraeus 0.3 pypi_0 pypi astroid 2.14.2 pypi_0 pypi astropy 5.1.1 pypi_0 pypi astropy-healpix 0.7 pypi_0 pypi astroquery 0.4.6 pypi_0 pypi astroscrappy 1.1.0 pypi_0 pypi asttokens 2.1.0 pypi_0 pypi atomicwrites 1.4.1 pypi_0 pypi attrs 22.1.0 pypi_0 pypi autopep8 1.6.0 pypi_0 pypi babel 2.11.0 pypi_0 pypi backcall 0.2.0 pypi_0 pypi batman-package 2.4.9 pypi_0 pypi bayesicfitting 3.0.1 pypi_0 pypi beautifulsoup4 4.11.1 pypi_0 pypi binaryornot 0.4.4 pypi_0 pypi black 23.1.0 pypi_0 pypi bleach 6.0.0 pypi_0 pypi bokeh 2.4.3 pypi_0 pypi bottleneck 1.3.5 pypi_0 pypi ca-certificates 2022.10.11 h06a4308_0
ccdproc 2.3.1 pypi_0 pypi celerite 0.4.2 pypi_0 pypi certifi 2022.9.24 py39h06a4308_0
cffi 1.15.1 pypi_0 pypi cftime 1.6.2 pypi_0 pypi chardet 5.1.0 pypi_0 pypi charset-normalizer 2.1.1 pypi_0 pypi click 8.1.3 pypi_0 pypi cloudpickle 2.2.0 pypi_0 pypi comm 0.1.2 pypi_0 pypi contourpy 1.0.6 pypi_0 pypi cookiecutter 2.1.1 pypi_0 pypi corner 2.2.1 pypi_0 pypi crds 11.16.16 pypi_0 pypi cryptography 38.0.3 pypi_0 pypi cycler 0.11.0 pypi_0 pypi cython 0.29.32 pypi_0 pypi dask 2022.6.0 pypi_0 pypi debugpy 1.6.6 pypi_0 pypi decorator 5.1.1 pypi_0 pypi defusedxml 0.7.1 pypi_0 pypi diff-match-patch 20200713 pypi_0 pypi dill 0.3.6 pypi_0 pypi docstring-to-markdown 0.11 pypi_0 pypi docutils 0.19 pypi_0 pypi drizzle 1.13.6 pypi_0 pypi dynesty 2.0.1 pypi_0 pypi emcee 3.1.3 pypi_0 pypi entrypoints 0.4 pypi_0 pypi eureka 0.8 pypi_0 pypi exceptiongroup 1.0.1 pypi_0 pypi executing 1.2.0 pypi_0 pypi exotic-ld 2.0.0 pypi_0 pypi fastjsonschema 2.16.2 pypi_0 pypi filelock 3.8.0 pypi_0 pypi flake8 6.0.0 pypi_0 pypi fonttools 4.38.0 pypi_0 pypi fsspec 2022.10.0 pypi_0 pypi future 0.18.2 pypi_0 pypi george 0.4.0 pypi_0 pypi gwcs 0.18.2 pypi_0 pypi h5netcdf 1.0.2 pypi_0 pypi h5py 3.1.0 pypi_0 pypi html5lib 1.1 pypi_0 pypi idna 3.4 pypi_0 pypi imageio 2.22.4 pypi_0 pypi imagesize 1.4.1 pypi_0 pypi importlib-metadata 5.0.0 pypi_0 pypi inflection 0.5.1 pypi_0 pypi iniconfig 1.1.1 pypi_0 pypi intervaltree 3.1.0 pypi_0 pypi ipykernel 6.21.2 pypi_0 pypi ipython 8.6.0 pypi_0 pypi ipython-genutils 0.2.0 pypi_0 pypi isort 5.12.0 pypi_0 pypi jaraco-classes 3.2.3 pypi_0 pypi jedi 0.18.1 pypi_0 pypi jeepney 0.8.0 pypi_0 pypi jellyfish 0.9.0 pypi_0 pypi jinja2 3.1.2 pypi_0 pypi jinja2-time 0.2.0 pypi_0 pypi jmespath 1.0.1 pypi_0 pypi jsonschema 4.9.1 pypi_0 pypi jupyter-client 7.4.9 pypi_0 pypi jupyter-core 5.2.0 pypi_0 pypi jupyterlab-pygments 0.2.2 pypi_0 pypi jwst 1.6.0 pypi_0 pypi keyring 23.11.0 pypi_0 pypi kiwisolver 1.4.4 pypi_0 pypi lazy-object-proxy 1.9.0 pypi_0 pypi ld_impl_linux-64 2.38 h1181459_1
libffi 3.3 he6710b0_2
libgcc-ng 11.2.0 h1234567_1
libgomp 11.2.0 h1234567_1
libstdcxx-ng 11.2.0 h1234567_1
lmfit 1.0.3 pypi_0 pypi locket 1.0.0 pypi_0 pypi lxml 4.9.1 pypi_0 pypi markupsafe 2.1.1 pypi_0 pypi matplotlib 3.6.2 pypi_0 pypi matplotlib-inline 0.1.6 pypi_0 pypi mccabe 0.7.0 pypi_0 pypi mistune 2.0.5 pypi_0 pypi more-itertools 9.0.0 pypi_0 pypi mypy-extensions 1.0.0 pypi_0 pypi nbclient 0.7.2 pypi_0 pypi nbconvert 7.2.9 pypi_0 pypi nbformat 5.7.3 pypi_0 pypi ncurses 6.3 h5eee18b_3
nest-asyncio 1.5.6 pypi_0 pypi netcdf4 1.6.1 pypi_0 pypi networkx 2.8.8 pypi_0 pypi numpy 1.22.0 pypi_0 pypi numpydoc 1.5.0 pypi_0 pypi opencv-python 4.7.0.68 pypi_0 pypi openssl 1.1.1s h7f8727e_0
packaging 23.0 pypi_0 pypi pandas 1.5.1 pypi_0 pypi pandocfilters 1.5.0 pypi_0 pypi parsley 1.3 pypi_0 pypi parso 0.8.3 pypi_0 pypi partd 1.3.0 pypi_0 pypi pathspec 0.11.0 pypi_0 pypi pexpect 4.8.0 pypi_0 pypi photutils 1.5.0 pypi_0 pypi pickleshare 0.7.5 pypi_0 pypi pillow 9.3.0 pypi_0 pypi pip 22.2.2 py39h06a4308_0
platformdirs 3.0.0 pypi_0 pypi pluggy 1.0.0 pypi_0 pypi poppy 1.0.3 pypi_0 pypi prompt-toolkit 3.0.32 pypi_0 pypi psutil 5.9.3 pypi_0 pypi ptyprocess 0.7.0 pypi_0 pypi pure-eval 0.2.2 pypi_0 pypi pycodestyle 2.10.0 pypi_0 pypi pycparser 2.21 pypi_0 pypi pydocstyle 6.2.3 pypi_0 pypi pyerfa 2.0.0.1 pypi_0 pypi pyflakes 3.0.1 pypi_0 pypi pygments 2.13.0 pypi_0 pypi pylint 2.16.2 pypi_0 pypi pylint-venv 2.3.0 pypi_0 pypi pyls-spyder 0.4.0 pypi_0 pypi pyparsing 3.0.9 pypi_0 pypi pyqt5 5.15.9 pypi_0 pypi pyqt5-qt5 5.15.2 pypi_0 pypi pyqt5-sip 12.11.1 pypi_0 pypi pyqtwebengine 5.15.6 pypi_0 pypi pyqtwebengine-qt5 5.15.2 pypi_0 pypi pyrsistent 0.19.2 pypi_0 pypi pysynphot 2.0.0 pypi_0 pypi pytest 7.2.0 pypi_0 pypi python 3.9.7 h12debd9_1
python-dateutil 2.8.2 pypi_0 pypi python-lsp-black 1.2.1 pypi_0 pypi python-lsp-jsonrpc 1.0.0 pypi_0 pypi python-lsp-server 1.7.1 pypi_0 pypi python-slugify 8.0.0 pypi_0 pypi pytoolconfig 1.2.5 pypi_0 pypi pytz 2022.6 pypi_0 pypi pyvo 1.4 pypi_0 pypi pywavelets 1.4.1 pypi_0 pypi pyxdg 0.28 pypi_0 pypi pyyaml 6.0 pypi_0 pypi pyzmq 25.0.0 pypi_0 pypi qdarkstyle 3.0.3 pypi_0 pypi qstylizer 0.2.2 pypi_0 pypi qtawesome 1.2.2 pypi_0 pypi qtconsole 5.4.0 pypi_0 pypi qtpy 2.3.0 pypi_0 pypi readline 8.2 h5eee18b_0
reproject 0.9.1 pypi_0 pypi requests 2.28.1 pypi_0 pypi rope 1.7.0 pypi_0 pypi rtree 1.0.1 pypi_0 pypi scikit-image 0.19.3 pypi_0 pypi scipy 1.9.3 pypi_0 pypi secretstorage 3.3.3 pypi_0 pypi semantic-version 2.10.0 pypi_0 pypi setuptools 65.5.0 py39h06a4308_0
setuptools-scm 7.0.5 pypi_0 pypi six 1.16.0 pypi_0 pypi snowballstemmer 2.2.0 pypi_0 pypi sortedcontainers 2.4.0 pypi_0 pypi soupsieve 2.3.2.post1 pypi_0 pypi spherical-geometry 1.2.23 pypi_0 pypi sphinx 5.3.0 pypi_0 pypi sphinxcontrib-applehelp 1.0.2 pypi_0 pypi sphinxcontrib-devhelp 1.0.2 pypi_0 pypi sphinxcontrib-htmlhelp 2.0.0 pypi_0 pypi sphinxcontrib-jsmath 1.0.1 pypi_0 pypi sphinxcontrib-qthelp 1.0.3 pypi_0 pypi sphinxcontrib-serializinghtml 1.1.5 pypi_0 pypi spyder 5.4.2 pypi_0 pypi spyder-kernels 2.4.2 pypi_0 pypi sqlite 3.39.3 h5082296_0
stack-data 0.6.0 pypi_0 pypi stcal 1.2.1 pypi_0 pypi stdatamodels 0.4.3 pypi_0 pypi stpipe 0.4.2 pypi_0 pypi stsci-image 2.3.5 pypi_0 pypi stsci-imagestats 1.6.3 pypi_0 pypi stsci-stimage 0.2.6 pypi_0 pypi svo-filters 0.4.4 pypi_0 pypi text-unidecode 1.3 pypi_0 pypi textdistance 4.5.0 pypi_0 pypi three-merge 0.1.1 pypi_0 pypi tifffile 2022.10.10 pypi_0 pypi tinycss2 1.2.1 pypi_0 pypi tk 8.6.12 h1ccaba5_0
toml 0.10.2 pypi_0 pypi tomli 2.0.1 pypi_0 pypi tomlkit 0.11.6 pypi_0 pypi toolz 0.12.0 pypi_0 pypi tornado 6.2 pypi_0 pypi tqdm 4.64.1 pypi_0 pypi traitlets 5.5.0 pypi_0 pypi tweakwcs 0.8.0 pypi_0 pypi typing-extensions 4.4.0 pypi_0 pypi tzdata 2022f h04d1e81_0
ujson 5.7.0 pypi_0 pypi uncertainties 3.1.7 pypi_0 pypi urllib3 1.26.12 pypi_0 pypi watchdog 2.2.1 pypi_0 pypi wcwidth 0.2.5 pypi_0 pypi webencodings 0.5.1 pypi_0 pypi whatthepatch 1.0.4 pypi_0 pypi wheel 0.37.1 pyhd3eb1b0_0
wiimatch 0.3.1 pypi_0 pypi wrapt 1.14.1 pypi_0 pypi wurlitzer 3.0.3 pypi_0 pypi xarray 2022.11.0 pypi_0 pypi xyzservices 2022.9.0 pypi_0 pypi xz 5.2.6 h5eee18b_0
yapf 0.32.0 pypi_0 pypi zipp 3.10.0 pypi_0 pypi zlib 1.2.13 h5eee18b_0
Code of Conduct