RAM pipeline stage 3

[EvgeniaKouts]

Hello everyone!

I would like to ask how much RAM memory should I have to run the whole MIRI pipeline. I have a device with 32GB RAM and the pipeline fails at stage 3. Is there any way to make it run with this memory available?

Thanks!

[braingram]

Hi!

Would you share which pipeline you're running? Are you using strun or calling it from python? Is it possible to share the command/code?

Finally what version of jwst are you using?

[EvgeniaKouts]

Hi! I am using the MIRI-MRS pipeline and I am calling it from this jupyter notebook https://github.com/spacetelescope/jwst-pipeline-notebooks/blob/main/notebooks/MIRI/JWPipeNB-MIRI-MRS.ipynb. Up until stage 2 it works fine. Then in stage 3 it produces the _crf.fits files and then I get a dead kernel. I tried also to run it as a .py file and again it took up all of my RAM after the _crf.fits files and it was killed. I am using JWST Calibration Pipeline Version = 1.16.0 and CRDS Context = jwst_1295.pmap. Thank you!

On Monday, November 4, 2024 at 10:36:44 PM GMT+2, Brett Graham ***@***.***> wrote:  

Hi!

Would you share which pipeline you're running? Are you using strun or calling it from python? Is it possible to share the command/code?

Finally what version of jwst are you using?

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: @.***>

[braingram]

Thanks!

Are you running the example data for that notebook or a different dataset/association? If a different one is it public and would you share the details to be able to find it on mast?

[EvgeniaKouts]

I am running the demo data. Thanks!

On Monday, November 4, 2024 at 10:51:05 PM GMT+2, Brett Graham ***@***.***> wrote:  

Thanks!

Are you running the example data for that notebook or a different dataset/association? If a different one is it public and would you share the details to be able to find it on mast?

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: @.***>

[braingram]

I was able to run through the demo notebook on my machine (with 32 GB of RAM). Below is the pip list of the environment. Would you compare it to the output of pip list for your environment and share that output here? Some key dependencies to check are:

• numpy
• stpipe
• stcal
• stdatamodels
• astropy

Now that the data is downloaded and intermediate files are on my system I'll run a memory profile of the spec3 pipeline and update this issue.

Click to see "pip list" output

Package Version ------------------------- ------------------- anyio 4.6.2.post1 appnope 0.1.4 argon2-cffi 23.1.0 argon2-cffi-bindings 21.2.0 arrow 1.3.0 asdf 3.5.0 asdf-astropy 0.6.1 asdf_coordinates_schemas 0.3.0 asdf_standard 1.1.1 asdf_transform_schemas 0.5.0 asdf_wcs_schemas 0.4.0 astropy 6.1.4 astropy-iers-data 0.2024.11.4.0.33.34 astroquery 0.4.7 asttokens 2.4.1 async-lru 2.0.4 attrs 24.2.0 babel 2.16.0 backports.tarfile 1.2.0 BayesicFitting 3.2.1 beautifulsoup4 4.12.3 bleach 6.2.0 certifi 2024.8.30 cffi 1.17.1 charset-normalizer 3.4.0 comm 0.2.2 contourpy 1.3.0 crds 12.0.4 cycler 0.12.1 debugpy 1.8.7 decorator 5.1.1 defusedxml 0.7.1 drizzle 2.0.0 exceptiongroup 1.2.2 executing 2.1.0 fastjsonschema 2.20.0 filelock 3.16.1 fonttools 4.54.1 fqdn 1.5.1 future 1.0.0 gwcs 0.21.0 h11 0.14.0 html5lib 1.1 httpcore 1.0.6 httpx 0.27.2 idna 3.10 imageio 2.36.0 importlib_metadata 8.5.0 ipykernel 6.29.5 ipython 8.29.0 ipywidgets 8.1.5 isoduration 20.11.0 jaraco.classes 3.4.0 jaraco.context 6.0.1 jaraco.functools 4.1.0 jedi 0.19.1 Jinja2 3.1.4 jmespath 1.0.1 json5 0.9.25 jsonpointer 3.0.0 jsonschema 4.23.0 jsonschema-specifications 2024.10.1 jupyter 1.1.1 jupyter_client 8.6.3 jupyter-console 6.6.3 jupyter_core 5.7.2 jupyter-events 0.10.0 jupyter-lsp 2.2.5 jupyter_server 2.14.2 jupyter_server_terminals 0.5.3 jupyterlab 4.2.5 jupyterlab_pygments 0.3.0 jupyterlab_server 2.27.3 jupyterlab_widgets 3.0.13 jwst 1.16.0 keyring 25.5.0 kiwisolver 1.4.7 lazy_loader 0.4 MarkupSafe 3.0.2 matplotlib 3.9.2 matplotlib-inline 0.1.7 mistune 3.0.2 more-itertools 10.5.0 nbclient 0.10.0 nbconvert 7.16.4 nbformat 5.10.4 nest-asyncio 1.6.0 networkx 3.4.2 notebook 7.2.2 notebook_shim 0.2.4 numpy 1.26.4 opencv-python-headless 4.10.0.84 overrides 7.7.0 packaging 24.1 pandocfilters 1.5.1 Parsley 1.3 parso 0.8.4 pexpect 4.9.0 photutils 2.0.2 pillow 11.0.0 pip 22.2.1 platformdirs 4.3.6 poppy 1.1.1 prometheus_client 0.21.0 prompt_toolkit 3.0.48 psutil 6.1.0 ptyprocess 0.7.0 pure_eval 0.2.3 pycparser 2.22 pyerfa 2.0.1.4 Pygments 2.18.0 pyparsing 3.2.0 python-dateutil 2.9.0.post0 python-json-logger 2.0.7 pyvo 1.6 PyYAML 6.0.2 pyzmq 26.2.0 referencing 0.35.1 requests 2.32.3 rfc3339-validator 0.1.4 rfc3986-validator 0.1.1 rpds-py 0.20.1 scikit-image 0.24.0 scipy 1.14.1 semantic-version 2.10.0 Send2Trash 1.8.3 setuptools 63.2.0 six 1.16.0 sniffio 1.3.1 soupsieve 2.6 spherical_geometry 1.3.2 stack-data 0.6.3 stcal 1.9.0 stdatamodels 2.1.1 stpipe 0.7.0 stsci.imagestats 1.8.3 stsci.stimage 0.2.9 synphot 1.4.0 terminado 0.18.1 tifffile 2024.9.20 tinycss2 1.4.0 tomli 2.0.2 tornado 6.4.1 traitlets 5.14.3 tweakwcs 0.8.9 types-python-dateutil 2.9.0.20241003 typing_extensions 4.12.2 uri-template 1.3.0 urllib3 2.2.3 wcwidth 0.2.13 webcolors 24.8.0 webencodings 0.5.1 websocket-client 1.8.0 widgetsnbextension 4.0.13 wiimatch 0.3.2 zipp 3.20.2

[braingram]

Here's a plot of the memory usage for the call to spec3 with a peak of 4.6G:

Would you share the error message(s) you're seeing? I am still suspicious that this may be due to a dependency and the error message will hopefully be helpful.