Closed jovian-explorer closed 2 years ago
First try: pip install git+https://github.com/kevin218/Astraeus@main
Otherwise, try: pip install git+https://github.com/kevin218/Astraeus@0.1
Let me know which one worked for you.
thanks, this worked. new errror though,
Traceback (most recent call last):
File "/home/keshav/Eureka/demos/JWST/run_eureka.py", line 3, in
Is there some way to check what else is missing or is this the only way? one at a time I mean
First try:
pip install git+https://github.com/kevin218/Astraeus@main
Otherwise, try:pip install git+https://github.com/kevin218/Astraeus@0.1
Let me know which one worked for you.
first one worked perfectly for me
Check from the command line to see if import cython
works. If it's found but doesn't import, it may be an indication of paths being wrong someplace. If cython isn't found, go back to ReadTheDocs to make sure you've installed Eureka! correctly.
(base) keshav@dev:~/Eureka/demos/JWST$ import cython import-im6.q16: attempt to perform an operation not allowed by the security policy `PS' @ error/constitute.c/IsCoderAuthorized/408.
I should note that we're in the middle of making major changes (including having stable releases and moving future changes to a development branch), so the main branch is a bit of a mess this week.
Welp shit. I needed to run the entire pipeline for NIRSpec to show to my project guide by 18th. I hope you guys are done before that
You can always revert back to an older version of Eureka! With that said, NIRSpec should work on the current version in main.
I was using that only. the one I had installed during the ERS workshop. But somehow it wasn't working for some reason. I thought it would be better to update to a newer version and not go through the hassle of sorting out errors in the older version. Now I realise how wrong I was.
(base) keshav@dev:~/Eureka/demos/JWST$ import cython import-im6.q16: attempt to perform an operation not allowed by the security policy `PS' @ error/constitute.c/IsCoderAuthorized/408.
It would help to open a python session first.
Check from the command line to see if
import cython
works. If it's found but doesn't import, it may be an indication of paths being wrong someplace. If cython isn't found, go back to ReadTheDocs to make sure you've installed Eureka! correctly.
It is working okay
Okay what now? :no_mouth:
I'm away from my computer this morning, but I can help troubleshoot in a few hours when I get back if you're still having issues. My first recommendation would be to install from scratch (make a new conda environment, and follow the instructions on the readthedocs page). Those instructions work well for me on Mac and Ubuntu.
this is the from scratch installation. Should I try it from scratch again? :no_mouth:
When did you last try starting from scratch? This morning or a long time ago?
this morning
Because if you'd started from scratch and followed the readthedocs instructions you wouldn't have needed to manually install Astraeus, so I think you haven't started from scratch in a while
Ah, but then you did git pull afterwards? Then do try starting from scratch again
i'll try again tomorrow then I guess
okay will do. Will let you know in this thread itself.
I have reinstalled everything. This time, the libraries are all okay I guess. Some other issue this time. I think with the files I am using themselves.
(base) keshav@dev:~/Data/JWST-Sim/NIRSpec$ python3 run_eureka.py WARNING: Unable to find an output metadata file from Eureka!'s S2 in the folder: "/home/keshav/Data/JWST-Sim/NIRSpec/Stage1/" Assuming this S2 data was produced by the JWST pipeline instead.
Starting Stage 2 Reduction Input directory: /home/keshav/Data/JWST-Sim/NIRSpec/Stage1/ Output directory: /home/keshav/Data/JWST-Sim/NIRSpec/Stage2/S2_2022-05-13_wasp39b_run6/ Copying S2 control file
Found 1 data file(s) ending in rateints.fits
2022-05-13 12:05:32,364 - stpipe.EurekaSpec2Pipeline - INFO - EurekaSpec2Pipeline instance created.
2022-05-13 12:05:32,365 - stpipe.EurekaSpec2Pipeline.bkg_subtract - INFO - BackgroundStep instance created.
2022-05-13 12:05:32,366 - stpipe.EurekaSpec2Pipeline.assign_wcs - INFO - AssignWcsStep instance created.
2022-05-13 12:05:32,366 - stpipe.EurekaSpec2Pipeline.imprint_subtract - INFO - ImprintStep instance created.
2022-05-13 12:05:32,367 - stpipe.EurekaSpec2Pipeline.msa_flagging - INFO - MSAFlagOpenStep instance created.
2022-05-13 12:05:32,367 - stpipe.EurekaSpec2Pipeline.extract_2d - INFO - Extract2dStep instance created.
2022-05-13 12:05:32,369 - stpipe.EurekaSpec2Pipeline.master_background - INFO - MasterBackgroundNrsSlitsStep instance created.
2022-05-13 12:05:32,370 - stpipe.EurekaSpec2Pipeline.master_background.flat_field - INFO - FlatFieldStep instance created.
2022-05-13 12:05:32,371 - stpipe.EurekaSpec2Pipeline.master_background.pathloss - INFO - PathLossStep instance created.
2022-05-13 12:05:32,371 - stpipe.EurekaSpec2Pipeline.master_background.barshadow - INFO - BarShadowStep instance created.
2022-05-13 12:05:32,372 - stpipe.EurekaSpec2Pipeline.master_background.photom - INFO - PhotomStep instance created.
2022-05-13 12:05:32,372 - stpipe.EurekaSpec2Pipeline.wavecorr - INFO - WavecorrStep instance created.
2022-05-13 12:05:32,373 - stpipe.EurekaSpec2Pipeline.flat_field - INFO - FlatFieldStep instance created.
2022-05-13 12:05:32,373 - stpipe.EurekaSpec2Pipeline.srctype - INFO - SourceTypeStep instance created.
2022-05-13 12:05:32,374 - stpipe.EurekaSpec2Pipeline.straylight - INFO - StraylightStep instance created.
2022-05-13 12:05:32,374 - stpipe.EurekaSpec2Pipeline.fringe - INFO - FringeStep instance created.
2022-05-13 12:05:32,375 - stpipe.EurekaSpec2Pipeline.pathloss - INFO - PathLossStep instance created.
2022-05-13 12:05:32,375 - stpipe.EurekaSpec2Pipeline.barshadow - INFO - BarShadowStep instance created.
2022-05-13 12:05:32,376 - stpipe.EurekaSpec2Pipeline.photom - INFO - PhotomStep instance created.
2022-05-13 12:05:32,377 - stpipe.EurekaSpec2Pipeline.resample_spec - INFO - ResampleSpecStep instance created.
2022-05-13 12:05:32,378 - stpipe.EurekaSpec2Pipeline.cube_build - INFO - CubeBuildStep instance created.
2022-05-13 12:05:32,378 - stpipe.EurekaSpec2Pipeline.extract_1d - INFO - Extract1dStep instance created.
Starting file 1 of 1
Editing (in place) the waverange in the input file
Traceback (most recent call last):
File "/home/keshav/Data/JWST-Sim/NIRSpec/run_eureka.py", line 23, in
Glad to hear that reinstalling seems to have resolved the issue you were experiencing before! Can you copy-paste the outputs of conda list
now, copy-paste the contents of your S2 ECF, and tell me which file you're trying to process so I can try and reproduce your new issue?
`(base) keshav@dev:~$ conda list
#
_libgcc_mutex 0.1 main
_openmp_mutex 4.5 1_gnu
alabaster 0.7.12 pypi_0 pypi
asdf 2.10.1 pypi_0 pypi
asdf-astropy 0.2.1 pypi_0 pypi
asdf-coordinates-schemas 0.1.0 pypi_0 pypi
asdf-standard 1.0.2 pypi_0 pypi
asdf-transform-schemas 0.2.2 pypi_0 pypi
asdf-wcs-schemas 0.1.1 pypi_0 pypi
asteval 0.9.26 pypi_0 pypi
astraeus 0.1 pypi_0 pypi
astropy 5.0.4 pypi_0 pypi
astropy-healpix 0.6 pypi_0 pypi
astroquery 0.4.6 pypi_0 pypi
astroscrappy 1.1.0 pypi_0 pypi
asttokens 2.0.5 pypi_0 pypi
attrs 21.4.0 pypi_0 pypi
babel 2.10.1 pypi_0 pypi
backcall 0.2.0 pypi_0 pypi
batman-package 2.4.8 pypi_0 pypi
bayesicfitting 3.0.1 pypi_0 pypi
beautifulsoup4 4.11.1 pypi_0 pypi
bleach 5.0.0 pypi_0 pypi
bokeh 2.4.2 pypi_0 pypi
bottleneck 1.3.4 pypi_0 pypi
brotlipy 0.7.0 py39h27cfd23_1003
ca-certificates 2021.10.26 h06a4308_2
ccdproc 2.3.1 pypi_0 pypi
celerite 0.4.2 pypi_0 pypi
certifi 2021.10.8 py39h06a4308_2
cffi 1.15.0 py39hd667e15_1
cftime 1.6.0 pypi_0 pypi
charset-normalizer 2.0.4 pyhd3eb1b0_0
cloudpickle 2.0.0 pypi_0 pypi
commonmark 0.9.1 pypi_0 pypi
conda 4.11.0 py39h06a4308_0
conda-content-trust 0.1.1 pyhd3eb1b0_0
conda-package-handling 1.7.3 py39h27cfd23_1
corner 2.2.1 pypi_0 pypi
crds 11.14.0 pypi_0 pypi
cryptography 36.0.0 py39h9ce1e76_0
cycler 0.11.0 pypi_0 pypi
cython 0.29.28 pypi_0 pypi
dask 2022.5.0 pypi_0 pypi
decorator 5.1.1 pypi_0 pypi
defusedxml 0.7.1 pypi_0 pypi
docutils 0.17.1 pypi_0 pypi
drizzle 1.13.4 pypi_0 pypi
dynesty 1.2.2 pypi_0 pypi
emcee 3.1.2 pypi_0 pypi
entrypoints 0.4 pypi_0 pypi
eureka 0.0.1 pypi_0 pypi
executing 0.8.3 pypi_0 pypi
fastjsonschema 2.15.3 pypi_0 pypi
filelock 3.6.0 pypi_0 pypi
fonttools 4.33.3 pypi_0 pypi
fsspec 2022.3.0 pypi_0 pypi
future 0.18.2 pypi_0 pypi
george 0.4.0 pypi_0 pypi
gwcs 0.18.1 pypi_0 pypi
h5netcdf 1.0.0 pypi_0 pypi
h5py 3.1.0 pypi_0 pypi
html5lib 1.1 pypi_0 pypi
idna 3.3 pyhd3eb1b0_0
imageio 2.19.1 pypi_0 pypi
imagesize 1.3.0 pypi_0 pypi
importlib-metadata 4.11.3 pypi_0 pypi
iniconfig 1.1.1 pypi_0 pypi
ipython 8.3.0 pypi_0 pypi
jedi 0.18.1 pypi_0 pypi
jeepney 0.8.0 pypi_0 pypi
jinja2 3.1.2 pypi_0 pypi
jmespath 1.0.0 pypi_0 pypi
jsonschema 3.2.0 pypi_0 pypi
jupyter-client 7.3.1 pypi_0 pypi
jupyter-core 4.10.0 pypi_0 pypi
jupyterlab-pygments 0.2.2 pypi_0 pypi
jwst 1.3.3 pypi_0 pypi
keyring 23.5.0 pypi_0 pypi
kiwisolver 1.4.2 pypi_0 pypi
ld_impl_linux-64 2.35.1 h7274673_9
libffi 3.3 he6710b0_2
libgcc-ng 9.3.0 h5101ec6_17
libgomp 9.3.0 h5101ec6_17
libstdcxx-ng 9.3.0 hd4cf53a_17
lmfit 1.0.3 pypi_0 pypi
locket 1.0.0 pypi_0 pypi
lxml 4.8.0 pypi_0 pypi
markdown-it-py 2.1.0 pypi_0 pypi
markupsafe 2.1.1 pypi_0 pypi
matplotlib 3.5.2 pypi_0 pypi
matplotlib-inline 0.1.3 pypi_0 pypi
mdit-py-plugins 0.3.0 pypi_0 pypi
mdurl 0.1.1 pypi_0 pypi
mistune 0.8.4 pypi_0 pypi
myst-parser 0.17.2 pypi_0 pypi
nbclient 0.6.3 pypi_0 pypi
nbconvert 6.5.0 pypi_0 pypi
nbformat 5.4.0 pypi_0 pypi
nbsphinx 0.8.8 pypi_0 pypi
ncurses 6.3 h7f8727e_2
nest-asyncio 1.5.5 pypi_0 pypi
netcdf4 1.5.8 pypi_0 pypi
networkx 2.8 pypi_0 pypi
numpy 1.22.3 pypi_0 pypi
numpydoc 1.3.1 pypi_0 pypi
openssl 1.1.1m h7f8727e_0
packaging 21.3 pypi_0 pypi
pandas 1.4.2 pypi_0 pypi
pandocfilters 1.5.0 pypi_0 pypi
parsley 1.3 pypi_0 pypi
parso 0.8.3 pypi_0 pypi
partd 1.2.0 pypi_0 pypi
pexpect 4.8.0 pypi_0 pypi
photutils 1.1.0 pypi_0 pypi
pickleshare 0.7.5 pypi_0 pypi
pillow 9.1.0 pypi_0 pypi
pip 21.2.4 py39h06a4308_0
pluggy 1.0.0 pypi_0 pypi
poppy 1.0.2 pypi_0 pypi
prompt-toolkit 3.0.29 pypi_0 pypi
psutil 5.9.0 pypi_0 pypi
ptyprocess 0.7.0 pypi_0 pypi
pure-eval 0.2.2 pypi_0 pypi
py 1.11.0 pypi_0 pypi
pycosat 0.6.3 py39h27cfd23_0
pycparser 2.21 pyhd3eb1b0_0
pyerfa 2.0.0.1 pypi_0 pypi
pygments 2.12.0 pypi_0 pypi
pyopenssl 21.0.0 pyhd3eb1b0_1
pyparsing 3.0.9 pypi_0 pypi
pyrsistent 0.18.1 pypi_0 pypi
pysocks 1.7.1 py39h06a4308_0
pysynphot 2.0.0 pypi_0 pypi
pytest 7.1.2 pypi_0 pypi
python 3.9.7 h12debd9_1
python-dateutil 2.8.2 pypi_0 pypi
pytz 2022.1 pypi_0 pypi
pyvo 1.3 pypi_0 pypi
pywavelets 1.3.0 pypi_0 pypi
pyyaml 6.0 pypi_0 pypi
pyzmq 22.3.0 pypi_0 pypi
readline 8.1.2 h7f8727e_1
recommonmark 0.7.1 pypi_0 pypi
reproject 0.8 pypi_0 pypi
requests 2.27.1 pyhd3eb1b0_0
ruamel_yaml 0.15.100 py39h27cfd23_0
scikit-image 0.19.2 pypi_0 pypi
scipy 1.8.0 pypi_0 pypi
secretstorage 3.3.2 pypi_0 pypi
semantic-version 2.9.0 pypi_0 pypi
setuptools 58.0.4 py39h06a4308_0
six 1.16.0 pyhd3eb1b0_0
snowballstemmer 2.2.0 pypi_0 pypi
soupsieve 2.3.2.post1 pypi_0 pypi
spherical-geometry 1.2.22 pypi_0 pypi
sphinx 4.5.0 pypi_0 pypi
sphinx-rtd-theme 1.0.0 pypi_0 pypi
sphinxcontrib-applehelp 1.0.2 pypi_0 pypi
sphinxcontrib-devhelp 1.0.2 pypi_0 pypi
sphinxcontrib-htmlhelp 2.0.0 pypi_0 pypi
sphinxcontrib-jsmath 1.0.1 pypi_0 pypi
sphinxcontrib-qthelp 1.0.3 pypi_0 pypi
sphinxcontrib-serializinghtml 1.1.5 pypi_0 pypi
sqlite 3.37.0 hc218d9a_0
stack-data 0.2.0 pypi_0 pypi
stcal 0.6.4 pypi_0 pypi
stdatamodels 0.2.4 pypi_0 pypi
stpipe 0.3.3 pypi_0 pypi
stsci-image 2.3.5 pypi_0 pypi
stsci-imagestats 1.6.3 pypi_0 pypi
stsci-stimage 0.2.5 pypi_0 pypi
svo-filters 0.4.1 pypi_0 pypi
tifffile 2022.5.4 pypi_0 pypi
tinycss2 1.1.1 pypi_0 pypi
tk 8.6.11 h1ccaba5_0
tomli 2.0.1 pypi_0 pypi
toolz 0.11.2 pypi_0 pypi
tornado 6.1 pypi_0 pypi
tqdm 4.62.3 pyhd3eb1b0_1
traitlets 5.2.0 pypi_0 pypi
tweakwcs 0.7.4 pypi_0 pypi
typing-extensions 4.2.0 pypi_0 pypi
tzdata 2021e hda174b7_0
uncertainties 3.1.6 pypi_0 pypi
urllib3 1.26.7 pyhd3eb1b0_0
wcwidth 0.2.5 pypi_0 pypi
webencodings 0.5.1 pypi_0 pypi
wheel 0.37.1 pyhd3eb1b0_0
xarray 2022.3.0 pypi_0 pypi
xz 5.2.5 h7b6447c_0
yaml 0.2.5 h7b6447c_0
zipp 3.8.0 pypi_0 pypi
zlib 1.2.11 h7f8727e_4
(base) keshav@dev:~$
`
suffix rateints # Data file suffix
slit_y_low -1 # Use None to rely on the default parameters slit_y_high 50 # Use None to rely on the default parameters
waverange_start 6e-08 # Use None to rely on the default parameters waverange_end 6e-06 # Use None to rely on the default parameters
skip_bkg_subtract True # Not run for TSO observations skip_imprint_subtract True # Not run for NIRSpec Fixed Slit skip_msa_flagging True # Not run for NIRSpec Fixed Slit skip_extract_2d False skip_srctype False skip_master_background True # Not run for NIRSpec Fixed Slit skip_wavecorr False skip_flat_field True # NOTE At the time the NIRSpec ERS Hackathon simulated data was created, this step did not work correctly and is by default turned off. skip_straylight True # Not run for NIRSpec Fixed Slit skip_fringe True # Not run for NIRSpec Fixed Slit skip_pathloss True # Not run for TSO observations skip_barshadow True # Not run for NIRSpec Fixed Slit skip_photom True # Recommended to skip to get better uncertainties out of Stage 3. skip_resample True # Not run for TSO observations skip_cube_build True # Not run for NIRSpec Fixed Slit skip_extract_1d False
testing_S2 False hide_plots False # If True, plots will automatically be closed rather than popping up
topdir /home/keshav/Data/JWST-Sim/NIRSpec
inputdir /Stage1 outputdir /Stage2
I think that the error was in the file I downloaded itself. with a tiny dataset I managed to run stage 4 as well. althought, stage 5 is giving some error.
Starting Stage 5: Light Curve Fitting
Finished loading parameters from /home/keshav/Data/JWST-Sim/NIRSpec/Stage4/S4_2022-05-14_wasp39b_run1/ap6_bg7/S4_wasp39b_ap6_bg7_LCData.h5
Input directory: /home/keshav/Data/JWST-Sim/NIRSpec/Stage4/S4_2022-05-14_wasp39b_run1/ap6_bg7/
Output directory: /home/keshav/Data/JWST-Sim/NIRSpec/Stage5/S5_2022-05-14_wasp39b_run1/ap6_bg7/
Copying S5 control file
Copying S5 parameter control file
Traceback (most recent call last):
File "/home/keshav/Data/JWST-Sim/NIRSpec/run_eureka.py", line 29, in
this is the eureka.py file I am running
`import sys, os sys.path.append(f'..{os.sep}..{os.sep}') import eureka.lib.plots import eureka.S1_detector_processing.s1_process as s1 import eureka.S2_calibrations.s2_calibrate as s2 import eureka.S3_data_reduction.s3_reduce as s3 import eureka.S4_generate_lightcurves.s4_genLC as s4 import eureka.S5_lightcurve_fitting.s5_fit as s5 import eureka.S6_planet_spectra.s6_spectra as s6
# Set up some parameters to make plots look nicer. You can set usetex=True if you have LaTeX installed
eureka.lib.plots.set_rc(style='eureka', usetex=False, filetype='.png')
# eventlabel = 'imaging_template'
# eventlabel = 'miri_lrs_template'
# eventlabel = 'nirspec_fs_template'
eventlabel = 'wasp39b' ecf_path = '.'+os.sep
if name == 'main':
s2_meta = s2.calibrateJWST(eventlabel, ecf_path=ecf_path)
s3_spec, s3_meta = s3.reduce(eventlabel, ecf_path=ecf_path, s2_meta=s2_meta)
s4_spec, s4_lc, s4_meta = s4.genlc(eventlabel, ecf_path=ecf_path, s3_meta=s3_meta)
s5_meta = s5.fitlc(eventlabel, ecf_path=ecf_path, s4_meta=s4_meta)
s6_meta = s6.plot_spectra(eventlabel, ecf_path=ecf_path, s5_meta=s5_meta)`
I wonder if the FITS file is corrupt? Have you tried running pytest on the NIRSpec Python file in the Tests directory? If that works, you may need to re-download your data.
The large one? it seems that way to me as well. I am downloading it again. The tiny one? not sure.
============================= test session starts ============================== platform linux -- Python 3.9.7, pytest-7.1.2, pluggy-1.0.0 rootdir: /home/keshav/Data/JWST-Sim/NIRSpec plugins: jwst-1.3.3, asdf-2.10.1 collected 1 item
test_NIRSpec.py
IMPORTANT: Make sure that any changes to the ecf files are included in demo ecf files and documentation (docs/source/ecf.rst)
NIRSpec S2-5 test: Killed
The fact that it says Killed suggests to me that you've hit a memory limit on your computer. This shouldn't happen unless you have lots of open files. Have you tried restarting your computer?
test_NIRSpec.py::test_NIRSpec /home/keshav/miniconda3/lib/python3.9/site-packages/astropy/io/fits/file.py:379: AstropyUserWarning: File may have been truncated: actual file length (25165824) is smaller than the expected size (33572160) warnings.warn('File may have been truncated: actual file length '
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html ================================================ short test summary info ================================================= FAILED test_NIRSpec.py::test_NIRSpec - AttributeError: 'Parameters' object has no attribute 'filename' ======================================== 1 failed, 1 warning in 139.44s (0:02:19) ======================================== Exception ignored in: <function CachingFileManager.del at 0x7fe85a002af0> Traceback (most recent call last): File "/home/keshav/miniconda3/lib/python3.9/site-packages/xarray/backends/file_manager.py", line 244, in del File "/home/keshav/miniconda3/lib/python3.9/site-packages/xarray/backends/file_manager.py", line 222, in close File "/home/keshav/miniconda3/lib/python3.9/site-packages/h5netcdf/core.py", line 1125, in close File "/home/keshav/miniconda3/lib/python3.9/site-packages/h5py/_hl/files.py", line 445, in close File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper File "h5py/h5f.pyx", line 275, in h5py.h5f.get_obj_ids File "h5py/h5i.pyx", line 46, in h5py.h5i.wrap_identifier ImportError: sys.meta_path is None, Python is likely shutting down Exception ignored in: <function File.close at 0x7fe85930f160> Traceback (most recent call last): File "/home/keshav/miniconda3/lib/python3.9/site-packages/h5netcdf/core.py", line 1125, in close File "/home/keshav/miniconda3/lib/python3.9/site-packages/h5py/_hl/files.py", line 445, in close File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper File "h5py/h5f.pyx", line 275, in h5py.h5f.get_obj_ids File "h5py/h5i.pyx", line 46, in h5py.h5i.wrap_identifier ImportError: sys.meta_path is None, Python is likely shutting down Exception ignored in: <function CachingFileManager.del at 0x7fe85a002af0> Traceback (most recent call last): File "/home/keshav/miniconda3/lib/python3.9/site-packages/xarray/backends/file_manager.py", line 244, in del File "/home/keshav/miniconda3/lib/python3.9/site-packages/xarray/backends/file_manager.py", line 222, in close File "/home/keshav/miniconda3/lib/python3.9/site-packages/h5netcdf/core.py", line 1125, in close File "/home/keshav/miniconda3/lib/python3.9/site-packages/h5py/_hl/files.py", line 445, in close File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper File "h5py/h5f.pyx", line 275, in h5py.h5f.get_obj_ids File "h5py/h5i.pyx", line 46, in h5py.h5i.wrap_identifier ImportError: sys.meta_path is None, Python is likely shutting down Exception ignored in: <function File.close at 0x7fe85930f160> Traceback (most recent call last): File "/home/keshav/miniconda3/lib/python3.9/site-packages/h5netcdf/core.py", line 1125, in close File "/home/keshav/miniconda3/lib/python3.9/site-packages/h5py/_hl/files.py", line 445, in close File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper File "h5py/h5f.pyx", line 275, in h5py.h5f.get_obj_ids File "h5py/h5i.pyx", line 46, in h5py.h5i.wrap_identifier ImportError: sys.meta_path is None, Python is likely shutting down
The fact that it says Killed suggests to me that you've hit a memory limit on your computer. This shouldn't happen unless you have lots of open files. Have you tried restarting your computer?
that was for the large dataset. The previous comment is for a tiny dataset I used.
Strange... I'm unable to reproduce your error while running pytest test_NIRSpec.py
. Maybe Taylor can offer more help?
I am able to run upto stage 4 using a tiny dataset. stage 5 gives this error
Starting Stage 5: Light Curve Fitting
Finished loading parameters from /home/keshav/Data/JWST-Sim/NIRSpec/Stage4/S4_2022-05-16_wasp39b_run1/ap6_bg7/S4_wasp39b_ap6_bg7_LCData.h5
Input directory: /home/keshav/Data/JWST-Sim/NIRSpec/Stage4/S4_2022-05-16_wasp39b_run1/ap6_bg7/
Output directory: /home/keshav/Data/JWST-Sim/NIRSpec/Stage5/S5_2022-05-16_wasp39b_run1/ap6_bg7/
Copying S5 control file
Copying S5 parameter control file
Traceback (most recent call last):
File "/home/keshav/Data/JWST-Sim/NIRSpec/run_eureka.py", line 29, in
Can you share your ECF and EPF files?
# Stage 5 Fit Parameters Documentation: https://eurekadocs.readthedocs.io/en/latest/ecf.html#stage-5-fit-parameters
#Name Value Free? PriorPar1 PriorPar2 PriorType
# PriorType can be U (Uniform), LU (Log Uniform), or N (Normal).
# If U/LU, PriorPar1 and PriorPar2 represent upper and lower limits of the parameter/log(the parameter).
# If N, PriorPar1 is the mean and PriorPar2 is the standard deviation of a Gaussian prior.
#-------------------------------------------------------------------------------------------------------
#
# ------------------
# ** Transit/eclipse parameters **
# ------------------
rp 0.173010 'free' 0.1 0.3 U
#fp 0.008 'free' 0 0.5 U
# ----------------------
# ** Phase curve parameters **
# ----------------------
#AmpCos1 0.4 'free' 0 1 U
#AmpSin1 0.01 'free' -1 1 U
#AmpCos2 0.01 'free' -1 1 U
#AmpSin2 0.01 'free' -1 1 U
# ------------------
# ** Orbital parameters **
# ------------------
per 4.055259 'free' 4.055259 1e-5 N
t0 0.5 'free' 0.4 0.6 U
time_offset 0 'independent'
inc 87.93 'free' 85 90 U
a 11.612 'free' 10 13 U
ecc 0.0 'fixed' 0 1 U
w 90. 'fixed' 0 180 U
# -------------------------
# ** Limb darkening parameters **
# Choose limb_dark from ['uniform', 'linear', 'quadratic', 'kipping2013', 'square-root', 'logarithmic', 'exponential', '4-parameter']
# -------------------------
limb_dark 'kipping2013' 'independent'
u1 0.5 'free' 0 1 U
u2 0.5 'free' 0 1 U
# --------------------
# ** Systematic variables **
# polynomial model variables (c0--c9 for 0th--3rd order polynomials in time); Fitting at least c0 is very strongly recommended!
# expramp model variables (r0--r2 for one exponential ramp, r3--r5 for a second exponential ramp)
# GP model parameters (A, WN, m1, m2) in log scale
# --------------------
c0 1 'free' 0.95 1.05 U
c1 0 'free' -0.1 0.1 U
#A -5 'free' -20. -5. U
#WN -15 'free' -20. -10. U
#m1 -5 'free' -10. 0. U
# -----------
# ** White noise **
# Use scatter_mult to fit a multiplier to the expected noise level from Stage 3 (recommended)
# Use scatter_ppm to fit the noise level in ppm
# Use WN parameter in GP model parameters to account for white noise in the GP model
# -----------
scatter_mult 1 'free' 1 0.1 N
# Eureka! Control File for Stage 5: Lightcurve Fitting
# Stage 5 Documentation: https://eurekadocs.readthedocs.io/en/latest/ecf.html#stage-5
ncpu 8 # The number of CPU threads to use when running emcee or dynesty in parallel
allapers False # Run S5 on all of the apertures considered in S4? Otherwise will use newest output in the inputdir
rescale_err False # Rescale uncertainties to have reduced chi-squared of unity
fit_par ./S5_fit_par_template.epf # What fitting epf do you want to use?
verbose True # If True, more details will be printed about steps
fit_method [dynesty] #options are: lsq, emcee, dynesty (can list multiple types separated by commas)
run_myfuncs [batman_tr,polynomial] #options are: batman_tr, batman_ecl, sinusoid_pc, expramp, GP, and polynomial (can list multiple types separated by commas)
# Limb darkening controls (not yet implemented)
#fix_ld False #use limb darkening file?
#ld_file /path/to/limbdarkening/ld_outputfile.txt #location of limb darkening file
# General fitter
old_fitparams None # filename relative to topdir that points to a fitparams csv to resume where you left off (set to None to start from scratch)
#lsq
lsq_method 'Nelder-Mead' # The scipy.optimize.minimize optimization method to use
lsq_tol 1e-6 # The tolerance for the scipy.optimize.minimize optimization method
lsq_maxiter None # Maximum number of iterations to perform. Depending on the method each iteration may use several function evaluations. Set to None to use the default value
#mcmc
old_chain None # Output folder relative to topdir that contains an old emcee chain to resume where you left off (set to None to start from scratch)
lsq_first True # Initialize with an initial lsq call (can help shorten burn-in, but turn off if lsq fails). Only used if old_chain is None
run_nsteps 1000
run_nwalkers 200
run_nburn 500 # How many of run_nsteps should be discarded as burn-in steps
#dynesty
run_nlive 1024 # Must be > ndim * (ndim + 1) // 2
run_bound 'multi'
run_sample 'auto'
run_tol 0.1
#GP inputs
kernel_inputs ['time'] #options: time
kernel_class ['Matern32'] #options: ExpSquared, Matern32, Exp, RationalQuadratic for george, Matern32 for celerite (sums of kernels possible for george separated by commas)
GP_package 'celerite' #options: george, celerite
# Plotting controls
interp False # Should astrophysical model be interpolated (useful for uneven sampling like that from HST)
# Diagnostics
isplots_S5 5 # Generate few (1), some (3), or many (5) figures (Options: 1 - 5)
testing_S5 False # Boolean, set True to only use the first spectral channel
testing_model False # Boolean, set True to only inject a model source of systematics
hide_plots False # If True, plots will automatically be closed rather than popping up
# Project directory
topdir /home/keshav/Data/JWST-Sim/NIRSpec
# Directories relative to topdir
inputdir /Stage4 # The folder containing the outputs from Eureka!'s S4 pipeline (will be overwritten if calling S4 and S5 sequentially)
outputdir /Stage5
Stage 5 is finally running now. I will keep you posted. Also, is there some way to increase the speed of this stage?
It has hit an error (again)
Traceback (most recent call last):
File "/home/keshav/Data/JWST-Sim/NIRSpec/run_eureka.py", line 29, in <module>
s5_meta = s5.fitlc(eventlabel, ecf_path=ecf_path, s4_meta=s4_meta)
File "/home/keshav/miniconda3/lib/python3.9/site-packages/eureka/S5_lightcurve_fitting/s5_fit.py", line 173, in fitlc
meta = fit_channel(meta,time,flux,channel,flux_err,eventlabel,sharedp,params,log,longparamlist,time_units,paramtitles,chanrng)
File "/home/keshav/miniconda3/lib/python3.9/site-packages/eureka/S5_lightcurve_fitting/s5_fit.py", line 257, in fit_channel
lc_model.fit(model, meta, log, fitter='dynesty')
File "/home/keshav/miniconda3/lib/python3.9/site-packages/eureka/S5_lightcurve_fitting/lightcurve.py", line 156, in fit
fit_model = self.fitter_func(self, model, meta, log, **kwargs)
File "/home/keshav/miniconda3/lib/python3.9/site-packages/eureka/S5_lightcurve_fitting/fitters.py", line 644, in dynestyfitter
plots.plot_corner(samples, lc, meta, freenames, fitter='dynesty')
File "/home/keshav/miniconda3/lib/python3.9/site-packages/eureka/S5_lightcurve_fitting/plots_s5.py", line 203, in plot_corner
fig = corner.corner(samples, fig=fig, quantiles=[0.16, 0.5, 0.84], max_n_ticks=3,
File "/home/keshav/miniconda3/lib/python3.9/site-packages/corner/corner.py", line 226, in corner
return corner_impl(
File "/home/keshav/miniconda3/lib/python3.9/site-packages/corner/core.py", line 121, in corner_impl
raise ValueError(
ValueError: It looks like the parameter(s) in column(s) 9 have no dynamic range. Please provide a `range` argument.
Hi Keshav, we've covered multiple problems in this one issue, so it is a pretty hard for me to understand the status of all the different questions you've asked. In general, please try to open one issue for each specific problem you encounter and please try to provide as many details about what caused that problem when you first open that issue so that we can get to the bottom of the issue quicker. In general, please also make a concerted effort to troubleshoot the issue yourself and read the documentation for the most up-to-date recommendations.
I've enumerated below the problems that you've encountered below with my understanding of what might have caused them or the current status of that issue:
ModuleNotFoundError: No module named 'astraeus.xarrayIO'
ModuleNotFoundError: No module named 'pyximport'
ValueError: cannot reshape array of size 0 into shape (8192,32,512)
and AstropyUserWarning: File may have been truncated: actual file length (25165824) is smaller than the expected size (33572160)
NIRSpec S2-5 test: Killed
pytest test_NIRSpec.py
within the test folder (without having edited any of the ECF files or anything). The file should be plenty small and shouldn't overflow the RAM or anything.AttributeError: 'Parameters' object has no attribute 'filename'
ValueError: It looks like the parameter(s) in column(s) 9 have no dynamic range. Please provide a
rangeargument.
Also, is there some way to increase the speed of this stage (Stage 5)?
ncpu
equal to the number of threads available on your cpu (typically 2 times the number of cpu cores you have) is the best option available for now. You should try just running the lsq fitter if you're debugging things though as that runs far faster than the other fitters. Once you have that working, you can try dynesty or something else to get uncertainties. You can also set testing_S5
to True if you want to try dynesty or something for just a single wavelength and then look at the plots afterward to understand what went well or what went wrongOkay yeah,this thread has been quite long. I'll open a new one with the last issue. Thanks.
Instrument
NIRSpec
What happened?
(base) keshav@dev:~/Eureka/demos/JWST$ python3 run_eureka.py
Error traceback output
(base) keshav@dev:~/Eureka/demos/JWST$ python3 run_eureka.py Traceback (most recent call last): File "/home/keshav/Eureka/demos/JWST/run_eureka.py", line 3, in import eureka.lib.plots File "/home/keshav/Eureka/demos/JWST/../../eureka/init.py", line 20, in from . import S3_data_reduction File "/home/keshav/Eureka/demos/JWST/../../eureka/S3_data_reduction/init.py", line 5, in from . import miri File "/home/keshav/Eureka/demos/JWST/../../eureka/S3_data_reduction/miri.py", line 4, in import astraeus.xarrayIO as xrio ModuleNotFoundError: No module named 'astraeus.xarrayIO'
What operating system are you using?
linux ubuntu 20.04
What version of Python are you running?
No response
What Python packages do you have installed?
(base) keshav@dev:~/Eureka/demos/JWST$ conda list
packages in environment at /home/keshav/miniconda3:
#
Name Version Build Channel
_libgcc_mutex 0.1 main
_openmp_mutex 4.5 1_gnu
asdf 2.11.1 pypi_0 pypi asdf-astropy 0.2.1 pypi_0 pypi asdf-coordinates-schemas 0.1.0 pypi_0 pypi asdf-standard 1.0.2 pypi_0 pypi asdf-transform-schemas 0.2.2 pypi_0 pypi asdf-wcs-schemas 0.1.1 pypi_0 pypi astropy 5.0.4 pypi_0 pypi astropy-healpix 0.6 pypi_0 pypi astroscrappy 1.1.0 pypi_0 pypi attrs 21.4.0 pypi_0 pypi bayesicfitting 3.0.1 pypi_0 pypi brotlipy 0.7.0 py39h27cfd23_1003
ca-certificates 2021.10.26 h06a4308_2
ccdproc 2.3.1 pypi_0 pypi certifi 2021.10.8 py39h06a4308_2
cffi 1.15.0 py39hd667e15_1
charset-normalizer 2.0.4 pyhd3eb1b0_0
conda 4.11.0 py39h06a4308_0
conda-content-trust 0.1.1 pyhd3eb1b0_0
conda-package-handling 1.7.3 py39h27cfd23_1
crds 11.14.0 pypi_0 pypi cryptography 36.0.0 py39h9ce1e76_0
cycler 0.11.0 pypi_0 pypi drizzle 1.13.4 pypi_0 pypi filelock 3.6.0 pypi_0 pypi fonttools 4.33.3 pypi_0 pypi future 0.18.2 pypi_0 pypi gwcs 0.18.1 pypi_0 pypi h5py 3.6.0 pypi_0 pypi idna 3.3 pyhd3eb1b0_0
imageio 2.19.1 pypi_0 pypi jmespath 1.0.0 pypi_0 pypi jsonschema 4.5.1 pypi_0 pypi jwst 1.5.0 pypi_0 pypi kiwisolver 1.4.2 pypi_0 pypi ld_impl_linux-64 2.35.1 h7274673_9
libffi 3.3 he6710b0_2
libgcc-ng 9.3.0 h5101ec6_17
libgomp 9.3.0 h5101ec6_17
libstdcxx-ng 9.3.0 hd4cf53a_17
lxml 4.8.0 pypi_0 pypi matplotlib 3.5.2 pypi_0 pypi ncurses 6.3 h7f8727e_2
networkx 2.8 pypi_0 pypi numpy 1.22.3 pypi_0 pypi openssl 1.1.1m h7f8727e_0
packaging 21.3 pypi_0 pypi parsley 1.3 pypi_0 pypi photutils 1.4.0 pypi_0 pypi pillow 9.1.0 pypi_0 pypi pip 21.2.4 py39h06a4308_0
poppy 1.0.2 pypi_0 pypi psutil 5.9.0 pypi_0 pypi pycosat 0.6.3 py39h27cfd23_0
pycparser 2.21 pyhd3eb1b0_0
pyerfa 2.0.0.1 pypi_0 pypi pyopenssl 21.0.0 pyhd3eb1b0_1
pyparsing 3.0.9 pypi_0 pypi pyrsistent 0.18.1 pypi_0 pypi pysocks 1.7.1 py39h06a4308_0
python 3.9.7 h12debd9_1
python-dateutil 2.8.2 pypi_0 pypi pywavelets 1.3.0 pypi_0 pypi pyyaml 6.0 pypi_0 pypi readline 8.1.2 h7f8727e_1
reproject 0.8 pypi_0 pypi requests 2.27.1 pyhd3eb1b0_0
ruamel_yaml 0.15.100 py39h27cfd23_0
scikit-image 0.19.2 pypi_0 pypi scipy 1.8.0 pypi_0 pypi semantic-version 2.9.0 pypi_0 pypi setuptools 58.0.4 py39h06a4308_0
six 1.16.0 pyhd3eb1b0_0
spherical-geometry 1.2.22 pypi_0 pypi sqlite 3.37.0 hc218d9a_0
stcal 0.6.4 pypi_0 pypi stdatamodels 0.4.2 pypi_0 pypi stpipe 0.3.3 pypi_0 pypi stsci-image 2.3.5 pypi_0 pypi stsci-imagestats 1.6.3 pypi_0 pypi stsci-stimage 0.2.5 pypi_0 pypi tifffile 2022.5.4 pypi_0 pypi tk 8.6.11 h1ccaba5_0
tqdm 4.62.3 pyhd3eb1b0_1
tweakwcs 0.7.4 pypi_0 pypi tzdata 2021e hda174b7_0
urllib3 1.26.7 pyhd3eb1b0_0
wheel 0.37.1 pyhd3eb1b0_0
xz 5.2.5 h7b6447c_0
yaml 0.2.5 h7b6447c_0
zlib 1.2.11 h7f8727e_4
Code of Conduct