mcnixon96 commented 1 year ago

FAQ check

[X] Yes, I checked the FAQ and my question has not been addressed.

Instrument

NIRSpec (Stages 1-3)

What happened?

Following the installation instructions and quickstart guide, I have encountered an issue at Stage 2 when running run_eureka.py. The problem appears to be with the endian-ness of some data, but I can't tell which file is actually causing the problem. Presumably this could be fixed using something like [object].byteswap().newbyteorder() somewhere, but I'm not sure where exactly.

Error traceback output

Traceback (most recent call last): File "/n/jwst1/mcnixon/anaconda/anaconda3/envs/eureka/lib/python3.9/site-packages/jwst/pipeline/calwebb_spec2.py", line 117, in process result = self.process_exposure_product( File "/n/jwst1/mcnixon/anaconda/anaconda3/envs/eureka/lib/python3.9/site-packages/jwst/pipeline/calwebb_spec2.py", line 249, in process_exposure_product calibrated = self._process_nirspec_slits(calibrated) File "/n/jwst1/mcnixon/anaconda/anaconda3/envs/eureka/lib/python3.9/site-packages/jwst/pipeline/calwebb_spec2.py", line 463, in _process_nirspec_slits calibrated = self.wavecorr(calibrated) File "/n/jwst1/mcnixon/anaconda/anaconda3/envs/eureka/lib/python3.9/site-packages/stpipe/step.py", line 454, in run step_result = self.process(args) File "/n/jwst1/mcnixon/anaconda/anaconda3/envs/eureka/lib/python3.9/site-packages/jwst/wavecorr/wavecorr_step.py", line 64, in process output_model = wavecorr.do_correction(input_model, reffile) File "/n/jwst1/mcnixon/anaconda/anaconda3/envs/eureka/lib/python3.9/site-packages/jwst/wavecorr/wavecorr.py", line 75, in do_correction apply_zero_point_correction(output_model, wavecorr_file) File "/n/jwst1/mcnixon/anaconda/anaconda3/envs/eureka/lib/python3.9/site-packages/jwst/wavecorr/wavecorr.py", line 132, in apply_zero_point_correction corr, dq_lam = compute_zero_point_correction(lam, reffile, source_xpos, File "/n/jwst1/mcnixon/anaconda/anaconda3/envs/eureka/lib/python3.9/site-packages/jwst/wavecorr/wavecorr.py", line 183, in compute_zero_point_correction correction = offset_model(lam_no_nans 10 -6, [deltax] lam_no_nans.size) File "/n/jwst1/mcnixon/anaconda/anaconda3/envs/eureka/lib/python3.9/site-packages/astropy/modeling/core.py", line 424, in call new_call = make_function_with_signature( File "/n/jwst1/mcnixon/anaconda/anaconda3/envs/eureka/lib/python3.9/site-packages/astropy/modeling/core.py", line 401, in call return super(cls, self).call(inputs, kwargs) File "/n/jwst1/mcnixon/anaconda/anaconda3/envs/eureka/lib/python3.9/site-packages/astropy/modeling/core.py", line 1136, in call outputs = self._generic_evaluate(evaluate, inputs, fill_value, with_bbox) File "/n/jwst1/mcnixon/anaconda/anaconda3/envs/eureka/lib/python3.9/site-packages/astropy/modeling/core.py", line 1098, in _generic_evaluate outputs = evaluate(_inputs) File "/n/jwst1/mcnixon/anaconda/anaconda3/envs/eureka/lib/python3.9/site-packages/astropy/modeling/core.py", line 984, in evaluate return self.evaluate(chain(_inputs, parameters)) File "/n/jwst1/mcnixon/anaconda/anaconda3/envs/eureka/lib/python3.9/site-packages/astropy/modeling/tabular.py", line 242, in evaluate result = interpn( File "/n/jwst1/mcnixon/anaconda/anaconda3/envs/eureka/lib/python3.9/site-packages/scipy/interpolate/_rgi.py", line 654, in interpn return interp(xi) File "/n/jwst1/mcnixon/anaconda/anaconda3/envs/eureka/lib/python3.9/site-packages/scipy/interpolate/_rgi.py", line 336, in call result = evaluate_linear_2d(self.values, File "_rgi_cython.pyx", line 19, in scipy.interpolate._rgi_cython.evaluate_linear_2d ValueError: Big-endian buffer not supported on little-endian compiler Traceback (most recent call last): File "/jwst1/mcnixon/eureka/MyFirstEureka/run_eureka.py", line 24, in s2_meta = s2.calibrateJWST(eventlabel, ecf_path=ecf_path) File "/jwst1/mcnixon/eureka/Eureka/src/eureka/S2_calibrations/s2_calibrate.py", line 175, in calibrateJWST pipeline.run_eurekaS2(filename, meta, log) File "/jwst1/mcnixon/eureka/Eureka/src/eureka/S2_calibrations/s2_calibrate.py", line 271, in run_eurekaS2 self(filename) File "/n/jwst1/mcnixon/anaconda/anaconda3/envs/eureka/lib/python3.9/site-packages/stpipe/step.py", line 454, in run step_result = self.process(args) File "/n/jwst1/mcnixon/anaconda/anaconda3/envs/eureka/lib/python3.9/site-packages/jwst/pipeline/calwebb_spec2.py", line 135, in process raise RuntimeError('\n'.join(failures)) RuntimeError: Traceback (most recent call last): File "/n/jwst1/mcnixon/anaconda/anaconda3/envs/eureka/lib/python3.9/site-packages/jwst/pipeline/calwebb_spec2.py", line 117, in process result = self.process_exposure_product( File "/n/jwst1/mcnixon/anaconda/anaconda3/envs/eureka/lib/python3.9/site-packages/jwst/pipeline/calwebb_spec2.py", line 249, in process_exposure_product calibrated = self._process_nirspec_slits(calibrated) File "/n/jwst1/mcnixon/anaconda/anaconda3/envs/eureka/lib/python3.9/site-packages/jwst/pipeline/calwebb_spec2.py", line 463, in _process_nirspec_slits calibrated = self.wavecorr(calibrated) File "/n/jwst1/mcnixon/anaconda/anaconda3/envs/eureka/lib/python3.9/site-packages/stpipe/step.py", line 454, in run step_result = self.process(args) File "/n/jwst1/mcnixon/anaconda/anaconda3/envs/eureka/lib/python3.9/site-packages/jwst/wavecorr/wavecorr_step.py", line 64, in process output_model = wavecorr.do_correction(input_model, reffile) File "/n/jwst1/mcnixon/anaconda/anaconda3/envs/eureka/lib/python3.9/site-packages/jwst/wavecorr/wavecorr.py", line 75, in do_correction apply_zero_point_correction(output_model, wavecorr_file) File "/n/jwst1/mcnixon/anaconda/anaconda3/envs/eureka/lib/python3.9/site-packages/jwst/wavecorr/wavecorr.py", line 132, in apply_zero_point_correction corr, dq_lam = compute_zero_point_correction(lam, reffile, source_xpos, File "/n/jwst1/mcnixon/anaconda/anaconda3/envs/eureka/lib/python3.9/site-packages/jwst/wavecorr/wavecorr.py", line 183, in compute_zero_point_correction correction = offset_model(lam_no_nans 10 -6, [deltax] lam_no_nans.size) File "/n/jwst1/mcnixon/anaconda/anaconda3/envs/eureka/lib/python3.9/site-packages/astropy/modeling/core.py", line 424, in call new_call = make_function_with_signature( File "/n/jwst1/mcnixon/anaconda/anaconda3/envs/eureka/lib/python3.9/site-packages/astropy/modeling/core.py", line 401, in call return super(cls, self).call(inputs, kwargs) File "/n/jwst1/mcnixon/anaconda/anaconda3/envs/eureka/lib/python3.9/site-packages/astropy/modeling/core.py", line 1136, in call outputs = self._generic_evaluate(evaluate, inputs, fill_value, with_bbox) File "/n/jwst1/mcnixon/anaconda/anaconda3/envs/eureka/lib/python3.9/site-packages/astropy/modeling/core.py", line 1098, in _generic_evaluate outputs = evaluate(_inputs) File "/n/jwst1/mcnixon/anaconda/anaconda3/envs/eureka/lib/python3.9/site-packages/astropy/modeling/core.py", line 984, in evaluate return self.evaluate(chain(_inputs, parameters)) File "/n/jwst1/mcnixon/anaconda/anaconda3/envs/eureka/lib/python3.9/site-packages/astropy/modeling/tabular.py", line 242, in evaluate result = interpn( File "/n/jwst1/mcnixon/anaconda/anaconda3/envs/eureka/lib/python3.9/site-packages/scipy/interpolate/_rgi.py", line 654, in interpn return interp(xi) File "/n/jwst1/mcnixon/anaconda/anaconda3/envs/eureka/lib/python3.9/site-packages/scipy/interpolate/_rgi.py", line 336, in call result = evaluate_linear_2d(self.values, File "_rgi_cython.pyx", line 19, in scipy.interpolate._rgi_cython.evaluate_linear_2d ValueError: Big-endian buffer not supported on little-endian compiler Traceback (most recent call last): File "/jwst1/mcnixon/eureka/MyFirstEureka/run_eureka.py", line 24, in s2_meta = s2.calibrateJWST(eventlabel, ecf_path=ecf_path) File "/jwst1/mcnixon/eureka/Eureka/src/eureka/S2_calibrations/s2_calibrate.py", line 175, in calibrateJWST pipeline.run_eurekaS2(filename, meta, log) File "/jwst1/mcnixon/eureka/Eureka/src/eureka/S2_calibrations/s2_calibrate.py", line 271, in run_eurekaS2 self(filename) File "/n/jwst1/mcnixon/anaconda/anaconda3/envs/eureka/lib/python3.9/site-packages/stpipe/step.py", line 454, in run step_result = self.process(args) File "/n/jwst1/mcnixon/anaconda/anaconda3/envs/eureka/lib/python3.9/site-packages/jwst/pipeline/calwebb_spec2.py", line 135, in process raise RuntimeError('\n'.join(failures)) RuntimeError: Traceback (most recent call last): File "/n/jwst1/mcnixon/anaconda/anaconda3/envs/eureka/lib/python3.9/site-packages/jwst/pipeline/calwebb_spec2.py", line 117, in process result = self.process_exposure_product( File "/n/jwst1/mcnixon/anaconda/anaconda3/envs/eureka/lib/python3.9/site-packages/jwst/pipeline/calwebb_spec2.py", line 249, in process_exposure_product calibrated = self._process_nirspec_slits(calibrated) File "/n/jwst1/mcnixon/anaconda/anaconda3/envs/eureka/lib/python3.9/site-packages/jwst/pipeline/calwebb_spec2.py", line 463, in _process_nirspec_slits calibrated = self.wavecorr(calibrated) File "/n/jwst1/mcnixon/anaconda/anaconda3/envs/eureka/lib/python3.9/site-packages/stpipe/step.py", line 454, in run step_result = self.process(args) File "/n/jwst1/mcnixon/anaconda/anaconda3/envs/eureka/lib/python3.9/site-packages/jwst/wavecorr/wavecorr_step.py", line 64, in process output_model = wavecorr.do_correction(input_model, reffile) File "/n/jwst1/mcnixon/anaconda/anaconda3/envs/eureka/lib/python3.9/site-packages/jwst/wavecorr/wavecorr.py", line 75, in do_correction apply_zero_point_correction(output_model, wavecorr_file) File "/n/jwst1/mcnixon/anaconda/anaconda3/envs/eureka/lib/python3.9/site-packages/jwst/wavecorr/wavecorr.py", line 132, in apply_zero_point_correction corr, dq_lam = compute_zero_point_correction(lam, reffile, source_xpos, File "/n/jwst1/mcnixon/anaconda/anaconda3/envs/eureka/lib/python3.9/site-packages/jwst/wavecorr/wavecorr.py", line 183, in compute_zero_point_correction correction = offset_model(lam_no_nans 10 -6, [deltax] lam_no_nans.size) File "/n/jwst1/mcnixon/anaconda/anaconda3/envs/eureka/lib/python3.9/site-packages/astropy/modeling/core.py", line 424, in call new_call = make_function_with_signature( File "/n/jwst1/mcnixon/anaconda/anaconda3/envs/eureka/lib/python3.9/site-packages/astropy/modeling/core.py", line 401, in call return super(cls, self).call(inputs, kwargs) File "/n/jwst1/mcnixon/anaconda/anaconda3/envs/eureka/lib/python3.9/site-packages/astropy/modeling/core.py", line 1136, in call outputs = self._generic_evaluate(evaluate, inputs, fill_value, with_bbox) File "/n/jwst1/mcnixon/anaconda/anaconda3/envs/eureka/lib/python3.9/site-packages/astropy/modeling/core.py", line 1098, in _generic_evaluate outputs = evaluate(_inputs) File "/n/jwst1/mcnixon/anaconda/anaconda3/envs/eureka/lib/python3.9/site-packages/astropy/modeling/core.py", line 984, in evaluate return self.evaluate(*chain(_inputs, parameters)) File "/n/jwst1/mcnixon/anaconda/anaconda3/envs/eureka/lib/python3.9/site-packages/astropy/modeling/tabular.py", line 242, in evaluate result = interpn( File "/n/jwst1/mcnixon/anaconda/anaconda3/envs/eureka/lib/python3.9/site-packages/scipy/interpolate/_rgi.py", line 654, in interpn return interp(xi) File "/n/jwst1/mcnixon/anaconda/anaconda3/envs/eureka/lib/python3.9/site-packages/scipy/interpolate/_rgi.py", line 336, in call result = evaluate_linear_2d(self.values, File "_rgi_cython.pyx", line 19, in scipy.interpolate._rgi_cython.evaluate_linear_2d ValueError: Big-endian buffer not supported on little-endian compiler

What operating system are you using?

Red Hat Enterprise Linux 8.7 (Ootpa)

What version of Python are you running?

Python 3.9.7

What Python packages do you have installed?

Name Version Build Channel

_libgcc_mutex 0.1 main
_openmp_mutex 5.1 1_gnu
alabaster 0.7.13 pypi_0 pypi asdf 2.14.3 pypi_0 pypi asdf-astropy 0.3.0 pypi_0 pypi asdf-coordinates-schemas 0.1.0 pypi_0 pypi asdf-standard 1.0.3 pypi_0 pypi asdf-transform-schemas 0.3.0 pypi_0 pypi asdf-unit-schemas 0.1.0 pypi_0 pypi asdf-wcs-schemas 0.1.1 pypi_0 pypi asteval 0.9.29 pypi_0 pypi astraeus 0.3 pypi_0 pypi astropy 5.2.1 pypi_0 pypi astropy-healpix 0.7 pypi_0 pypi astroquery 0.4.6 pypi_0 pypi astroscrappy 1.1.0 pypi_0 pypi asttokens 2.2.1 pypi_0 pypi attrs 22.2.0 pypi_0 pypi babel 2.11.0 pypi_0 pypi backcall 0.2.0 pypi_0 pypi batman-package 2.4.9 pypi_0 pypi bayesicfitting 3.1.1 pypi_0 pypi beautifulsoup4 4.11.2 pypi_0 pypi bokeh 2.4.3 pypi_0 pypi bottleneck 1.3.6 pypi_0 pypi ca-certificates 2023.01.10 h06a4308_0
ccdproc 2.4.0 pypi_0 pypi celerite 0.4.2 pypi_0 pypi certifi 2022.12.7 py39h06a4308_0
cffi 1.15.1 pypi_0 pypi cftime 1.6.2 pypi_0 pypi charset-normalizer 3.0.1 pypi_0 pypi cloudpickle 2.2.1 pypi_0 pypi crds 11.16.20 pypi_0 pypi cryptography 39.0.1 pypi_0 pypi cycler 0.11.0 pypi_0 pypi cython 0.29.33 pypi_0 pypi dask 2022.6.0 pypi_0 pypi decorator 5.1.1 pypi_0 pypi docutils 0.19 pypi_0 pypi drizzle 1.13.7 pypi_0 pypi emcee 3.1.4 pypi_0 pypi eureka 0.8 pypi_0 pypi exceptiongroup 1.1.0 pypi_0 pypi executing 1.2.0 pypi_0 pypi exotic-ld 3.0.0 pypi_0 pypi filelock 3.9.0 pypi_0 pypi fonttools 4.38.0 pypi_0 pypi fsspec 2023.1.0 pypi_0 pypi future 0.18.3 pypi_0 pypi george 0.4.0 pypi_0 pypi gwcs 0.18.3 pypi_0 pypi h5netcdf 1.1.0 pypi_0 pypi h5py 3.1.0 pypi_0 pypi html5lib 1.1 pypi_0 pypi idna 3.4 pypi_0 pypi imageio 2.25.1 pypi_0 pypi imagesize 1.4.1 pypi_0 pypi importlib-metadata 6.0.0 pypi_0 pypi iniconfig 2.0.0 pypi_0 pypi ipython 8.10.0 pypi_0 pypi jaraco-classes 3.2.3 pypi_0 pypi jedi 0.18.2 pypi_0 pypi jeepney 0.8.0 pypi_0 pypi jinja2 3.1.2 pypi_0 pypi jmespath 1.0.1 pypi_0 pypi jsonschema 4.9.1 pypi_0 pypi jwst 1.6.0 pypi_0 pypi keyring 23.13.1 pypi_0 pypi kiwisolver 1.4.4 pypi_0 pypi ld_impl_linux-64 2.38 h1181459_1
libffi 3.3 he6710b0_2
libgcc-ng 11.2.0 h1234567_1
libgomp 11.2.0 h1234567_1
libstdcxx-ng 11.2.0 h1234567_1
lmfit 1.1.0 pypi_0 pypi locket 1.0.0 pypi_0 pypi lxml 4.9.2 pypi_0 pypi markupsafe 2.1.2 pypi_0 pypi matplotlib-inline 0.1.6 pypi_0 pypi more-itertools 9.0.0 pypi_0 pypi ncurses 6.4 h6a678d5_0
netcdf4 1.6.2 pypi_0 pypi networkx 3.0 pypi_0 pypi numpy 1.22.0 pypi_0 pypi numpydoc 1.5.0 pypi_0 pypi openssl 1.1.1t h7f8727e_0
packaging 23.0 pypi_0 pypi pandas 1.5.3 pypi_0 pypi parsley 1.3 pypi_0 pypi parso 0.8.3 pypi_0 pypi partd 1.3.0 pypi_0 pypi pexpect 4.8.0 pypi_0 pypi photutils 1.6.0 pypi_0 pypi pickleshare 0.7.5 pypi_0 pypi pillow 9.4.0 pypi_0 pypi pip 22.3.1 py39h06a4308_0
pluggy 1.0.0 pypi_0 pypi poppy 1.0.3 pypi_0 pypi prompt-toolkit 3.0.36 pypi_0 pypi psutil 5.9.4 pypi_0 pypi ptyprocess 0.7.0 pypi_0 pypi pure-eval 0.2.2 pypi_0 pypi pycparser 2.21 pypi_0 pypi pyerfa 2.0.0.1 pypi_0 pypi pygments 2.14.0 pypi_0 pypi pyparsing 3.0.9 pypi_0 pypi pyrsistent 0.19.3 pypi_0 pypi pysynphot 2.0.0 pypi_0 pypi pytest 7.2.1 pypi_0 pypi python 3.9.7 h12debd9_1
python-dateutil 2.8.2 pypi_0 pypi pytz 2022.7.1 pypi_0 pypi pyvo 1.4 pypi_0 pypi pywavelets 1.4.1 pypi_0 pypi pyyaml 6.0 pypi_0 pypi readline 8.2 h5eee18b_0
reproject 0.10.0 pypi_0 pypi requests 2.28.2 pypi_0 pypi scikit-image 0.19.3 pypi_0 pypi scipy 1.10.0 pypi_0 pypi secretstorage 3.3.3 pypi_0 pypi semantic-version 2.10.0 pypi_0 pypi setuptools 65.6.3 py39h06a4308_0
setuptools-scm 7.1.0 pypi_0 pypi six 1.16.0 pypi_0 pypi snowballstemmer 2.2.0 pypi_0 pypi soupsieve 2.4 pypi_0 pypi spherical-geometry 1.2.23 pypi_0 pypi sphinx 6.1.3 pypi_0 pypi sphinxcontrib-applehelp 1.0.4 pypi_0 pypi sphinxcontrib-devhelp 1.0.2 pypi_0 pypi sphinxcontrib-htmlhelp 2.0.1 pypi_0 pypi sphinxcontrib-jsmath 1.0.1 pypi_0 pypi sphinxcontrib-qthelp 1.0.3 pypi_0 pypi sphinxcontrib-serializinghtml 1.1.5 pypi_0 pypi sqlite 3.40.1 h5082296_0
stack-data 0.6.2 pypi_0 pypi stcal 1.3.4 pypi_0 pypi stdatamodels 0.4.5 pypi_0 pypi stpipe 0.4.5 pypi_0 pypi stsci-image 2.3.5 pypi_0 pypi stsci-imagestats 1.6.3 pypi_0 pypi stsci-stimage 0.2.6 pypi_0 pypi svo-filters 0.4.4 pypi_0 pypi tifffile 2023.2.3 pypi_0 pypi tk 8.6.12 h1ccaba5_0
tomli 2.0.1 pypi_0 pypi toolz 0.12.0 pypi_0 pypi tornado 6.2 pypi_0 pypi tqdm 4.64.1 pypi_0 pypi traitlets 5.9.0 pypi_0 pypi tweakwcs 0.8.1 pypi_0 pypi typing-extensions 4.5.0 pypi_0 pypi tzdata 2022g h04d1e81_0
uncertainties 3.1.7 pypi_0 pypi urllib3 1.26.14 pypi_0 pypi wcwidth 0.2.6 pypi_0 pypi webencodings 0.5.1 pypi_0 pypi wheel 0.37.1 pyhd3eb1b0_0
wiimatch 0.3.1 pypi_0 pypi xarray 2023.2.0 pypi_0 pypi xz 5.2.10 h5eee18b_1
zipp 3.13.0 pypi_0 pypi zlib 1.2.13 h5eee18b_0

Code of Conduct

[X] I agree to follow this project's Code of Conduct

taylorbell57 commented 1 year ago

So reading through that error traceback (thanks for copying that - it helps a lot!), it appears the issue you're encountering is not really with any code us Eureka! developers have written but with the jwst pipeline's code which we make heavy use of in Stages 1 and 2. As a result, there's not really anything we can do to fix the issue and you may need to open an issue with them at https://github.com/spacetelescope/jwst.

That said, I know jwst works well on Ubuntu (and almost certainly other similar OSs), so your issue is likely related to the particular linux flavour you have installed. We've had users report issues with Mac M1 and M2 CPUs in the past, and in those cases we found that installing using the conda environment method (instructions here) was able to resolve their issues. I'd recommend first trying that conda installation method to see if that also fixes things for you, and if that doesn't work then you'll have to reach out to the jwst developers directly.

Please let me know here if the conda install method ends up resolving your issue, in which case I'll add a note to the documentation pages about how to solve this big vs little endian issue.