search

kevin218 / Eureka

Eureka! is a data reduction and analysis pipeline intended for time-series observations with JWST and HST.
https://eurekadocs.readthedocs.io/
MIT License
62 stars 48 forks source link

[Bug]: time array shape does not agree with sci array shape for MIRI drift NEW data of NGTS10 #362

Closed sebastian-zieba closed 2 years ago

sebastian-zieba commented 2 years ago

FAQ check

Instrument

MIRI (Stages 1-3)

What happened?

I ran Stage 3 of MIRI using the Stage 2 Output on the Box (MIRI Stage 2 output drift NEW). There seems to be a dimension issue in the time array.

For segment no1, time has the shape of (46) and sci has (45, 72, 416). So time is too long by one element.

I believe the bug arises from line 118 in miri.py. Instead of data.attrs['intend'] it should be data.attrs['intend'] - 1. Like in line 113 or 132. The same error can be seen in line 126.

I can create a quick PR if that's actually a bug and not my fault in some way..

Error traceback output

/home/zieba/anaconda3/envs/eureka_repo_reorg/bin/python /home/zieba/Desktop/Projects/Open_source/Eureka_repo_reorg/Eureka/demos/JWST/run_eureka.py WARNING: The package jwst has not been installed. As a result, Eureka!'s Stages 1 and 2 will not work. WARNING: Unable to find an output metadata file from Eureka!'s S2 in the folder: "/home/zieba/Desktop/Data/JWST-Sim-Data_Challenge/MIRI/Stage_2_output_drift_NEW/" Assuming this S2 data was produced by the JWST pipeline instead.

Starting Stage 3 Reduction

Input directory: /home/zieba/Desktop/Data/JWST-Sim-Data_Challenge/MIRI/Stage_2_output_drift_NEW/ Output directory: /home/zieba/Desktop/Data/JWST-Sim-Data_Challenge/MIRI/my_Stage_2_output_drift_NEW_June2022/S3_2022-06-27_miri_lrs_template_run3/ap4_bg8/ Using ap=4, bg=8 Copying S3 control file

Found 42 data file(s) ending in calints.fits Reading file 1 of 42 WARNING: The wavelength for the simulated MIRI data are currently hardcoded because they are not in the .fits files themselves WARNING: The timestamps for the simulated MIRI data are currently hardcoded because they are not in the .fits files themselves Traceback (most recent call last): File "/home/zieba/Desktop/Projects/Open_source/Eureka_repo_reorg/Eureka/demos/JWST/run_eureka.py", line 27, in s3_spec, s3_meta = s3.reduce(eventlabel, ecf_path=ecf_path) File "/home/zieba/anaconda3/envs/eureka_repo_reorg/lib/python3.9/site-packages/eureka/S3_data_reduction/s3_reduce.py", line 214, in reduce data, meta = inst.read(meta.segment_list[m], data, meta) File "/home/zieba/anaconda3/envs/eureka_repo_reorg/lib/python3.9/site-packages/eureka/S3_data_reduction/miri.py", line 161, in read data['flux'] = xrio.makeFluxLikeDA(sci, time, flux_units, time_units, File "/home/zieba/anaconda3/envs/eureka_repo_reorg/lib/python3.9/site-packages/astraeus/xarrayIO.py", line 114, in makeFluxLikeDA da = xr.DataArray( File "/home/zieba/anaconda3/envs/eureka_repo_reorg/lib/python3.9/site-packages/xarray/core/dataarray.py", line 402, in init coords, dims = _infer_coords_and_dims(data.shape, coords, dims) File "/home/zieba/anaconda3/envs/eureka_repo_reorg/lib/python3.9/site-packages/xarray/core/dataarray.py", line 152, in _infer_coords_and_dims raise ValueError( ValueError: conflicting sizes for dimension 'time': length 45 on the data but length 46 on coordinate 'time'

Process finished with exit code 1

What operating system are you using?

Ubuntu 18.04.6 LTS

What version of Python are you running?

Python 3.9

What Python packages do you have installed?

_libgcc_mutex 0.1 main
_openmp_mutex 5.1 1_gnu
alabaster 0.7.12 pypi_0 pypi asdf 2.12.0 pypi_0 pypi asdf-astropy 0.2.1 pypi_0 pypi asdf-coordinates-schemas 0.1.0 pypi_0 pypi asdf-standard 1.0.2 pypi_0 pypi asdf-transform-schemas 0.2.2 pypi_0 pypi asdf-wcs-schemas 0.1.1 pypi_0 pypi asteval 0.9.27 pypi_0 pypi astraeus 0.2 pypi_0 pypi astropy 5.1 pypi_0 pypi astropy-healpix 0.6 pypi_0 pypi astroquery 0.4.6 pypi_0 pypi astroscrappy 1.1.0 pypi_0 pypi asttokens 2.0.5 pypi_0 pypi attrs 21.4.0 pypi_0 pypi babel 2.10.3 pypi_0 pypi backcall 0.2.0 pypi_0 pypi batman-package 2.4.9 pypi_0 pypi beautifulsoup4 4.11.1 pypi_0 pypi bokeh 2.4.3 pypi_0 pypi bottleneck 1.3.4 pypi_0 pypi ca-certificates 2022.4.26 h06a4308_0
ccdproc 2.3.1 pypi_0 pypi celerite 0.4.2 pypi_0 pypi certifi 2022.5.18.1 py39h06a4308_0
cffi 1.15.0 pypi_0 pypi cftime 1.6.0 pypi_0 pypi charset-normalizer 2.0.12 pypi_0 pypi cloudpickle 2.1.0 pypi_0 pypi corner 2.2.1 pypi_0 pypi crds 11.16.3 pypi_0 pypi cryptography 37.0.2 pypi_0 pypi cycler 0.11.0 pypi_0 pypi cython 0.29.30 pypi_0 pypi dask 2022.6.0 pypi_0 pypi decorator 5.1.1 pypi_0 pypi docutils 0.18.1 pypi_0 pypi dynesty 1.2.3 pypi_0 pypi emcee 3.1.2 pypi_0 pypi eureka 0.0.1 pypi_0 pypi executing 0.8.3 pypi_0 pypi filelock 3.7.1 pypi_0 pypi fonttools 4.33.3 pypi_0 pypi fsspec 2022.5.0 pypi_0 pypi future 0.18.2 pypi_0 pypi george 0.4.0 pypi_0 pypi gwcs 0.18.1 pypi_0 pypi h5netcdf 1.0.0 pypi_0 pypi h5py 3.1.0 pypi_0 pypi html5lib 1.1 pypi_0 pypi idna 3.3 pypi_0 pypi imageio 2.19.3 pypi_0 pypi imagesize 1.3.0 pypi_0 pypi importlib-metadata 4.11.4 pypi_0 pypi iniconfig 1.1.1 pypi_0 pypi ipython 8.4.0 pypi_0 pypi jedi 0.18.1 pypi_0 pypi jeepney 0.8.0 pypi_0 pypi jinja2 3.1.2 pypi_0 pypi jmespath 1.0.1 pypi_0 pypi jsonschema 4.6.0 pypi_0 pypi keyring 23.6.0 pypi_0 pypi kiwisolver 1.4.3 pypi_0 pypi ld_impl_linux-64 2.38 h1181459_1
libffi 3.3 he6710b0_2
libgcc-ng 11.2.0 h1234567_1
libgomp 11.2.0 h1234567_1
libstdcxx-ng 11.2.0 h1234567_1
lmfit 1.0.3 pypi_0 pypi locket 1.0.0 pypi_0 pypi lxml 4.9.0 pypi_0 pypi markupsafe 2.1.1 pypi_0 pypi matplotlib 3.5.2 pypi_0 pypi matplotlib-inline 0.1.3 pypi_0 pypi ncurses 6.3 h7f8727e_2
netcdf4 1.5.8 pypi_0 pypi networkx 2.8.4 pypi_0 pypi numpy 1.22.4 pypi_0 pypi numpydoc 1.4.0 pypi_0 pypi openssl 1.1.1o h7f8727e_0
packaging 21.3 pypi_0 pypi pandas 1.4.2 pypi_0 pypi parsley 1.3 pypi_0 pypi parso 0.8.3 pypi_0 pypi partd 1.2.0 pypi_0 pypi pexpect 4.8.0 pypi_0 pypi photutils 1.4.0 pypi_0 pypi pickleshare 0.7.5 pypi_0 pypi pillow 9.1.1 pypi_0 pypi pip 21.2.4 py39h06a4308_0
pluggy 1.0.0 pypi_0 pypi prompt-toolkit 3.0.29 pypi_0 pypi psutil 5.9.1 pypi_0 pypi ptyprocess 0.7.0 pypi_0 pypi pure-eval 0.2.2 pypi_0 pypi py 1.11.0 pypi_0 pypi pycparser 2.21 pypi_0 pypi pyerfa 2.0.0.1 pypi_0 pypi pygments 2.12.0 pypi_0 pypi pyparsing 3.0.9 pypi_0 pypi pyrsistent 0.18.1 pypi_0 pypi pysynphot 2.0.0 pypi_0 pypi pytest 7.1.2 pypi_0 pypi python 3.9.7 h12debd9_1
python-dateutil 2.8.2 pypi_0 pypi pytz 2022.1 pypi_0 pypi pyvo 1.3 pypi_0 pypi pywavelets 1.3.0 pypi_0 pypi pyyaml 6.0 pypi_0 pypi readline 8.1.2 h7f8727e_1
reproject 0.8 pypi_0 pypi requests 2.28.0 pypi_0 pypi scikit-image 0.19.3 pypi_0 pypi scipy 1.8.1 pypi_0 pypi secretstorage 3.3.2 pypi_0 pypi semantic-version 2.10.0 pypi_0 pypi setuptools 61.2.0 py39h06a4308_0
six 1.16.0 pypi_0 pypi snowballstemmer 2.2.0 pypi_0 pypi soupsieve 2.3.2.post1 pypi_0 pypi sphinx 5.0.2 pypi_0 pypi sphinxcontrib-applehelp 1.0.2 pypi_0 pypi sphinxcontrib-devhelp 1.0.2 pypi_0 pypi sphinxcontrib-htmlhelp 2.0.0 pypi_0 pypi sphinxcontrib-jsmath 1.0.1 pypi_0 pypi sphinxcontrib-qthelp 1.0.3 pypi_0 pypi sphinxcontrib-serializinghtml 1.1.5 pypi_0 pypi sqlite 3.38.3 hc218d9a_0
stack-data 0.3.0 pypi_0 pypi stdatamodels 0.4.3 pypi_0 pypi svo-filters 0.4.2 pypi_0 pypi tifffile 2022.5.4 pypi_0 pypi tk 8.6.12 h1ccaba5_0
tomli 2.0.1 pypi_0 pypi toolz 0.11.2 pypi_0 pypi tornado 6.1 pypi_0 pypi tqdm 4.64.0 pypi_0 pypi traitlets 5.3.0 pypi_0 pypi typing-extensions 4.2.0 pypi_0 pypi tzdata 2022a hda174b7_0
uncertainties 3.1.7 pypi_0 pypi urllib3 1.26.9 pypi_0 pypi wcwidth 0.2.5 pypi_0 pypi webencodings 0.5.1 pypi_0 pypi wheel 0.37.1 pyhd3eb1b0_0
xarray 2022.3.0 pypi_0 pypi xz 5.2.5 h7f8727e_1
zipp 3.8.0 pypi_0 pypi zlib 1.2.12 h7f8727e_2

Code of Conduct

kevin218 commented 2 years ago

This might be a problem with the simulated data. If I recall correctly, there was an additional integration in a separate file that got dropped during S1 (because the code fails when there's only a single integration).

Alternatively, this could mean we aren't handling multiple segments correctly.

taylorbell57 commented 2 years ago

I'm surprised you encountered this since I used Eureka! to analyze those same data without any issue. I'll look into this now to see what might have happened, but first a quick comment. I believe there has been bugs with the MIRISIM_TSO pipeline and its setting of the INTSTART and INTEND FITS header values because for some data sets I needed to use data.attrs['intend'] and for other data sets I needed to use data.attrs['intend'] - 1. Hopefully this wont be an issue with the real data, but for now it has been a coin toss as to which we'll actually need to use

taylorbell57 commented 2 years ago

Hmm, very strange - I too now need to have data.attrs['intend'] - 1 on the line you referred to. No idea why that changed... I'll make a tiny PR to patch that