search

kevin218 / Eureka

Eureka! is a data reduction and analysis pipeline intended for time-series observations with JWST.
https://eurekadocs.readthedocs.io/
MIT License
60 stars 47 forks source link

[Bug]: MIRI stage 3 flux calibration #662

Closed Pat-Wachiraphan closed 3 months ago

Pat-Wachiraphan commented 3 months ago

FAQ check

Instrument

MIRI (Stages 1-3)

What happened?

I tried flux calibration option on Stage 3 (MIRI LRS). It ran into what I think is a bug(?)

Error traceback output

Found 9 data file(s) ending in calints.fits

Starting Stage 3 Reduction

Input directory: Mydir Output directory: Mydir Using ap=4, bg=9, expand=1 Copying S3 control file Starting batch 1 of 1 with 9 files Reading file 1... Reading file 2... Reading file 3... Reading file 4... Reading file 5... Reading file 6... Reading file 7... Reading file 8... Reading file 9... Concatenating files... Masking NaNs/infs in data arrays... FLUX has 2354787 NaNs/infs, which is 0.81% of all pixels. ERR has 2354787 NaNs/infs, which is 0.81% of all pixels. V0 has 2354787 NaNs/infs, which is 0.81% of all pixels. Locating source position... Source position on detector is row 25. Converting from MJy/sr to mJy... Traceback (most recent call last): File "Mydir/run_eureka.py", line 25, in s3_spec, s3_meta = s3.reduce(eventlabel, ecf_path=ecf_path,s2_meta=s2_meta) File "Mydir/Eureka/src/eureka/S3_data_reduction/s3_reduce.py", line 450, in reduce data = inst.calibrated_spectra(data, meta, log) File "Mydir/Eureka/src/eureka/S3_data_reduction/miri.py", line 479, in calibrated_spectra data['flux'].data = 1e9data.shdr['PIXAR_SR'] File "Mydir/miniforge3/envs/eureka/lib/python3.10/site-packages/astropy/io/fits/header.py", line 170, in getitem card = self._cards[self._cardindex(key)] File "Mydir/miniforge3/envs/eureka/lib/python3.10/site-packages/astropy/io/fits/header.py", line 1725, in _cardindex raise KeyError(f"Keyword {keyword!r} not found.") KeyError: "Keyword 'PIXAR_SR' not found."

What operating system are you using?

Mac OS Sonoma (14.5)

What version of Python are you running?

Python 3.10.14

What Python packages do you have installed?

packages in environment at Mydir/miniforge3/envs/eureka:

#

Name Version Build Channel

alabaster 0.7.16 pypi_0 pypi asciitree 0.3.3 pypi_0 pypi asdf 3.2.0 pypi_0 pypi asdf-astropy 0.6.1 pypi_0 pypi asdf-coordinates-schemas 0.3.0 pypi_0 pypi asdf-standard 1.1.1 pypi_0 pypi asdf-transform-schemas 0.5.0 pypi_0 pypi asdf-wcs-schemas 0.4.0 pypi_0 pypi asteval 0.9.32 pypi_0 pypi astraeus 0.3 pypi_0 pypi astropy 6.1.0 pypi_0 pypi astropy-healpix 1.0.3 pypi_0 pypi astropy-iers-data 0.2024.5.20.0.29.40 pypi_0 pypi astroquery 0.4.7 pypi_0 pypi astroscrappy 1.2.0 pypi_0 pypi asttokens 2.4.1 pyhd8ed1ab_0 conda-forge attrs 23.2.0 pypi_0 pypi babel 2.15.0 pypi_0 pypi backports-tarfile 1.1.1 pypi_0 pypi batman-package 2.4.9 pypi_0 pypi bayesicfitting 3.2.1 pypi_0 pypi beautifulsoup4 4.12.3 pypi_0 pypi bokeh 2.4.3 pypi_0 pypi bottleneck 1.3.8 pypi_0 pypi bzip2 1.0.8 h10d778d_5 conda-forge ca-certificates 2024.2.2 h8857fd0_0 conda-forge ccdproc 2.4.2 pypi_0 pypi celerite2 0.3.1 pypi_0 pypi certifi 2024.2.2 pypi_0 pypi cftime 1.6.3 pypi_0 pypi charset-normalizer 3.3.2 pypi_0 pypi click 8.1.7 pypi_0 pypi cloudpickle 3.0.0 pypi_0 pypi contourpy 1.2.1 pypi_0 pypi corner 2.2.2 pypi_0 pypi crds 11.17.22 pypi_0 pypi cycler 0.12.1 pypi_0 pypi dask 2024.5.1 pypi_0 pypi decorator 5.1.1 pyhd8ed1ab_0 conda-forge dill 0.3.8 pypi_0 pypi docutils 0.21.2 pypi_0 pypi drizzle 1.15.1 pypi_0 pypi dynesty 2.1.3 pypi_0 pypi emcee 3.1.6 pypi_0 pypi eureka 0.11.dev286+gde5b373b pypi_0 pypi exceptiongroup 1.2.0 pyhd8ed1ab_2 conda-forge executing 2.0.1 pyhd8ed1ab_0 conda-forge exotic-ld 3.0.0 pypi_0 pypi fasteners 0.19 pypi_0 pypi filelock 3.14.0 pypi_0 pypi fonttools 4.51.0 pypi_0 pypi fsspec 2024.5.0 pypi_0 pypi future 1.0.0 pypi_0 pypi george 0.4.1 pypi_0 pypi gwcs 0.21.0 pypi_0 pypi h5netcdf 1.3.0 pypi_0 pypi h5py 3.11.0 pypi_0 pypi html5lib 1.1 pypi_0 pypi idna 3.7 pypi_0 pypi imageio 2.34.1 pypi_0 pypi imagesize 1.4.1 pypi_0 pypi importlib-metadata 7.1.0 pypi_0 pypi iniconfig 2.0.0 pypi_0 pypi ipython 8.24.0 pyh707e725_0 conda-forge jaraco-classes 3.4.0 pypi_0 pypi jaraco-context 5.3.0 pypi_0 pypi jaraco-functools 4.0.1 pypi_0 pypi jedi 0.19.1 pyhd8ed1ab_0 conda-forge jinja2 3.1.4 pypi_0 pypi jmespath 1.0.1 pypi_0 pypi jsonschema 4.22.0 pypi_0 pypi jsonschema-specifications 2023.12.1 pypi_0 pypi jwst 1.14.0 pypi_0 pypi keyring 25.2.1 pypi_0 pypi kiwisolver 1.4.5 pypi_0 pypi lazy-loader 0.4 pypi_0 pypi libffi 3.4.2 h0d85af4_5 conda-forge libsqlite 3.45.3 h92b6c6a_0 conda-forge libzlib 1.2.13 h8a1eda9_5 conda-forge lmfit 1.3.1 pypi_0 pypi locket 1.0.0 pypi_0 pypi markupsafe 2.1.5 pypi_0 pypi matplotlib 3.9.0 pypi_0 pypi matplotlib-inline 0.1.7 pyhd8ed1ab_0 conda-forge mc3 3.1.3 pypi_0 pypi more-itertools 10.2.0 pypi_0 pypi ncurses 6.5 h5846eda_0 conda-forge netcdf4 1.6.5 pypi_0 pypi networkx 3.3 pypi_0 pypi numcodecs 0.12.1 pypi_0 pypi numpy 1.24.4 pypi_0 pypi numpydoc 1.7.0 pypi_0 pypi opencv-python-headless 4.9.0.80 pypi_0 pypi openssl 3.3.0 h87427d6_2 conda-forge packaging 24.0 pypi_0 pypi pandas 2.2.2 pypi_0 pypi parsley 1.3 pypi_0 pypi parso 0.8.4 pyhd8ed1ab_0 conda-forge partd 1.4.2 pypi_0 pypi pexpect 4.9.0 pyhd8ed1ab_0 conda-forge photutils 1.12.0 pypi_0 pypi pickleshare 0.7.5 py_1003 conda-forge pillow 10.3.0 pypi_0 pypi pip 24.0 pyhd8ed1ab_0 conda-forge pluggy 1.5.0 pypi_0 pypi poppy 1.1.1 pypi_0 pypi prompt-toolkit 3.0.42 pyha770c72_0 conda-forge psutil 5.9.8 pypi_0 pypi ptyprocess 0.7.0 pyhd3deb0d_0 conda-forge pure_eval 0.2.2 pyhd8ed1ab_0 conda-forge pyerfa 2.0.1.4 pypi_0 pypi pygments 2.18.0 pyhd8ed1ab_0 conda-forge pyparsing 3.1.2 pypi_0 pypi pysynphot 2.0.0 pypi_0 pypi pytest 8.2.1 pypi_0 pypi python 3.10.14 h00d2728_0_cpython conda-forge python-dateutil 2.9.0.post0 pypi_0 pypi pytz 2024.1 pypi_0 pypi pyvo 1.5.1 pypi_0 pypi pyyaml 6.0.1 pypi_0 pypi readline 8.2 h9e318b2_1 conda-forge referencing 0.35.1 pypi_0 pypi reproject 0.13.1 pypi_0 pypi requests 2.32.1 pypi_0 pypi rpds-py 0.18.1 pypi_0 pypi scikit-image 0.23.2 pypi_0 pypi scipy 1.13.0 pypi_0 pypi semantic-version 2.10.0 pypi_0 pypi setuptools 69.5.1 pyhd8ed1ab_0 conda-forge setuptools-scm 8.1.0 pypi_0 pypi six 1.16.0 pyh6c4a22f_0 conda-forge snowballstemmer 2.2.0 pypi_0 pypi soupsieve 2.5 pypi_0 pypi spherical-geometry 1.3.1 pypi_0 pypi sphinx 7.3.7 pypi_0 pypi sphinxcontrib-applehelp 1.0.8 pypi_0 pypi sphinxcontrib-devhelp 1.0.6 pypi_0 pypi sphinxcontrib-htmlhelp 2.0.5 pypi_0 pypi sphinxcontrib-jsmath 1.0.1 pypi_0 pypi sphinxcontrib-qthelp 1.0.7 pypi_0 pypi sphinxcontrib-serializinghtml 1.1.10 pypi_0 pypi stack_data 0.6.2 pyhd8ed1ab_0 conda-forge stcal 1.7.1 pypi_0 pypi stdatamodels 1.10.1 pypi_0 pypi stpipe 0.5.2 pypi_0 pypi stsci-image 2.3.8 pypi_0 pypi stsci-imagestats 1.6.3 pypi_0 pypi stsci-stimage 0.2.9 pypi_0 pypi svo-filters 0.4.4 pypi_0 pypi synphot 1.4.0 pypi_0 pypi tabulate 0.9.0 pypi_0 pypi tifffile 2024.5.10 pypi_0 pypi tk 8.6.13 h1abcd95_1 conda-forge tomli 2.0.1 pypi_0 pypi toolz 0.12.1 pypi_0 pypi tornado 6.4 pypi_0 pypi tqdm 4.66.4 pypi_0 pypi traitlets 5.14.3 pyhd8ed1ab_0 conda-forge tweakwcs 0.8.7 pypi_0 pypi typing_extensions 4.11.0 pyha770c72_0 conda-forge tzdata 2024.1 pypi_0 pypi uncertainties 3.1.7 pypi_0 pypi urllib3 2.2.1 pypi_0 pypi wcwidth 0.2.13 pyhd8ed1ab_0 conda-forge webencodings 0.5.1 pypi_0 pypi wheel 0.43.0 pyhd8ed1ab_1 conda-forge wiimatch 0.3.2 pypi_0 pypi xarray 2024.5.0 pypi_0 pypi xz 5.2.6 h775f41a_0 conda-forge zarr 2.18.1 pypi_0 pypi zipp 3.18.2 pypi_0 pypi

Code of Conduct

taylorbell57 commented 3 months ago

My suspicion is that your input Stage 2 files did not use the photometric calibration step - you need to set skip_photom to False in Stage 2 in order to produce calibrated spectra. Please check that you ran Stage 2 with skip_photom set to False and that those photometrically calibrated Stage 2 files are what you're using in Stage 3 when trying to produce flux-calibrated spectra

Pat-Wachiraphan commented 3 months ago

got it, thank a lot! feel free to close this issue.

zkbt commented 3 months ago

@taylorbell57 , thank you so much! I wonder, can we please impose on you to ask a scientific question about this? Would you treat a photometrically calibrated Stage 3 output as a good estimate of the intrinsic absolutely calibrated flux of the star, which could be used to convert eclipse depths into planetary fluxes? Or would you do some other magic, like using a really large extraction aperture, applying an aperture correction, and/or worrying more about non-linearity?

zkbt commented 3 months ago

For context, I think we (= @Pat-Wachiraphan) are finding the Eureka "flux calibrated spectra" during eclipse are delightfully consistent across three visits (like 1% or better), but systematically lower by about 20% compared to a model SED coming soon from @hdiamondlowe.

zkbt commented 3 months ago

(Sorry to use a Closed issue, but I wanted to keep it attached to @Pat-Wachiraphan 's question!)