search

kevin218 / Eureka

Eureka! is a data reduction and analysis pipeline intended for time-series observations with JWST.
https://eurekadocs.readthedocs.io/
MIT License
56 stars 43 forks source link

[Bug]: 'MetaClass' object has no attribute 'photfile' #619

Closed lheinke closed 4 months ago

lheinke commented 4 months ago

FAQ check

Instrument

MIRI (Stages 1-3)

What happened?

I am trying to reduce MIRI transit spectroscopy data starting from stage 2 calints outputs of the jwst pipeline that I downloaded from MAST. The pipeline seems to recognize that the output is not from previous Eureka! stages, but nonetheless quickly fails at the last line of output:

WARNING: Unable to find an output metadata file from Eureka!'s S2 in the folder:
"/home/lheinke/JWST_XXXX_X/MAST/Stage2/"
Assuming this S2 data was produced by the JWST pipeline instead.

Found 8 data file(s) ending in calints.fits

Starting Stage 3 Reduction

Input directory: /home/lheinke/JWST_XXXX_X/MAST/Stage2/
Output directory: /home/lheinke/JWST_XXXX_X/Eureka!/Stage3/S3_2024-02-12_1281_1_run3/ap5_bg11/
Using ap=5, bg=11, expand=1
Copying S3 control file
Starting batch 1 of 1 with 8 files
  Reading file 1...
  Reading file 2...
  Reading file 3...
  Reading file 4...
  Reading file 5...
  Reading file 6...
  Reading file 7...
  Reading file 8...
  Concatenating files...
  Masking NaNs/infs in data arrays...
  ERR has 103266 NaNs/infs, which is 0.81% of all pixels.
  V0 has 103266 NaNs/infs, which is 0.81% of all pixels.
  Locating source position...
    Source position on detector is row 25.
  Using provided gain=3.1 to convert units to electrons...
  Converting from brightness units (MJy/sr) to electrons...

Error traceback output

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[4], [line 56](vscode-notebook-cell:?execution_count=4&line=56)
     [53](vscode-notebook-cell:?execution_count=4&line=53) with open(f'S3_{eventlabel}.ecf', 'w') as f:
     [54](vscode-notebook-cell:?execution_count=4&line=54)     f.write(stage3_config)
---> [56](vscode-notebook-cell:?execution_count=4&line=56) s3_spec, s3_meta = s3_reduce.reduce(eventlabel)

File [/home/lheinke/.local/repos/Eureka/src/eureka/S3_data_reduction/s3_reduce.py:454](https://vscode-remote+ssh-002dremote-server.vscode-resource.vscode-cdn.net/home/lheinke/.local/repos/Eureka/src/eureka/S3_data_reduction/s3_reduce.py:454), in reduce(eventlabel, ecf_path, s2_meta, input_meta)
    [450](https://vscode-remote+ssh-002dremote-server.vscode-resource.vscode-cdn.net/home/lheinke/.local/repos/Eureka/src/eureka/S3_data_reduction/s3_reduce.py:450)     data = inst.calibrated_spectra(data, meta, log)
    [451](https://vscode-remote+ssh-002dremote-server.vscode-resource.vscode-cdn.net/home/lheinke/.local/repos/Eureka/src/eureka/S3_data_reduction/s3_reduce.py:451) else:
    [452](https://vscode-remote+ssh-002dremote-server.vscode-resource.vscode-cdn.net/home/lheinke/.local/repos/Eureka/src/eureka/S3_data_reduction/s3_reduce.py:452)     # Convert flux units to electrons
    [453](https://vscode-remote+ssh-002dremote-server.vscode-resource.vscode-cdn.net/home/lheinke/.local/repos/Eureka/src/eureka/S3_data_reduction/s3_reduce.py:453)     # (eg. MJy/sr -> DN -> Electrons)
--> [454](https://vscode-remote+ssh-002dremote-server.vscode-resource.vscode-cdn.net/home/lheinke/.local/repos/Eureka/src/eureka/S3_data_reduction/s3_reduce.py:454)     data, meta = b2f.convert_to_e(data, meta, log)
    [456](https://vscode-remote+ssh-002dremote-server.vscode-resource.vscode-cdn.net/home/lheinke/.local/repos/Eureka/src/eureka/S3_data_reduction/s3_reduce.py:456) # Perform outlier rejection of
    [457](https://vscode-remote+ssh-002dremote-server.vscode-resource.vscode-cdn.net/home/lheinke/.local/repos/Eureka/src/eureka/S3_data_reduction/s3_reduce.py:457) # full frame along time axis
    [458](https://vscode-remote+ssh-002dremote-server.vscode-resource.vscode-cdn.net/home/lheinke/.local/repos/Eureka/src/eureka/S3_data_reduction/s3_reduce.py:458) if hasattr(meta, 'ff_outlier') and meta.ff_outlier:

File [/home/lheinke/.local/repos/Eureka/src/eureka/S3_data_reduction/bright2flux.py:330](https://vscode-remote+ssh-002dremote-server.vscode-resource.vscode-cdn.net/home/lheinke/.local/repos/Eureka/src/eureka/S3_data_reduction/bright2flux.py:330), in convert_to_e(data, meta, log)
    [326](https://vscode-remote+ssh-002dremote-server.vscode-resource.vscode-cdn.net/home/lheinke/.local/repos/Eureka/src/eureka/S3_data_reduction/bright2flux.py:326) if data.attrs['shdr']['BUNIT'] == 'MJy/sr':
    [327](https://vscode-remote+ssh-002dremote-server.vscode-resource.vscode-cdn.net/home/lheinke/.local/repos/Eureka/src/eureka/S3_data_reduction/bright2flux.py:327)     # Convert from brightness units (MJy/sr) to DN/s
    [328](https://vscode-remote+ssh-002dremote-server.vscode-resource.vscode-cdn.net/home/lheinke/.local/repos/Eureka/src/eureka/S3_data_reduction/bright2flux.py:328)     log.writelog('  Converting from brightness units (MJy/sr) to '
    [329](https://vscode-remote+ssh-002dremote-server.vscode-resource.vscode-cdn.net/home/lheinke/.local/repos/Eureka/src/eureka/S3_data_reduction/bright2flux.py:329)                  'electrons...')
--> [330](https://vscode-remote+ssh-002dremote-server.vscode-resource.vscode-cdn.net/home/lheinke/.local/repos/Eureka/src/eureka/S3_data_reduction/bright2flux.py:330)     data = bright2dn(data, meta)
    [331](https://vscode-remote+ssh-002dremote-server.vscode-resource.vscode-cdn.net/home/lheinke/.local/repos/Eureka/src/eureka/S3_data_reduction/bright2flux.py:331)     data = dn2electrons(data, meta)
    [332](https://vscode-remote+ssh-002dremote-server.vscode-resource.vscode-cdn.net/home/lheinke/.local/repos/Eureka/src/eureka/S3_data_reduction/bright2flux.py:332) elif data.attrs['shdr']['BUNIT'] == 'MJy':
    [333](https://vscode-remote+ssh-002dremote-server.vscode-resource.vscode-cdn.net/home/lheinke/.local/repos/Eureka/src/eureka/S3_data_reduction/bright2flux.py:333)     # Convert from brightness units (MJy) to DN/s

File [/home/lheinke/.local/repos/Eureka/src/eureka/S3_data_reduction/bright2flux.py:157](https://vscode-remote+ssh-002dremote-server.vscode-resource.vscode-cdn.net/home/lheinke/.local/repos/Eureka/src/eureka/S3_data_reduction/bright2flux.py:157), in bright2dn(data, meta, mjy)
    [125](https://vscode-remote+ssh-002dremote-server.vscode-resource.vscode-cdn.net/home/lheinke/.local/repos/Eureka/src/eureka/S3_data_reduction/bright2flux.py:125) """This function converts the data, uncertainty, and variance arrays from
    [126](https://vscode-remote+ssh-002dremote-server.vscode-resource.vscode-cdn.net/home/lheinke/.local/repos/Eureka/src/eureka/S3_data_reduction/bright2flux.py:126) brightness units (MJy/sr) or (MJy) to raw units (DN).
    [127](https://vscode-remote+ssh-002dremote-server.vscode-resource.vscode-cdn.net/home/lheinke/.local/repos/Eureka/src/eureka/S3_data_reduction/bright2flux.py:127) 
   (...)
    [154](https://vscode-remote+ssh-002dremote-server.vscode-resource.vscode-cdn.net/home/lheinke/.local/repos/Eureka/src/eureka/S3_data_reduction/bright2flux.py:154)     Convert to using Xarray Dataset
    [155](https://vscode-remote+ssh-002dremote-server.vscode-resource.vscode-cdn.net/home/lheinke/.local/repos/Eureka/src/eureka/S3_data_reduction/bright2flux.py:155) """
    [156](https://vscode-remote+ssh-002dremote-server.vscode-resource.vscode-cdn.net/home/lheinke/.local/repos/Eureka/src/eureka/S3_data_reduction/bright2flux.py:156) # Load response function and wavelength
--> [157](https://vscode-remote+ssh-002dremote-server.vscode-resource.vscode-cdn.net/home/lheinke/.local/repos/Eureka/src/eureka/S3_data_reduction/bright2flux.py:157) phot = fits.getdata(meta.photfile)
    [158](https://vscode-remote+ssh-002dremote-server.vscode-resource.vscode-cdn.net/home/lheinke/.local/repos/Eureka/src/eureka/S3_data_reduction/bright2flux.py:158) if meta.inst == 'nircam':
    [159](https://vscode-remote+ssh-002dremote-server.vscode-resource.vscode-cdn.net/home/lheinke/.local/repos/Eureka/src/eureka/S3_data_reduction/bright2flux.py:159)     if meta.photometry:

AttributeError: 'MetaClass' object has no attribute 'photfile'

What operating system are you using?

Ubuntu 20.04.6 LTS

What version of Python are you running?

Python 3.9.7

What Python packages do you have installed?

# packages in environment at /home/lheinke/.local/miniconda3/envs/eureka_git:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
alabaster                 0.7.16                   pypi_0    pypi
asciitree                 0.3.3                    pypi_0    pypi
asdf                      3.0.1                    pypi_0    pypi
asdf-astropy              0.5.0                    pypi_0    pypi
asdf-coordinates-schemas  0.2.0                    pypi_0    pypi
asdf-standard             1.0.3                    pypi_0    pypi
asdf-transform-schemas    0.4.0                    pypi_0    pypi
asdf-unit-schemas         0.1.0                    pypi_0    pypi
asdf-wcs-schemas          0.3.0                    pypi_0    pypi
asteval                   0.9.31                   pypi_0    pypi
astraeus                  0.3                      pypi_0    pypi
astropy                   6.0.0                    pypi_0    pypi
astropy-healpix           1.0.2                    pypi_0    pypi
astropy-iers-data         0.2024.2.12.0.31.0          pypi_0    pypi
astroquery                0.4.6                    pypi_0    pypi
astroscrappy              1.1.0                    pypi_0    pypi
asttokens                 2.4.1                    pypi_0    pypi
attrs                     23.2.0                   pypi_0    pypi
babel                     2.14.0                   pypi_0    pypi
batman-package            2.4.9                    pypi_0    pypi
bayesicfitting            3.2.0                    pypi_0    pypi
beautifulsoup4            4.12.3                   pypi_0    pypi
bokeh                     2.4.3                    pypi_0    pypi
bottleneck                1.3.7                    pypi_0    pypi
ca-certificates           2024.2.2             hbcca054_0    conda-forge
ccdproc                   2.4.1                    pypi_0    pypi
celerite2                 0.3.0                    pypi_0    pypi
certifi                   2024.2.2                 pypi_0    pypi
cffi                      1.16.0                   pypi_0    pypi
cftime                    1.6.3                    pypi_0    pypi
charset-normalizer        3.3.2                    pypi_0    pypi
click                     8.1.7                    pypi_0    pypi
cloudpickle               3.0.0                    pypi_0    pypi
comm                      0.2.1                    pypi_0    pypi
contourpy                 1.2.0                    pypi_0    pypi
corner                    2.2.2                    pypi_0    pypi
crds                      11.17.16                 pypi_0    pypi
cryptography              42.0.2                   pypi_0    pypi
cycler                    0.12.1                   pypi_0    pypi
dask                      2024.2.0                 pypi_0    pypi
debugpy                   1.8.1                    pypi_0    pypi
decorator                 5.1.1                    pypi_0    pypi
docutils                  0.20.1                   pypi_0    pypi
drizzle                   1.14.4                   pypi_0    pypi
dynesty                   2.1.3                    pypi_0    pypi
emcee                     3.1.4                    pypi_0    pypi
eureka                    0.11.dev94+g7d7f7041          pypi_0    pypi
exceptiongroup            1.2.0                    pypi_0    pypi
executing                 2.0.1                    pypi_0    pypi
exotic-ld                 3.0.0                    pypi_0    pypi
fasteners                 0.19                     pypi_0    pypi
filelock                  3.13.1                   pypi_0    pypi
fonttools                 4.48.1                   pypi_0    pypi
fsspec                    2024.2.0                 pypi_0    pypi
future                    0.18.3                   pypi_0    pypi
george                    0.4.1                    pypi_0    pypi
gwcs                      0.20.0                   pypi_0    pypi
h5netcdf                  1.3.0                    pypi_0    pypi
h5py                      3.10.0                   pypi_0    pypi
html5lib                  1.1                      pypi_0    pypi
idna                      3.6                      pypi_0    pypi
imageio                   2.34.0                   pypi_0    pypi
imagesize                 1.4.1                    pypi_0    pypi
importlib-metadata        7.0.1                    pypi_0    pypi
importlib-resources       6.1.1                    pypi_0    pypi
iniconfig                 2.0.0                    pypi_0    pypi
ipykernel                 6.29.2                   pypi_0    pypi
ipython                   8.18.1                   pypi_0    pypi
jaraco-classes            3.3.1                    pypi_0    pypi
jedi                      0.19.1                   pypi_0    pypi
jeepney                   0.8.0                    pypi_0    pypi
jinja2                    3.1.3                    pypi_0    pypi
jmespath                  1.0.1                    pypi_0    pypi
jsonschema                4.21.1                   pypi_0    pypi
jsonschema-specifications 2023.12.1                pypi_0    pypi
jupyter-client            8.6.0                    pypi_0    pypi
jupyter-core              5.7.1                    pypi_0    pypi
jwst                      1.12.2                   pypi_0    pypi
keyring                   24.3.0                   pypi_0    pypi
kiwisolver                1.4.5                    pypi_0    pypi
lazy-loader               0.3                      pypi_0    pypi
ld_impl_linux-64          2.40                 h41732ed_0    conda-forge
libffi                    3.4.2                h7f98852_5    conda-forge
libgcc-ng                 13.2.0               h807b86a_5    conda-forge
libgomp                   13.2.0               h807b86a_5    conda-forge
libsqlite                 3.45.1               h2797004_0    conda-forge
libzlib                   1.2.13               hd590300_5    conda-forge
lmfit                     1.2.2                    pypi_0    pypi
locket                    1.0.0                    pypi_0    pypi
markupsafe                2.1.5                    pypi_0    pypi
matplotlib                3.8.2                    pypi_0    pypi
matplotlib-inline         0.1.6                    pypi_0    pypi
mc3                       3.1.3                    pypi_0    pypi
more-itertools            10.2.0                   pypi_0    pypi
ncurses                   6.4                  h59595ed_2    conda-forge
nest-asyncio              1.6.0                    pypi_0    pypi
netcdf4                   1.6.5                    pypi_0    pypi
networkx                  3.2.1                    pypi_0    pypi
numcodecs                 0.12.1                   pypi_0    pypi
numpy                     1.24.4                   pypi_0    pypi
numpydoc                  1.6.0                    pypi_0    pypi
opencv-python-headless    4.9.0.80                 pypi_0    pypi
openssl                   3.2.1                hd590300_0    conda-forge
packaging                 23.2                     pypi_0    pypi
pandas                    2.2.0                    pypi_0    pypi
parsley                   1.3                      pypi_0    pypi
parso                     0.8.3                    pypi_0    pypi
partd                     1.4.1                    pypi_0    pypi
pexpect                   4.9.0                    pypi_0    pypi
photutils                 1.10.0                   pypi_0    pypi
pillow                    10.2.0                   pypi_0    pypi
pip                       24.0               pyhd8ed1ab_0    conda-forge
platformdirs              4.2.0                    pypi_0    pypi
pluggy                    1.4.0                    pypi_0    pypi
poppy                     1.1.1                    pypi_0    pypi
prompt-toolkit            3.0.43                   pypi_0    pypi
psutil                    5.9.8                    pypi_0    pypi
ptyprocess                0.7.0                    pypi_0    pypi
pure-eval                 0.2.2                    pypi_0    pypi
pycparser                 2.21                     pypi_0    pypi
pyerfa                    2.0.1.1                  pypi_0    pypi
pygments                  2.17.2                   pypi_0    pypi
pyparsing                 3.1.1                    pypi_0    pypi
pysynphot                 2.0.0                    pypi_0    pypi
pytest                    8.0.0                    pypi_0    pypi
python                    3.9.7           hf930737_3_cpython    conda-forge
python-dateutil           2.8.2                    pypi_0    pypi
pytz                      2024.1                   pypi_0    pypi
pyvo                      1.5                      pypi_0    pypi
pyyaml                    6.0.1                    pypi_0    pypi
pyzmq                     25.1.2                   pypi_0    pypi
readline                  8.2                  h8228510_1    conda-forge
referencing               0.33.0                   pypi_0    pypi
reproject                 0.13.0                   pypi_0    pypi
requests                  2.31.0                   pypi_0    pypi
rpds-py                   0.17.1                   pypi_0    pypi
scikit-image              0.22.0                   pypi_0    pypi
scipy                     1.9.3                    pypi_0    pypi
secretstorage             3.3.3                    pypi_0    pypi
semantic-version          2.10.0                   pypi_0    pypi
setuptools                69.0.3             pyhd8ed1ab_0    conda-forge
setuptools-scm            8.0.4                    pypi_0    pypi
six                       1.16.0                   pypi_0    pypi
snowballstemmer           2.2.0                    pypi_0    pypi
soupsieve                 2.5                      pypi_0    pypi
spherical-geometry        1.3.1                    pypi_0    pypi
sphinx                    7.2.6                    pypi_0    pypi
sphinxcontrib-applehelp   1.0.8                    pypi_0    pypi
sphinxcontrib-devhelp     1.0.6                    pypi_0    pypi
sphinxcontrib-htmlhelp    2.0.5                    pypi_0    pypi
sphinxcontrib-jsmath      1.0.1                    pypi_0    pypi
sphinxcontrib-qthelp      1.0.7                    pypi_0    pypi
sphinxcontrib-serializinghtml 1.1.10                   pypi_0    pypi
sqlite                    3.45.1               h2c6b66d_0    conda-forge
stack-data                0.6.3                    pypi_0    pypi
stcal                     1.4.4                    pypi_0    pypi
stdatamodels              1.8.4                    pypi_0    pypi
stpipe                    0.5.1                    pypi_0    pypi
stsci-image               2.3.5                    pypi_0    pypi
stsci-imagestats          1.8.0                    pypi_0    pypi
stsci-stimage             0.2.6                    pypi_0    pypi
svo-filters               0.4.4                    pypi_0    pypi
tabulate                  0.9.0                    pypi_0    pypi
tifffile                  2024.1.30                pypi_0    pypi
tk                        8.6.13          noxft_h4845f30_101    conda-forge
tomli                     2.0.1                    pypi_0    pypi
toolz                     0.12.1                   pypi_0    pypi
tornado                   6.4                      pypi_0    pypi
tqdm                      4.66.2                   pypi_0    pypi
traitlets                 5.14.1                   pypi_0    pypi
tweakwcs                  0.8.6                    pypi_0    pypi
typing-extensions         4.9.0                    pypi_0    pypi
tzdata                    2024.1                   pypi_0    pypi
uncertainties             3.1.7                    pypi_0    pypi
urllib3                   2.2.0                    pypi_0    pypi
wcwidth                   0.2.13                   pypi_0    pypi
webencodings              0.5.1                    pypi_0    pypi
wheel                     0.42.0             pyhd8ed1ab_0    conda-forge
wiimatch                  0.3.2                    pypi_0    pypi
xarray                    2024.1.1                 pypi_0    pypi
xz                        5.2.6                h166bdaf_0    conda-forge
zarr                      2.16.1                   pypi_0    pypi
zipp                      3.17.0                   pypi_0    pypi
zlib                      1.2.13               hd590300_5    conda-forge

Code of Conduct

taylorbell57 commented 4 months ago

Thanks for submitting this - very strange, I've been running Stage 3 myself lately and haven't encountered this. There are multiple checks before the point where it crashed that should've populated meta.photfile with a value. I'm just confused because the below-linked lines shouldn't let this happen: https://github.com/kevin218/Eureka/blob/7d7f7041ba77d086cb03ad1eb3ac167ef39ae8ea/src/eureka/S3_data_reduction/bright2flux.py#L305-L314

taylorbell57 commented 4 months ago

Oh wait, I see what happened here. You downloaded the Stage 2 files (_calints) from MAST which isn't recommended since they automatically run the photometric calibration step; this step is not helpful for time-series observations and is a bit hard to undo cleanly. I strongly recommend you download the Stage 1 files (_rateints) from MAST at a minimum, and also consider starting instead with the raw _uncal files from MAST (there are some potential improvements to be made during Stage 1, including increaseing the jump threshold value to something higher like 7 which will give fewer false positives and may give better overall results, but that should be verified by you).

That said, while starting from the MAST Stage 2 files is strongly discouraged, there is nonetheless a bug that you've encountered. Because you've provided a gain value in your ECF you ended up triggering this clause: https://github.com/kevin218/Eureka/blob/7d7f7041ba77d086cb03ad1eb3ac167ef39ae8ea/src/eureka/S3_data_reduction/bright2flux.py#L292-L295 which does not populate the photfile attribute of meta since I'd mistakenly assumed we wouldn't need the photfile in that case.

This should be an easy fix for me and I'll hopefully have a patch submitted today/tomorrow, but if you don't want to wait for that, in the meantime you have two options:

  1. Download the _rateints or _uncal files from MAST and start running Eureka! from an earlier stage, or
  2. Comment-out the gain line in your Stage 3 ECF which should automatically populate the photfile and gainfile attributes with the current defaults from CRDS pmap. I believe there's been a new gainfile released which is more accurate than the old constant value of 5.5 that STScI had assumed and might be more accurate than the constant 3.1 value I'd estimated.
lheinke commented 4 months ago

Thanks for the quick response, Taylor! In the meantime, I did already run Eureka! starting from stage 1 and indeed the problem disappears.

I was aware that photometric flux calibration is not needed for transit spectroscopy, but did not know that it would cause problems if applied. Is this mentioned somewhere in the Eureka! docs? Otherwise, it might be useful to warn against using the MAST stage 2 output explicitly.