kevin218 / Eureka

Eureka! is a data reduction and analysis pipeline intended for time-series observations with JWST and HST.
https://eurekadocs.readthedocs.io/
MIT License
62 stars 48 forks source link

[Bug]: ASDFValidator in Stage 2 (NIRSpec test) #353

Closed cpiaulet closed 2 years ago

cpiaulet commented 2 years ago

FAQ check

Instrument

NIRSpec (Stages 1-3)

What happened?

I had to re-install Eureka! from scratch and followed the step-by-step installation instructions in the documentation, then went through the Quickstart to make sure I was not forgetting anything. However, when running Stage 2 on the NIRSpec Tiny dataset, I get a weird error which seems to come from some attribute of an asdf object not being there when the Stage1 FITS file is read in (?). I can't run through Stage 2 yet because of this error.

Error traceback output

(eureka_env) caroline@Carolines-MacBook-Pro demos % python -m pdb run_eureka.py

> /Users/caroline/Research/GitHub/Eureka/eureka_analysis/JWST/MyFirstEureka/demos/run_eureka.py(1)<module>()
-> import sys
(Pdb) c
WARNING: Unable to find an output metadata file from Eureka!'s S2 in the folder:
"/Users/caroline/Research/Data/JWST-Sim/NIRSpec-2022SimData/Tiny/Stage1/"
Assuming this S2 data was produced by the JWST pipeline instead.

Starting Stage 2 Reduction
Input directory: /Users/caroline/Research/Data/JWST-Sim/NIRSpec-2022SimData/Tiny/Stage1/
Output directory: /Users/caroline/Research/GitHub/Eureka/eureka_analysis/JWST/MyFirstEureka/Stage2/S2_2022-06-20_wasp39b_run1/
Copying S2 control file

Found 1 data file(s) ending in rateints.fits
2022-06-20 10:46:48,583 - stpipe.EurekaSpec2Pipeline - INFO - EurekaSpec2Pipeline instance created.
2022-06-20 10:46:48,585 - stpipe.EurekaSpec2Pipeline.bkg_subtract - INFO - BackgroundStep instance created.
2022-06-20 10:46:48,587 - stpipe.EurekaSpec2Pipeline.assign_wcs - INFO - AssignWcsStep instance created.
2022-06-20 10:46:48,588 - stpipe.EurekaSpec2Pipeline.imprint_subtract - INFO - ImprintStep instance created.
2022-06-20 10:46:48,589 - stpipe.EurekaSpec2Pipeline.msa_flagging - INFO - MSAFlagOpenStep instance created.
2022-06-20 10:46:48,590 - stpipe.EurekaSpec2Pipeline.extract_2d - INFO - Extract2dStep instance created.
2022-06-20 10:46:48,596 - stpipe.EurekaSpec2Pipeline.master_background - INFO - MasterBackgroundNrsSlitsStep instance created.
2022-06-20 10:46:48,597 - stpipe.EurekaSpec2Pipeline.master_background.flat_field - INFO - FlatFieldStep instance created.
2022-06-20 10:46:48,598 - stpipe.EurekaSpec2Pipeline.master_background.pathloss - INFO - PathLossStep instance created.
2022-06-20 10:46:48,599 - stpipe.EurekaSpec2Pipeline.master_background.barshadow - INFO - BarShadowStep instance created.
2022-06-20 10:46:48,600 - stpipe.EurekaSpec2Pipeline.master_background.photom - INFO - PhotomStep instance created.
2022-06-20 10:46:48,601 - stpipe.EurekaSpec2Pipeline.wavecorr - INFO - WavecorrStep instance created.
2022-06-20 10:46:48,603 - stpipe.EurekaSpec2Pipeline.flat_field - INFO - FlatFieldStep instance created.
2022-06-20 10:46:48,604 - stpipe.EurekaSpec2Pipeline.srctype - INFO - SourceTypeStep instance created.
2022-06-20 10:46:48,605 - stpipe.EurekaSpec2Pipeline.straylight - INFO - StraylightStep instance created.
2022-06-20 10:46:48,608 - stpipe.EurekaSpec2Pipeline.fringe - INFO - FringeStep instance created.
2022-06-20 10:46:48,609 - stpipe.EurekaSpec2Pipeline.pathloss - INFO - PathLossStep instance created.
2022-06-20 10:46:48,611 - stpipe.EurekaSpec2Pipeline.barshadow - INFO - BarShadowStep instance created.
2022-06-20 10:46:48,612 - stpipe.EurekaSpec2Pipeline.photom - INFO - PhotomStep instance created.
2022-06-20 10:46:48,613 - stpipe.EurekaSpec2Pipeline.resample_spec - INFO - ResampleSpecStep instance created.
2022-06-20 10:46:48,615 - stpipe.EurekaSpec2Pipeline.cube_build - INFO - CubeBuildStep instance created.
2022-06-20 10:46:48,617 - stpipe.EurekaSpec2Pipeline.extract_1d - INFO - Extract1dStep instance created.
Starting file 1 of 1
Editing (in place) the waverange in the input file
Traceback (most recent call last):
  File "/Users/caroline/opt/anaconda3/envs/eureka_env/lib/python3.9/pdb.py", line 1723, in main
    pdb._runscript(mainpyfile)
  File "/Users/caroline/opt/anaconda3/envs/eureka_env/lib/python3.9/pdb.py", line 1583, in _runscript
    self.run(statement)
  File "/Users/caroline/opt/anaconda3/envs/eureka_env/lib/python3.9/bdb.py", line 580, in run
    exec(cmd, globals, locals)
  File "<string>", line 1, in <module>
  File "/Users/caroline/Research/GitHub/Eureka/eureka_analysis/JWST/MyFirstEureka/demos/run_eureka.py", line 1, in <module>
    import sys
  File "/Users/caroline/Research/GitHub/Eureka/eureka/S2_calibrations/s2_calibrate.py", line 157, in calibrateJWST
    pipeline.run_eurekaS2(filename, meta, log)
  File "/Users/caroline/Research/GitHub/Eureka/eureka/S2_calibrations/s2_calibrate.py", line 226, in run_eurekaS2
    m.save(filename)
  File "/Users/caroline/opt/anaconda3/envs/eureka_env/lib/python3.9/site-packages/stdatamodels/model_base.py", line 512, in save
    self.to_fits(output_path, *args, **kwargs)
  File "/Users/caroline/opt/anaconda3/envs/eureka_env/lib/python3.9/site-packages/stdatamodels/model_base.py", line 624, in to_fits
    hdulist, tree = fits_support.to_fits(self._instance, self._schema)
  File "/Users/caroline/opt/anaconda3/envs/eureka_env/lib/python3.9/site-packages/stdatamodels/fits_support.py", line 416, in to_fits
    _save_from_schema(hdulist, tree, schema)
  File "/Users/caroline/opt/anaconda3/envs/eureka_env/lib/python3.9/site-packages/stdatamodels/fits_support.py", line 352, in _save_from_schema
    validator.validate(tree, _schema=schema)
  File "/Users/caroline/opt/anaconda3/envs/eureka_env/lib/python3.9/site-packages/jsonschema/validators.py", line 266, in validate
    for error in self.iter_errors(*args, **kwargs):
  File "/Users/caroline/opt/anaconda3/envs/eureka_env/lib/python3.9/site-packages/asdf/schema.py", line 315, in iter_errors
    yield from super(ASDFValidator, self).iter_errors(instance)
  File "/Users/caroline/opt/anaconda3/envs/eureka_env/lib/python3.9/site-packages/jsonschema/validators.py", line 242, in iter_errors
    for error in errors:
  File "/Users/caroline/opt/anaconda3/envs/eureka_env/lib/python3.9/site-packages/stdatamodels/fits_support.py", line 214, in _fits_comment_section_handler
    for error in validator.descend(
  File "/Users/caroline/opt/anaconda3/envs/eureka_env/lib/python3.9/site-packages/jsonschema/validators.py", line 258, in descend
    for error in self.evolve(schema=schema).iter_errors(instance):
  File "/Users/caroline/opt/anaconda3/envs/eureka_env/lib/python3.9/site-packages/asdf/schema.py", line 315, in iter_errors
    yield from super(ASDFValidator, self).iter_errors(instance)
  File "/Users/caroline/opt/anaconda3/envs/eureka_env/lib/python3.9/site-packages/jsonschema/validators.py", line 242, in iter_errors
    for error in errors:
  File "/Users/caroline/opt/anaconda3/envs/eureka_env/lib/python3.9/site-packages/stdatamodels/fits_support.py", line 214, in _fits_comment_section_handler
    for error in validator.descend(
  File "/Users/caroline/opt/anaconda3/envs/eureka_env/lib/python3.9/site-packages/jsonschema/validators.py", line 258, in descend
    for error in self.evolve(schema=schema).iter_errors(instance):
  File "/Users/caroline/opt/anaconda3/envs/eureka_env/lib/python3.9/site-packages/asdf/schema.py", line 315, in iter_errors
    yield from super(ASDFValidator, self).iter_errors(instance)
  File "/Users/caroline/opt/anaconda3/envs/eureka_env/lib/python3.9/site-packages/jsonschema/validators.py", line 241, in iter_errors
    errors = validator(self, v, instance, _schema) or ()
  File "/Users/caroline/opt/anaconda3/envs/eureka_env/lib/python3.9/site-packages/stdatamodels/fits_support.py", line 232, in _fits_element_writer
    hdu = _get_or_make_hdu(validator.hdulist, hdu_name, index=index)
AttributeError: 'ASDFValidator' object has no attribute 'hdulist'

What operating system are you using?

Mac OS Monterey 12.0.1

What version of Python are you running?

Python 3.9.7

What Python packages do you have installed?

# packages in environment at /Users/caroline/opt/anaconda3/envs/eureka_env:
#
# Name                    Version                   Build  Channel
alabaster                 0.7.12                   pypi_0    pypi
appnope                   0.1.3                    pypi_0    pypi
asdf                      2.12.0                   pypi_0    pypi
asdf-astropy              0.2.1                    pypi_0    pypi
asdf-coordinates-schemas  0.1.0                    pypi_0    pypi
asdf-standard             1.0.2                    pypi_0    pypi
asdf-transform-schemas    0.2.2                    pypi_0    pypi
asdf-wcs-schemas          0.1.1                    pypi_0    pypi
asteval                   0.9.27                   pypi_0    pypi
astraeus                  0.1                      pypi_0    pypi
astropy                   5.1                      pypi_0    pypi
astropy-healpix           0.6                      pypi_0    pypi
astroquery                0.4.6                    pypi_0    pypi
astroscrappy              1.1.0                    pypi_0    pypi
asttokens                 2.0.5                    pypi_0    pypi
attrs                     21.4.0                   pypi_0    pypi
babel                     2.10.3                   pypi_0    pypi
backcall                  0.2.0                    pypi_0    pypi
batman-package            2.4.9                    pypi_0    pypi
beautifulsoup4            4.11.1                   pypi_0    pypi
bleach                    5.0.0                    pypi_0    pypi
bokeh                     2.4.3                    pypi_0    pypi
bottleneck                1.3.4                    pypi_0    pypi
ca-certificates           2022.4.26            hecd8cb5_0  
ccdproc                   2.3.1                    pypi_0    pypi
celerite                  0.4.2                    pypi_0    pypi
certifi                   2022.5.18.1      py39hecd8cb5_0  
cftime                    1.6.0                    pypi_0    pypi
charset-normalizer        2.0.12                   pypi_0    pypi
cloudpickle               2.1.0                    pypi_0    pypi
commonmark                0.9.1                    pypi_0    pypi
corner                    2.2.1                    pypi_0    pypi
crds                      11.16.3                  pypi_0    pypi
cycler                    0.11.0                   pypi_0    pypi
cython                    0.29.30                  pypi_0    pypi
dask                      2022.6.0                 pypi_0    pypi
decorator                 5.1.1                    pypi_0    pypi
defusedxml                0.7.1                    pypi_0    pypi
docutils                  0.17.1                   pypi_0    pypi
drizzle                   1.13.4                   pypi_0    pypi
dynesty                   1.2.3                    pypi_0    pypi
emcee                     3.1.2                    pypi_0    pypi
entrypoints               0.4                      pypi_0    pypi
eureka                    0.0.1                     dev_0    <develop>
executing                 0.8.3                    pypi_0    pypi
fastjsonschema            2.15.3                   pypi_0    pypi
filelock                  3.7.1                    pypi_0    pypi
fonttools                 4.33.3                   pypi_0    pypi
fsspec                    2022.5.0                 pypi_0    pypi
future                    0.18.2                   pypi_0    pypi
george                    0.4.0                    pypi_0    pypi
gwcs                      0.18.1                   pypi_0    pypi
h5netcdf                  1.0.0                    pypi_0    pypi
h5py                      3.1.0                    pypi_0    pypi
html5lib                  1.1                      pypi_0    pypi
idna                      3.3                      pypi_0    pypi
imageio                   2.19.3                   pypi_0    pypi
imagesize                 1.3.0                    pypi_0    pypi
importlib-metadata        4.11.4                   pypi_0    pypi
iniconfig                 1.1.1                    pypi_0    pypi
ipython                   8.4.0                    pypi_0    pypi
jedi                      0.18.1                   pypi_0    pypi
jinja2                    3.1.2                    pypi_0    pypi
jmespath                  1.0.1                    pypi_0    pypi
jsonschema                4.6.0                    pypi_0    pypi
jupyter-client            7.3.4                    pypi_0    pypi
jupyter-core              4.10.0                   pypi_0    pypi
jupyterlab-pygments       0.2.2                    pypi_0    pypi
jwst                      1.3.3                    pypi_0    pypi
keyring                   23.6.0                   pypi_0    pypi
kiwisolver                1.4.3                    pypi_0    pypi
libcxx                    12.0.0               h2f01273_0  
libffi                    3.3                  hb1e8313_2  
lmfit                     1.0.3                    pypi_0    pypi
locket                    1.0.0                    pypi_0    pypi
lxml                      4.9.0                    pypi_0    pypi
markdown-it-py            2.1.0                    pypi_0    pypi
markupsafe                2.1.1                    pypi_0    pypi
matplotlib                3.5.2                    pypi_0    pypi
matplotlib-inline         0.1.3                    pypi_0    pypi
mdit-py-plugins           0.3.0                    pypi_0    pypi
mdurl                     0.1.1                    pypi_0    pypi
mistune                   0.8.4                    pypi_0    pypi
myst-parser               0.18.0                   pypi_0    pypi
nbclient                  0.6.4                    pypi_0    pypi
nbconvert                 6.5.0                    pypi_0    pypi
nbformat                  5.4.0                    pypi_0    pypi
nbsphinx                  0.8.9                    pypi_0    pypi
ncurses                   6.3                  hca72f7f_2  
nest-asyncio              1.5.5                    pypi_0    pypi
netcdf4                   1.5.8                    pypi_0    pypi
networkx                  2.8.4                    pypi_0    pypi
numpy                     1.22.4                   pypi_0    pypi
numpydoc                  1.4.0                    pypi_0    pypi
openssl                   1.1.1o               hca72f7f_0  
packaging                 21.3                     pypi_0    pypi
pandas                    1.4.2                    pypi_0    pypi
pandocfilters             1.5.0                    pypi_0    pypi
parsley                   1.3                      pypi_0    pypi
parso                     0.8.3                    pypi_0    pypi
partd                     1.2.0                    pypi_0    pypi
pexpect                   4.8.0                    pypi_0    pypi
photutils                 1.1.0                    pypi_0    pypi
pickleshare               0.7.5                    pypi_0    pypi
pillow                    9.1.1                    pypi_0    pypi
pip                       21.2.4           py39hecd8cb5_0  
pluggy                    1.0.0                    pypi_0    pypi
poppy                     1.0.2                    pypi_0    pypi
prompt-toolkit            3.0.29                   pypi_0    pypi
psutil                    5.9.1                    pypi_0    pypi
ptyprocess                0.7.0                    pypi_0    pypi
pure-eval                 0.2.2                    pypi_0    pypi
py                        1.11.0                   pypi_0    pypi
pyerfa                    2.0.0.1                  pypi_0    pypi
pygments                  2.12.0                   pypi_0    pypi
pyparsing                 3.0.9                    pypi_0    pypi
pyrsistent                0.18.1                   pypi_0    pypi
pysynphot                 2.0.0                    pypi_0    pypi
pytest                    7.1.2                    pypi_0    pypi
python                    3.9.7                h88f2d9e_1  
python-dateutil           2.8.2                    pypi_0    pypi
pytz                      2022.1                   pypi_0    pypi
pyvo                      1.3                      pypi_0    pypi
pywavelets                1.3.0                    pypi_0    pypi
pyyaml                    6.0                      pypi_0    pypi
pyzmq                     23.1.0                   pypi_0    pypi
readline                  8.1.2                hca72f7f_1  
recommonmark              0.7.1                    pypi_0    pypi
reproject                 0.8                      pypi_0    pypi
requests                  2.28.0                   pypi_0    pypi
scikit-image              0.19.3                   pypi_0    pypi
scipy                     1.8.1                    pypi_0    pypi
semantic-version          2.10.0                   pypi_0    pypi
setuptools                61.2.0           py39hecd8cb5_0  
six                       1.16.0                   pypi_0    pypi
snowballstemmer           2.2.0                    pypi_0    pypi
soupsieve                 2.3.2.post1              pypi_0    pypi
spherical-geometry        1.2.22                   pypi_0    pypi
sphinx                    5.0.2                    pypi_0    pypi
sphinx-rtd-theme          1.0.0                    pypi_0    pypi
sphinxcontrib-applehelp   1.0.2                    pypi_0    pypi
sphinxcontrib-devhelp     1.0.2                    pypi_0    pypi
sphinxcontrib-htmlhelp    2.0.0                    pypi_0    pypi
sphinxcontrib-jsmath      1.0.1                    pypi_0    pypi
sphinxcontrib-qthelp      1.0.3                    pypi_0    pypi
sphinxcontrib-serializinghtml 1.1.5                    pypi_0    pypi
sqlite                    3.38.3               h707629a_0  
stack-data                0.3.0                    pypi_0    pypi
stcal                     0.7.3                    pypi_0    pypi
stdatamodels              0.2.4                    pypi_0    pypi
stpipe                    0.3.3                    pypi_0    pypi
stsci-image               2.3.5                    pypi_0    pypi
stsci-imagestats          1.6.3                    pypi_0    pypi
stsci-stimage             0.2.5                    pypi_0    pypi
svo-filters               0.4.2                    pypi_0    pypi
tifffile                  2022.5.4                 pypi_0    pypi
tinycss2                  1.1.1                    pypi_0    pypi
tk                        8.6.12               h5d9f67b_0  
tomli                     2.0.1                    pypi_0    pypi
toolz                     0.11.2                   pypi_0    pypi
tornado                   6.1                      pypi_0    pypi
tqdm                      4.64.0                   pypi_0    pypi
traitlets                 5.3.0                    pypi_0    pypi
tweakwcs                  0.7.4                    pypi_0    pypi
typing-extensions         4.2.0                    pypi_0    pypi
tzdata                    2022a                hda174b7_0  
uncertainties             3.1.6                    pypi_0    pypi
urllib3                   1.26.9                   pypi_0    pypi
wcwidth                   0.2.5                    pypi_0    pypi
webencodings              0.5.1                    pypi_0    pypi
wheel                     0.37.1             pyhd3eb1b0_0  
xarray                    2022.3.0                 pypi_0    pypi
xz                        5.2.5                hca72f7f_1  
zipp                      3.8.0                    pypi_0    pypi
zlib                      1.2.12               h4dc903c_2  

Code of Conduct

taylorbell57 commented 2 years ago

If a fresh install doesn't work, can you try replacing the following from the run_eurekaS2 method of the EurekaSpec2Pipeline class in s2_reduce.py

        if meta.waverange_start is not None:
            # Control the dispersion extraction
            # FIX: Does not actually change dispersion direction extraction
            if meta.waverange_start is None:
                # Only log this once
                log.writelog('Editing (in place) the waverange in the input '
                             'file')
            with datamodels.open(filename) as m:
                m.meta.wcsinfo.waverange_start = meta.waverange_start
                m.save(filename)

        if meta.waverange_end is not None:
            # Control the dispersion extraction
            # FIX: Does not actually change dispersion direction extraction
            if meta.waverange_start is None:
                # Only log this once
                log.writelog('Editing (in place) the waverange in the input '
                             'file')
            with datamodels.open(filename) as m:
                m.meta.wcsinfo.waverange_end = meta.waverange_end
                m.save(filename)

with

        if meta.waverange_start is not None:
            # Control the dispersion extraction
            # FIX: Does not actually change dispersion direction extraction
            if meta.waverange_start is None:
                # Only log this once
                log.writelog('Editing (in place) the waverange in the input '
                             'file')
            with fits.open(filename, mode='update') as hdulist:
                hdulist[1].header['WAVSTART'] = meta.waverange_start

        if meta.waverange_end is not None:
            # Control the dispersion extraction
            # FIX: Does not actually change dispersion direction extraction
            if meta.waverange_start is None:
                # Only log this once
                log.writelog('Editing (in place) the waverange in the input '
                             'file')
            with fits.open(filename, mode='update') as hdulist:
                hdulist[1].header['WAVEND'] = meta.waverange_end
erinmmay commented 2 years ago

Hi! Kevin asked me to respond here because I had a similar, possibly related, problem.

When I updated Eureka! I also updated my conda environment with the new yml file and one of the asdf packages (specifically asdf-astropy) ended up having some missing components for an unknown reason, probably just an installation hiccup. I had the same "missing attributes" error and the solution was to pip uninstall that specific package and reinstall, so I recommend tracing back to which package ASDFValidator comes from and uninstalling/reinstalling.

cpiaulet commented 2 years ago

ok, I uninstalled and re-installed asdf-astropy and asdf==2.8.1 and now it ran through! Thanks for your help, the issue can be closed.