[Bug]: Setting meta.centroids fails when nreads is not constant in each WFC3 exposure.

[jbrande]

FAQ check

• [X] Yes, I checked the FAQ and my question has not been addressed.

Instrument

Other (any stage)

What happened?

Some WFC3 datasets don't have consistent numbers of reads in each exposure. When this is the case, setting meta.centroids = np.array(meta.centroids) in the WFC conclusion_step() will fail, due to very so slightly ragged array with an error ValueError: setting an array element with a sequence. The requested array has an inhomogeneous shape after 1 dimensions. The detected shape was (85,) + inhomogeneous part.

I guess that step should have a dtype=object parameter to preserve all the read-level information for future steps rather than trying to trim or pad the number of reads.

Error traceback output

Traceback (most recent call last):
  File "/home/jbrande/miniconda3/envs/eureka/lib/python3.10/pdb.py", line 1723, in main
    pdb._runscript(mainpyfile)
  File "/home/jbrande/miniconda3/envs/eureka/lib/python3.10/pdb.py", line 1583, in _runscript
    self.run(statement)
  File "/home/jbrande/miniconda3/envs/eureka/lib/python3.10/bdb.py", line 598, in run
    exec(cmd, globals, locals)
  File "<string>", line 1, in <module>
  File "/mnt/c/Users/yonib/OneDrive - University of Kansas/ExoLab/hst/TOI1759/ecfs_laptop/run_eureka_wfc3.py", line 16, in <module>
    s3_spec, s3_meta = s3.reduce(eventlabel, ecf_path=ecf_path)
  File "/home/jbrande/Planetary/Eureka/src/eureka/S3_data_reduction/s3_reduce.py", line 755, in reduce
    spec, meta, log = inst.conclusion_step(spec, meta, log)
  File "/home/jbrande/Planetary/Eureka/src/eureka/S3_data_reduction/wfc3.py", line 162, in conclusion_step
    meta.centroids = np.array(meta.centroids)
ValueError: setting an array element with a sequence. The requested array has an inhomogeneous shape after 1 dimensions. The detected shape was (85,) + inhomogeneous part.

What operating system are you using?

Ubuntu 22.04.03 on WSLg on Win10

What version of Python are you running?

3.10.14

What Python packages do you have installed?

_libgcc_mutex             0.1                        main
_openmp_mutex             5.1                       1_gnu
alabaster                 0.7.16                   pypi_0    pypi
anyio                     4.3.0                    pypi_0    pypi
argon2-cffi               23.1.0                   pypi_0    pypi
argon2-cffi-bindings      21.2.0                   pypi_0    pypi
arrow                     1.3.0                    pypi_0    pypi
asciitree                 0.3.3                    pypi_0    pypi
asdf                      3.2.0                    pypi_0    pypi
asdf-astropy              0.6.1                    pypi_0    pypi
asdf-coordinates-schemas  0.3.0                    pypi_0    pypi
asdf-standard             1.1.1                    pypi_0    pypi
asdf-transform-schemas    0.5.0                    pypi_0    pypi
asdf-wcs-schemas          0.4.0                    pypi_0    pypi
asteval                   0.9.32                   pypi_0    pypi
astraeus                  0.3                      pypi_0    pypi
astropy                   6.0.1                    pypi_0    pypi
astropy-healpix           1.0.3                    pypi_0    pypi
astropy-iers-data         0.2024.4.8.0.32.4          pypi_0    pypi
astroquery                0.4.7                    pypi_0    pypi
astroscrappy              1.1.0                    pypi_0    pypi
asttokens                 2.4.1                    pypi_0    pypi
async-lru                 2.0.4                    pypi_0    pypi
attrs                     23.2.0                   pypi_0    pypi
babel                     2.14.0                   pypi_0    pypi
backports-tarfile         1.0.0                    pypi_0    pypi
batman-package            2.4.9                    pypi_0    pypi
bayesicfitting            3.2.1                    pypi_0    pypi
beautifulsoup4            4.12.3                   pypi_0    pypi
bleach                    6.1.0                    pypi_0    pypi
bokeh                     2.4.3                    pypi_0    pypi
bottleneck                1.3.8                    pypi_0    pypi
bzip2                     1.0.8                h5eee18b_5
ca-certificates           2024.3.11            h06a4308_0
ccdproc                   2.4.1                    pypi_0    pypi
celerite2                 0.3.1                    pypi_0    pypi
certifi                   2024.2.2                 pypi_0    pypi
cffi                      1.16.0                   pypi_0    pypi
cftime                    1.6.3                    pypi_0    pypi
charset-normalizer        3.3.2                    pypi_0    pypi
click                     8.1.7                    pypi_0    pypi
cloudpickle               3.0.0                    pypi_0    pypi
comm                      0.2.2                    pypi_0    pypi
contourpy                 1.2.1                    pypi_0    pypi
corner                    2.2.2                    pypi_0    pypi
crds                      11.17.19                 pypi_0    pypi
cryptography              42.0.5                   pypi_0    pypi
cycler                    0.12.1                   pypi_0    pypi
dask                      2024.4.1                 pypi_0    pypi
debugpy                   1.8.1                    pypi_0    pypi
decorator                 5.1.1                    pypi_0    pypi
defusedxml                0.7.1                    pypi_0    pypi
dill                      0.3.8                    pypi_0    pypi
docutils                  0.20.1                   pypi_0    pypi
drizzle                   1.15.1                   pypi_0    pypi
dynesty                   2.1.3                    pypi_0    pypi
emcee                     3.1.4                    pypi_0    pypi
eureka                    0.11.dev276+g4e12d23d          pypi_0    pypi
exceptiongroup            1.2.0                    pypi_0    pypi
executing                 2.0.1                    pypi_0    pypi
exotic-ld                 3.0.0                    pypi_0    pypi
fasteners                 0.19                     pypi_0    pypi
fastjsonschema            2.19.1                   pypi_0    pypi
filelock                  3.13.4                   pypi_0    pypi
fonttools                 4.51.0                   pypi_0    pypi
fqdn                      1.5.1                    pypi_0    pypi
fsspec                    2024.3.1                 pypi_0    pypi
future                    1.0.0                    pypi_0    pypi
george                    0.4.1                    pypi_0    pypi
gwcs                      0.21.0                   pypi_0    pypi
h11                       0.14.0                   pypi_0    pypi
h5netcdf                  1.3.0                    pypi_0    pypi
h5py                      3.11.0                   pypi_0    pypi
html5lib                  1.1                      pypi_0    pypi
httpcore                  1.0.5                    pypi_0    pypi
httpx                     0.27.0                   pypi_0    pypi
idna                      3.7                      pypi_0    pypi
image-registration        0.2.9.dev9+ge11d912          pypi_0    pypi
imageio                   2.34.0                   pypi_0    pypi
imagesize                 1.4.1                    pypi_0    pypi
importlib-metadata        7.1.0                    pypi_0    pypi
iniconfig                 2.0.0                    pypi_0    pypi
ipykernel                 6.29.4                   pypi_0    pypi
ipython                   8.23.0                   pypi_0    pypi
ipywidgets                8.1.2                    pypi_0    pypi
isoduration               20.11.0                  pypi_0    pypi
jaraco-classes            3.4.0                    pypi_0    pypi
jaraco-context            5.3.0                    pypi_0    pypi
jaraco-functools          4.0.0                    pypi_0    pypi
jedi                      0.19.1                   pypi_0    pypi
jeepney                   0.8.0                    pypi_0    pypi
jinja2                    3.1.3                    pypi_0    pypi
jmespath                  1.0.1                    pypi_0    pypi
json5                     0.9.24                   pypi_0    pypi
jsonpointer               2.4                      pypi_0    pypi
jsonschema                4.21.1                   pypi_0    pypi
jsonschema-specifications 2023.12.1                pypi_0    pypi
jupyter                   1.0.0                    pypi_0    pypi
jupyter-client            8.6.1                    pypi_0    pypi
jupyter-console           6.6.3                    pypi_0    pypi
jupyter-core              5.7.2                    pypi_0    pypi
jupyter-events            0.10.0                   pypi_0    pypi
jupyter-lsp               2.2.5                    pypi_0    pypi
jupyter-server            2.14.0                   pypi_0    pypi
jupyter-server-terminals  0.5.3                    pypi_0    pypi
jupyterlab                4.1.6                    pypi_0    pypi
jupyterlab-pygments       0.3.0                    pypi_0    pypi
jupyterlab-server         2.26.0                   pypi_0    pypi
jupyterlab-widgets        3.0.10                   pypi_0    pypi
jwst                      1.14.0                   pypi_0    pypi
keyring                   25.1.0                   pypi_0    pypi
kiwisolver                1.4.5                    pypi_0    pypi
lazy-loader               0.4                      pypi_0    pypi
ld_impl_linux-64          2.38                 h1181459_1
libffi                    3.4.4                h6a678d5_0
libgcc-ng                 11.2.0               h1234567_1
libgomp                   11.2.0               h1234567_1
libstdcxx-ng              11.2.0               h1234567_1
libuuid                   1.41.5               h5eee18b_0
lmfit                     1.3.0                    pypi_0    pypi
locket                    1.0.0                    pypi_0    pypi
markupsafe                2.1.5                    pypi_0    pypi
matplotlib                3.8.4                    pypi_0    pypi
matplotlib-inline         0.1.6                    pypi_0    pypi
mc3                       3.1.3                    pypi_0    pypi
mistune                   3.0.2                    pypi_0    pypi
more-itertools            10.2.0                   pypi_0    pypi
nbclient                  0.10.0                   pypi_0    pypi
nbconvert                 7.16.3                   pypi_0    pypi
nbformat                  5.10.4                   pypi_0    pypi
ncurses                   6.4                  h6a678d5_0
nest-asyncio              1.6.0                    pypi_0    pypi
netcdf4                   1.6.5                    pypi_0    pypi
networkx                  3.3                      pypi_0    pypi
notebook                  7.1.2                    pypi_0    pypi
notebook-shim             0.2.4                    pypi_0    pypi
numcodecs                 0.12.1                   pypi_0    pypi
numpy                     1.24.4                   pypi_0    pypi
numpydoc                  1.7.0                    pypi_0    pypi
opencv-python-headless    4.9.0.80                 pypi_0    pypi
openssl                   3.0.13               h7f8727e_0
overrides                 7.7.0                    pypi_0    pypi
packaging                 24.0                     pypi_0    pypi
pandas                    2.2.2                    pypi_0    pypi
pandocfilters             1.5.1                    pypi_0    pypi
parsley                   1.3                      pypi_0    pypi
parso                     0.8.4                    pypi_0    pypi
partd                     1.4.1                    pypi_0    pypi
pexpect                   4.9.0                    pypi_0    pypi
photutils                 1.11.0                   pypi_0    pypi
pillow                    10.3.0                   pypi_0    pypi
pip                       23.3.1          py310h06a4308_0
platformdirs              4.2.0                    pypi_0    pypi
pluggy                    1.4.0                    pypi_0    pypi
poppy                     1.1.1                    pypi_0    pypi
prometheus-client         0.20.0                   pypi_0    pypi
prompt-toolkit            3.0.43                   pypi_0    pypi
psutil                    5.9.8                    pypi_0    pypi
ptyprocess                0.7.0                    pypi_0    pypi
pure-eval                 0.2.2                    pypi_0    pypi
pycparser                 2.22                     pypi_0    pypi
pyerfa                    2.0.1.3                  pypi_0    pypi
pygments                  2.17.2                   pypi_0    pypi
pyparsing                 3.1.2                    pypi_0    pypi
pysynphot                 2.0.0                    pypi_0    pypi
pytest                    8.1.1                    pypi_0    pypi
python                    3.10.14              h955ad1f_0
python-dateutil           2.9.0.post0              pypi_0    pypi
python-json-logger        2.0.7                    pypi_0    pypi
pytz                      2024.1                   pypi_0    pypi
pyvo                      1.5.1                    pypi_0    pypi
pyyaml                    6.0.1                    pypi_0    pypi
pyzmq                     25.1.2                   pypi_0    pypi
qtconsole                 5.5.1                    pypi_0    pypi
qtpy                      2.4.1                    pypi_0    pypi
readline                  8.2                  h5eee18b_0
referencing               0.34.0                   pypi_0    pypi
reproject                 0.13.1                   pypi_0    pypi
requests                  2.31.0                   pypi_0    pypi
rfc3339-validator         0.1.4                    pypi_0    pypi
rfc3986-validator         0.1.1                    pypi_0    pypi
rpds-py                   0.18.0                   pypi_0    pypi
scikit-image              0.23.1                   pypi_0    pypi
scipy                     1.13.0                   pypi_0    pypi
secretstorage             3.3.3                    pypi_0    pypi
semantic-version          2.10.0                   pypi_0    pypi
send2trash                1.8.3                    pypi_0    pypi
setuptools                68.2.2          py310h06a4308_0
setuptools-scm            8.0.4                    pypi_0    pypi
six                       1.16.0                   pypi_0    pypi
sniffio                   1.3.1                    pypi_0    pypi
snowballstemmer           2.2.0                    pypi_0    pypi
soupsieve                 2.5                      pypi_0    pypi
spherical-geometry        1.3.1                    pypi_0    pypi
sphinx                    7.2.6                    pypi_0    pypi
sphinxcontrib-applehelp   1.0.8                    pypi_0    pypi
sphinxcontrib-devhelp     1.0.6                    pypi_0    pypi
sphinxcontrib-htmlhelp    2.0.5                    pypi_0    pypi
sphinxcontrib-jsmath      1.0.1                    pypi_0    pypi
sphinxcontrib-qthelp      1.0.7                    pypi_0    pypi
sphinxcontrib-serializinghtml 1.1.10                   pypi_0    pypi
sqlite                    3.41.2               h5eee18b_0
stack-data                0.6.3                    pypi_0    pypi
stcal                     1.7.0                    pypi_0    pypi
stdatamodels              1.10.1                   pypi_0    pypi
stpipe                    0.5.2                    pypi_0    pypi
stsci-image               2.3.5                    pypi_0    pypi
stsci-imagestats          1.6.3                    pypi_0    pypi
stsci-stimage             0.2.6                    pypi_0    pypi
svo-filters               0.4.4                    pypi_0    pypi
synphot                   1.3.post0                pypi_0    pypi
tabulate                  0.9.0                    pypi_0    pypi
terminado                 0.18.1                   pypi_0    pypi
tifffile                  2024.2.12                pypi_0    pypi
tinycss2                  1.2.1                    pypi_0    pypi
tk                        8.6.12               h1ccaba5_0
tomli                     2.0.1                    pypi_0    pypi
toolz                     0.12.1                   pypi_0    pypi
tornado                   6.4                      pypi_0    pypi
tqdm                      4.66.2                   pypi_0    pypi
traitlets                 5.14.2                   pypi_0    pypi
tweakwcs                  0.8.7                    pypi_0    pypi
types-python-dateutil     2.9.0.20240316           pypi_0    pypi
typing-extensions         4.11.0                   pypi_0    pypi
tzdata                    2024.1                   pypi_0    pypi
uncertainties             3.1.7                    pypi_0    pypi
uri-template              1.3.0                    pypi_0    pypi
urllib3                   2.2.1                    pypi_0    pypi
wcwidth                   0.2.13                   pypi_0    pypi
webcolors                 1.13                     pypi_0    pypi
webencodings              0.5.1                    pypi_0    pypi
websocket-client          1.7.0                    pypi_0    pypi
wheel                     0.41.2          py310h06a4308_0
widgetsnbextension        4.0.10                   pypi_0    pypi
wiimatch                  0.3.2                    pypi_0    pypi
xarray                    2024.3.0                 pypi_0    pypi
xz                        5.4.6                h5eee18b_0
zarr                      2.17.2                   pypi_0    pypi
zipp                      3.18.1                   pypi_0    pypi
zlib                      1.2.13               h5eee18b_0

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

[jbrande]

Still not sure why two of the scans in this dataset I'm working on are missing a read, but maybe we should check for consistent numbers of reads and then come up with some more informative failure mode that informs the user about the problem. Attempting to continue with the analysis gives useless results due to the inability of Eureka! to correct for "spectral drift" when a significant chunk of the scan is just missing.

[jbrande]

As it happens, this is sometimes done for scheduling reasons (like in this case), so we're not actually guaranteed to have spatial scans of the same size. Those scans are still good data, so there should be some way to recover them, I guess. Can we anchor the drift calculation to a particular common start point?

e.g. measure the drift relative just to the green line (subpixel drift) as opposed to the red rectangle (tens of pixels)