search

kevin218 / Eureka

Eureka! is a data reduction and analysis pipeline intended for time-series observations with JWST.
https://eurekadocs.readthedocs.io/
MIT License
56 stars 43 forks source link

HST WFC3 Stage3 ValueError: setting an array element with a sequence. #660

Closed andreok closed 2 weeks ago

andreok commented 3 weeks ago

Instrument

Other (any stage)

What happened?

Stage 3 reduction fails after plotting the drifts along each axis. I'm processing HST WFC3 G141 observations. Stage 3 runs for each IMA file of the visit, but fails at the step that aggregates them into a light curve. I get an error "ValueError: setting an array element with a sequence". Couldn't figure out where the issue is.

Error traceback output

ValueError                                Traceback (most recent call last)

[<ipython-input-36-404241701919>](https://localhost:8080/#) in <cell line: 1>()
----> 1 s3_spec, s3_meta = eureka.S3_data_reduction.s3_reduce.reduce(eventlabel)

1 frames

[/usr/local/lib/python3.10/dist-packages/eureka/S3_data_reduction/s3_reduce.py](https://localhost:8080/#) in reduce(eventlabel, ecf_path, s2_meta, input_meta)
    753                 # WFC3 needs a conclusion step to convert lists into
    754                 # arrays before saving
--> 755                 spec, meta, log = inst.conclusion_step(spec, meta, log)
    756 
    757             # Save Dataset object containing time-series of 1D spectra

[/usr/local/lib/python3.10/dist-packages/eureka/S3_data_reduction/wfc3.py](https://localhost:8080/#) in conclusion_step(data, meta, log)
    160         The updated log.
    161     """
--> 162     meta.centroids = np.array(meta.centroids)
    163     meta.guess = np.array(meta.guess)
    164     meta.subdata_ref = np.array(meta.subdata_ref)

ValueError: setting an array element with a sequence. The requested array has an inhomogeneous shape after 1 dimensions. The detected shape was (56,) + inhomogeneous part.

What operating system are you using?

Google Colab

What version of Python are you running?

Python 3.10.12

What Python packages do you have installed?

Package                          Version
-------------------------------- ---------------------
absl-py                          1.4.0
aiohttp                          3.9.5
aiosignal                        1.3.1
alabaster                        0.7.16
albumentations                   1.3.1
altair                           4.2.2
annotated-types                  0.7.0
anyio                            3.7.1
argon2-cffi                      23.1.0
argon2-cffi-bindings             21.2.0
array_record                     0.5.1
arviz                            0.15.1
asciitree                        0.3.3
asdf                             3.2.0
asdf-astropy                     0.6.1
asdf_coordinates_schemas         0.3.0
asdf_standard                    1.1.1
asdf_transform_schemas           0.5.0
asteval                          0.9.33
Astraeus                         0.3
astropy                          5.3.4
astropy_healpix                  1.0.3
astroquery                       0.4.7
astroscrappy                     1.2.0
astunparse                       1.6.3
async-timeout                    4.0.3
atpublic                         4.1.0
attrs                            23.2.0
audioread                        3.0.1
autograd                         1.6.2
Babel                            2.15.0
backcall                         0.2.0
backports.tarfile                1.2.0
batman-package                   2.4.9
beautifulsoup4                   4.12.3
bidict                           0.23.1
bigframes                        1.8.0
bleach                           6.1.0
blinker                          1.4
blis                             0.7.11
blosc2                           2.0.0
bokeh                            2.4.3
Bottleneck                       1.3.8
bqplot                           0.12.43
branca                           0.7.2
build                            1.2.1
CacheControl                     0.14.0
cachetools                       5.3.3
catalogue                        2.0.10
ccdproc                          2.4.2
celerite2                        0.3.1
certifi                          2024.6.2
cffi                             1.16.0
cftime                           1.6.4
chardet                          5.2.0
charset-normalizer               3.3.2
chex                             0.1.86
click                            8.1.7
click-plugins                    1.1.1
cligj                            0.7.2
cloudpathlib                     0.16.0
cloudpickle                      2.2.1
cmake                            3.27.9
cmdstanpy                        1.2.3
colorcet                         3.1.0
colorlover                       0.3.0
colour                           0.1.5
community                        1.0.0b1
confection                       0.1.5
cons                             0.4.6
contextlib2                      21.6.0
contourpy                        1.2.1
corner                           2.2.2
crds                             11.17.23
cryptography                     42.0.7
cuda-python                      12.2.1
cudf-cu12                        24.4.1
cufflinks                        0.17.3
cupy-cuda12x                     12.2.0
cvxopt                           1.3.2
cvxpy                            1.3.4
cycler                           0.12.1
cymem                            2.0.8
Cython                           3.0.10
dask                             2023.8.1
datascience                      0.17.6
db-dtypes                        1.2.0
dbus-python                      1.2.18
debugpy                          1.6.6
decorator                        4.4.2
defusedxml                       0.7.1
dill                             0.3.8
distributed                      2023.8.1
distro                           1.7.0
dlib                             19.24.4
dm-tree                          0.1.8
docstring_parser                 0.16
docutils                         0.18.1
dopamine_rl                      4.0.9
duckdb                           0.10.3
dynesty                          2.1.3
earthengine-api                  0.1.405
easydict                         1.13
ecos                             2.0.13
editdistance                     0.6.2
eerepr                           0.0.4
emcee                            3.1.6
en-core-web-sm                   3.7.1
entrypoints                      0.4
et-xmlfile                       1.1.0
etils                            1.7.0
etuples                          0.3.9
eureka                           0.11.dev292+ga344dbed
exceptiongroup                   1.2.1
exotic-ld                        3.0.0
fastai                           2.7.15
fastcore                         1.5.43
fastdownload                     0.0.7
fasteners                        0.19
fastjsonschema                   2.19.1
fastprogress                     1.0.3
fastrlock                        0.8.2
filelock                         3.14.0
fiona                            1.9.6
firebase-admin                   5.3.0
Flask                            2.2.5
flatbuffers                      24.3.25
flax                             0.8.4
folium                           0.14.0
fonttools                        4.53.0
frozendict                       2.4.4
frozenlist                       1.4.1
fsspec                           2023.6.0
future                           0.18.3
gast                             0.5.4
gcsfs                            2023.6.0
GDAL                             3.6.4
gdown                            5.1.0
geemap                           0.32.1
gensim                           4.3.2
geocoder                         1.38.1
geographiclib                    2.0
geopandas                        0.13.2
geopy                            2.3.0
george                           0.4.1
gin-config                       0.5.0
glob2                            0.7
google                           2.0.3
google-ai-generativelanguage     0.6.4
google-api-core                  2.11.1
google-api-python-client         2.84.0
google-auth                      2.27.0
google-auth-httplib2             0.1.1
google-auth-oauthlib             1.2.0
google-cloud-aiplatform          1.52.0
google-cloud-bigquery            3.21.0
google-cloud-bigquery-connection 1.12.1
google-cloud-bigquery-storage    2.25.0
google-cloud-core                2.3.3
google-cloud-datastore           2.15.2
google-cloud-firestore           2.11.1
google-cloud-functions           1.13.3
google-cloud-iam                 2.15.0
google-cloud-language            2.13.3
google-cloud-resource-manager    1.12.3
google-cloud-storage             2.8.0
google-cloud-translate           3.11.3
google-colab                     1.0.0
google-crc32c                    1.5.0
google-generativeai              0.5.4
google-pasta                     0.2.0
google-resumable-media           2.7.0
googleapis-common-protos         1.63.1
googledrivedownloader            0.4
graphviz                         0.20.3
greenlet                         3.0.3
grpc-google-iam-v1               0.13.0
grpcio                           1.64.1
grpcio-status                    1.48.2
gspread                          6.0.2
gspread-dataframe                3.3.1
gym                              0.25.2
gym-notices                      0.0.8
h5netcdf                         1.3.0
h5py                             3.9.0
holidays                         0.50
holoviews                        1.17.1
html5lib                         1.1
httpimport                       1.3.1
httplib2                         0.22.0
huggingface-hub                  0.23.2
humanize                         4.7.0
hyperopt                         0.2.7
ibis-framework                   8.0.0
idna                             3.7
image_registration               0.2.10.dev1+g8a3a723
imageio                          2.31.6
imageio-ffmpeg                   0.5.1
imagesize                        1.4.1
imbalanced-learn                 0.10.1
imgaug                           0.4.0
immutabledict                    4.2.0
importlib_metadata               7.1.0
importlib_resources              6.4.0
imutils                          0.5.4
inflect                          7.0.0
iniconfig                        2.0.0
intel-openmp                     2023.2.4
ipyevents                        2.0.2
ipyfilechooser                   0.6.0
ipykernel                        5.5.6
ipyleaflet                       0.18.2
ipython                          7.34.0
ipython-genutils                 0.2.0
ipython-sql                      0.5.0
ipytree                          0.2.2
ipywidgets                       7.7.1
itsdangerous                     2.2.0
jaraco.classes                   3.4.0
jaraco.context                   5.3.0
jax                              0.4.26
jaxlib                           0.4.26+cuda12.cudnn89
jedi                             0.19.1
jeepney                          0.7.1
jellyfish                        1.0.4
jieba                            0.42.1
Jinja2                           3.1.4
jmespath                         1.0.1
joblib                           1.4.2
jsonpickle                       3.0.4
jsonschema                       4.19.2
jsonschema-specifications        2023.12.1
jupyter-client                   6.1.12
jupyter-console                  6.1.0
jupyter_core                     5.7.2
jupyter-server                   1.24.0
jupyterlab_pygments              0.3.0
jupyterlab_widgets               3.0.11
kaggle                           1.6.14
kagglehub                        0.2.5
keras                            2.15.0
keyring                          23.5.0
keyrings.alt                     5.0.1
kiwisolver                       1.4.5
langcodes                        3.4.0
language_data                    1.2.0
launchpadlib                     1.10.16
lazr.restfulclient               0.14.4
lazr.uri                         1.0.6
lazy_loader                      0.4
libclang                         18.1.1
librosa                          0.10.2.post1
lightgbm                         4.1.0
linkify-it-py                    2.0.3
llvmlite                         0.41.1
lmfit                            1.3.1
locket                           1.0.0
logical-unification              0.4.6
lxml                             4.9.4
malloy                           2023.1067
marisa-trie                      1.1.1
Markdown                         3.6
markdown-it-py                   3.0.0
MarkupSafe                       2.1.5
matplotlib                       3.7.1
matplotlib-inline                0.1.7
matplotlib-venn                  0.11.10
mc3                              3.1.5
mdit-py-plugins                  0.4.1
mdurl                            0.1.2
miniKanren                       1.0.3
missingno                        0.5.2
mistune                          0.8.4
mizani                           0.9.3
mkl                              2023.2.0
ml-dtypes                        0.2.0
mlxtend                          0.22.0
more-itertools                   10.1.0
moviepy                          1.0.3
mpmath                           1.3.0
msgpack                          1.0.8
multidict                        6.0.5
multipledispatch                 1.0.0
multitasking                     0.0.11
murmurhash                       1.0.10
music21                          9.1.0
natsort                          8.4.0
nbclassic                        1.1.0
nbclient                         0.10.0
nbconvert                        6.5.4
nbformat                         5.10.4
nest-asyncio                     1.6.0
netCDF4                          1.6.5
networkx                         3.3
nibabel                          4.0.2
nltk                             3.8.1
notebook                         6.5.5
notebook_shim                    0.2.4
numba                            0.58.1
numcodecs                        0.12.1
numexpr                          2.10.0
numpy                            1.24.4
numpydoc                         1.7.0
nvtx                             0.2.10
oauth2client                     4.1.3
oauthlib                         3.2.2
opencv-contrib-python            4.8.0.76
opencv-python                    4.8.0.76
opencv-python-headless           4.10.0.82
openpyxl                         3.1.3
opt-einsum                       3.3.0
optax                            0.2.2
orbax-checkpoint                 0.4.4
osqp                             0.6.2.post8
packaging                        24.0
pandas                           2.0.3
pandas-datareader                0.10.0
pandas-gbq                       0.19.2
pandas-stubs                     2.0.3.230814
pandocfilters                    1.5.1
panel                            1.3.8
param                            2.1.0
Parsley                          1.3
parso                            0.8.4
parsy                            2.1
partd                            1.4.2
pathlib                          1.0.1
patsy                            0.5.6
peewee                           3.17.5
pexpect                          4.9.0
photutils                        1.12.0
pickleshare                      0.7.5
Pillow                           9.4.0
pip                              23.1.2
pip-tools                        6.13.0
platformdirs                     4.2.2
plotly                           5.15.0
plotnine                         0.12.4
pluggy                           1.5.0
polars                           0.20.2
pooch                            1.8.1
portpicker                       1.5.2
prefetch-generator               1.0.3
preshed                          3.0.9
prettytable                      3.10.0
proglog                          0.1.10
progressbar2                     4.2.0
prometheus_client                0.20.0
promise                          2.3
prompt_toolkit                   3.0.45
prophet                          1.1.5
proto-plus                       1.23.0
protobuf                         3.20.3
psutil                           5.9.5
psycopg2                         2.9.9
ptyprocess                       0.7.0
py-cpuinfo                       9.0.0
py4j                             0.10.9.7
pyarrow                          14.0.2
pyarrow-hotfix                   0.6
pyasn1                           0.6.0
pyasn1_modules                   0.4.0
pycocotools                      2.0.7
pycparser                        2.22
pydantic                         2.7.3
pydantic_core                    2.18.4
pydata-google-auth               1.8.2
pydot                            1.4.2
pydot-ng                         2.0.0
pydotplus                        2.0.2
PyDrive                          1.3.1
PyDrive2                         1.6.3
pyerfa                           2.0.1.4
pygame                           2.5.2
Pygments                         2.16.1
PyGObject                        3.42.1
PyJWT                            2.3.0
pymc                             5.10.4
pymystem3                        0.2.0
pynvjitlink-cu12                 0.2.3
PyOpenGL                         3.1.7
pyOpenSSL                        24.1.0
pyparsing                        3.1.2
pyperclip                        1.8.2
pyproj                           3.6.1
pyproject_hooks                  1.1.0
pyshp                            2.3.1
PySocks                          1.7.1
pysynphot                        2.0.0
pytensor                         2.18.6
pytest                           7.4.4
python-apt                       0.0.0
python-box                       7.1.1
python-dateutil                  2.8.2
python-louvain                   0.16
python-slugify                   8.0.4
python-utils                     3.8.2
pytz                             2023.4
pyviz_comms                      3.0.2
pyvo                             1.5.2
PyWavelets                       1.6.0
PyYAML                           6.0.1
pyzmq                            24.0.1
qdldl                            0.1.7.post2
qudida                           0.0.4
ratelim                          0.1.6
referencing                      0.35.1
regex                            2024.5.15
reproject                        0.13.1
requests                         2.31.0
requests-oauthlib                1.3.1
requirements-parser              0.9.0
rich                             13.7.1
rmm-cu12                         24.4.0
rpds-py                          0.18.1
rpy2                             3.4.2
rsa                              4.9
safetensors                      0.4.3
scikit-image                     0.19.3
scikit-learn                     1.2.2
scipy                            1.11.4
scooby                           0.10.0
scs                              3.2.4.post2
seaborn                          0.13.1
SecretStorage                    3.3.1
semantic-version                 2.10.0
Send2Trash                       1.8.3
sentencepiece                    0.1.99
setuptools                       67.7.2
setuptools-scm                   8.1.0
shapely                          2.0.4
simple_parsing                   0.1.5
six                              1.16.0
sklearn-pandas                   2.2.0
smart-open                       6.4.0
sniffio                          1.3.1
snowballstemmer                  2.2.0
sortedcontainers                 2.4.0
soundfile                        0.12.1
soupsieve                        2.5
soxr                             0.3.7
spacy                            3.7.4
spacy-legacy                     3.0.12
spacy-loggers                    1.0.5
Sphinx                           7.3.7
sphinxcontrib-applehelp          1.0.8
sphinxcontrib-devhelp            1.0.6
sphinxcontrib-htmlhelp           2.0.5
sphinxcontrib-jsmath             1.0.1
sphinxcontrib-qthelp             1.0.7
sphinxcontrib-serializinghtml    1.1.10
SQLAlchemy                       2.0.30
sqlglot                          20.11.0
sqlparse                         0.5.0
srsly                            2.4.8
stanio                           0.5.0
statsmodels                      0.14.2
stdatamodels                     1.10.1
StrEnum                          0.4.15
svo-filters                      0.4.4
sympy                            1.12.1
tables                           3.8.0
tabulate                         0.9.0
tbb                              2021.12.0
tblib                            3.0.0
tenacity                         8.3.0
tensorboard                      2.15.2
tensorboard-data-server          0.7.2
tensorflow                       2.15.0
tensorflow-datasets              4.9.5
tensorflow-estimator             2.15.0
tensorflow-gcs-config            2.15.0
tensorflow-hub                   0.16.1
tensorflow-io-gcs-filesystem     0.37.0
tensorflow-metadata              1.15.0
tensorflow-probability           0.23.0
tensorstore                      0.1.45
termcolor                        2.4.0
terminado                        0.18.1
text-unidecode                   1.3
textblob                         0.17.1
tf_keras                         2.15.1
tf-slim                          1.1.0
thinc                            8.2.3
threadpoolctl                    3.5.0
tifffile                         2024.5.22
tinycss2                         1.3.0
tokenizers                       0.19.1
toml                             0.10.2
tomli                            2.0.1
toolz                            0.12.1
torch                            2.3.0+cu121
torchaudio                       2.3.0+cu121
torchsummary                     1.5.1
torchtext                        0.18.0
torchvision                      0.18.0+cu121
tornado                          6.3.3
tqdm                             4.66.4
traitlets                        5.7.1
traittypes                       0.2.1
transformers                     4.41.2
triton                           2.3.0
tweepy                           4.14.0
typer                            0.9.4
types-pytz                       2024.1.0.20240417
types-setuptools                 70.0.0.20240524
typing_extensions                4.12.1
tzdata                           2024.1
tzlocal                          5.2
uc-micro-py                      1.0.3
uncertainties                    3.2.1
uritemplate                      4.1.1
urllib3                          2.0.7
vega-datasets                    0.9.0
wadllib                          1.3.6
wasabi                           1.1.3
wcwidth                          0.2.13
weasel                           0.3.4
webcolors                        1.13
webencodings                     0.5.1
websocket-client                 1.8.0
Werkzeug                         3.0.3
wheel                            0.43.0
widgetsnbextension               3.6.6
wordcloud                        1.9.3
wrapt                            1.14.1
xarray                           2023.7.0
xarray-einstats                  0.7.0
xgboost                          2.0.3
xlrd                             2.0.1
xyzservices                      2024.4.0
yarl                             1.9.4
yellowbrick                      1.5
yfinance                         0.2.40
zarr                             2.18.2
zict                             3.0.0
zipp                             3.19.1

Code of Conduct

taylorbell57 commented 3 weeks ago

@jbrande (who is part of our development team) recently encountered this issue as well which he mentioned in #649. Since this appears to be an exact duplicate of that issue, I'm going to close this one, but I recommend you click the "Subscribe" button on Yoni's issue to get updates as he tries to resolve the bug.