search

LSSTDESC / parsnip

Deep generative modeling of astronomical transient light curves
MIT License
7 stars 10 forks source link

IndexError when trying to prprocess data #18

Open Vernonicuhhh opened 8 months ago

Vernonicuhhh commented 8 months ago

Hi, I have been running into the following issue when trying to predict parsnip encoding for any given dataset. Below is an example using the ps1 dataset downloaded using lcdata

import lcdata

model = parsnip.load_model('ps1')
data = parsnip.load_dataset('/global/homes/v/vdiaz/data/ps1.h5')

result = model.preprocess(data)

when running this, I then get the following output:

"""
Traceback (most recent call last):
  File "/global/common/software/nersc/pe/conda-envs/23.10.0/python-3.11/nersc-python/lib/python3.11/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
                    ^^^^^^^^^^^^^^^^^^^
  File "/global/common/software/nersc/pe/conda-envs/23.10.0/python-3.11/nersc-python/lib/python3.11/multiprocessing/pool.py", line 48, in mapstar
    return list(map(*args))
           ^^^^^^^^^^^^^^^^
  File "/global/homes/v/vdiaz/.local/perlmutter/python-3.11/lib/python3.11/site-packages/parsnip/light_curve.py", line 167, in preprocess_light_curve
    reference_time = _determine_time_grid(light_curve)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/global/homes/v/vdiaz/.local/perlmutter/python-3.11/lib/python3.11/site-packages/parsnip/light_curve.py", line 32, in _determine_time_grid
    guess_offset = mode[0] - 0.05
                   ~~~~^^^
IndexError: invalid index to scalar variable.
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/global/u2/v/vdiaz/test.py", line 7, in <module>
    result = model.preprocess(data)
             ^^^^^^^^^^^^^^^^^^^^^^
  File "/global/homes/v/vdiaz/.local/perlmutter/python-3.11/lib/python3.11/site-packages/parsnip/parsnip.py", line 421, in preprocess
    preprocessed_light_curves = list(iterator)
                                ^^^^^^^^^^^^^^
  File "/global/common/software/nersc/pe/conda-envs/23.10.0/python-3.11/nersc-python/lib/python3.11/site-packages/tqdm/std.py", line 1182, in __iter__
    for obj in iterable:
  File "/global/common/software/nersc/pe/conda-envs/23.10.0/python-3.11/nersc-python/lib/python3.11/multiprocessing/pool.py", line 423, in <genexpr>
    return (item for chunk in result for item in chunk)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/global/common/software/nersc/pe/conda-envs/23.10.0/python-3.11/nersc-python/lib/python3.11/multiprocessing/pool.py", line 873, in next
    raise value
  File "/global/common/software/nersc/pe/conda-envs/23.10.0/python-3.11/nersc-python/lib/python3.11/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
^^^^^^^^^^^^^^^
  File "/global/common/software/nersc/pe/conda-envs/23.10.0/python-3.11/nersc-python/lib/python3.11/multiprocessing/pool.py", line 48, in mapstar
    return list(map(*args))
      ^^^^^^^^^^^^^^^^^
  File "/global/homes/v/vdiaz/.local/perlmutter/python-3.11/lib/python3.11/site-packages/parsnip/light_curve.py", line 167, in preprocess_light_curve
    reference_time = _determine_time_grid(light_curve)
      ^^^^^^^^^^^^^^^^^
  File "/global/homes/v/vdiaz/.local/perlmutter/python-3.11/lib/python3.11/site-packages/parsnip/light_curve.py", line 32, in _determine_time_grid
    guess_offset = mode[0] - 0.05
      ^^^^^^^^^^^^^^^^^
IndexError: invalid index to scalar variable.

this happens whenever I try to predict encodings for any dataset. I am using python 3.11.6 and have the following packages installed:

argon2-cffi                   23.1.0
argon2-cffi-bindings          21.2.0
arrow                         1.3.0
astro-parsnip                 1.4.1
astropy                       5.3.4
asttokens                     2.4.0
async-lru                     2.0.4
attrs                         23.1.0
Authlib                       1.2.1
Babel                         2.12.1
backcall                      0.2.0
backports.functools-lru-cache 1.6.5
beautifulsoup4                4.12.2
bleach                        6.0.0
blosc2                        2.3.2
bokeh                         3.2.2
Brotli                        1.0.9
cached-property               1.5.2
Cartopy                       0.22.0
certifi                       2023.7.22
cffi                          1.16.0
cftime                        1.6.2
charset-normalizer            3.3.0
click                         8.1.7
cloudpickle                   2.2.1
colorama                      0.4.6
comm                          0.1.4
contourpy                     1.1.1
cryptography                  41.0.4
cupy-cuda11x                  12.2.0
cycler                        0.12.0
Cython                        0.29.36
cytoolz                       0.12.2
dask                          2023.9.3
dask-mpi                      2022.4.0
debugpy                       1.8.0
decorator                     5.1.1
defusedxml                    0.7.1
distributed                   2023.9.3
dnspython                     2.4.2
entrypoints                   0.4
exceptiongroup                1.1.3
executing                     1.2.0
extinction                    0.4.6
fastjsonschema                2.18.1
fastrlock                     0.8.2
filelock                      3.13.1
fonttools                     4.43.1
fqdn                          1.5.1
fsspec                        2023.9.2
gmpy2                         2.1.2
h11                           0.14.0
h2                            4.1.0
h5py                          3.9.0
hpack                         4.0.0
httpcore                      1.0.0
hyperframe                    6.0.1
idna                          3.4
imagecodecs                   2023.8.12
imageio                       2.31.1
importlib-metadata            6.8.0
importlib-resources           6.1.0
ipykernel                     6.25.2
ipympl                        0.9.3
ipython                       8.16.1
ipython-genutils              0.2.0
ipywidgets                    8.1.1
isoduration                   20.11.0
jedi                          0.19.1
Jinja2                        3.1.2
joblib                        1.3.2
json5                         0.9.14
jsonpointer                   2.4
jsonschema                    4.19.1
jsonschema-specifications     2023.7.1
jupyter_client                8.3.1
jupyter_core                  5.3.2
jupyter-events                0.7.0
jupyter-lsp                   2.2.0
jupyter_server                2.7.3
jupyter_server_terminals      0.4.4
jupyterlab                    4.0.6
jupyterlab-pygments           0.2.2
jupyterlab_server             2.25.0
jupyterlab-widgets            3.0.9
kiwisolver                    1.4.5
lazy_loader                   0.3
lcdata                        1.1.2
lightgbm                      4.2.0
llvmlite                      0.40.1
locket                        1.0.0
lz4                           4.3.2
MarkupSafe                    2.1.3
matplotlib                    3.8.0
matplotlib-inline             0.1.6
mistune                       3.0.1
mpi4py                        3.1.5
mpmath                        1.3.0
msgpack                       1.0.6
munkres                       1.1.4
mysql-connector-python        8.0.31
nbclient                      0.8.0
nbconvert                     7.9.2
nbformat                      5.9.2
ndindex                       1.7
nest-asyncio                  1.5.6
netCDF4                       1.6.4
networkx                      3.1
notebook_shim                 0.2.3
numba                         0.57.1
numexpr                       2.8.8
numpy                         1.24.4
nvidia-cublas-cu12            12.1.3.1
nvidia-cuda-cupti-cu12        12.1.105
nvidia-cuda-nvrtc-cu12        12.1.105
nvidia-cuda-runtime-cu12      12.1.105
nvidia-cudnn-cu12             8.9.2.26
nvidia-cufft-cu12             11.0.2.54
nvidia-curand-cu12            10.3.2.106
nvidia-cusolver-cu12          11.4.5.107
nvidia-cusparse-cu12          12.1.0.106
nvidia-nccl-cu12              2.18.1
nvidia-nvjitlink-cu12         12.3.101
nvidia-nvtx-cu12              12.1.105
overrides                     7.4.0
packaging                     23.2
pandas                        2.1.1
pandocfilters                 1.5.0
parso                         0.8.3
partd                         1.4.1
patsy                         0.5.3
pexpect                       4.8.0
pickleshare                   0.7.5
Pillow                        10.0.0
pip                           23.2.1
pkgutil_resolve_name          1.3.10
platformdirs                  3.11.0
ply                           3.11
prometheus-client             0.17.1
prompt-toolkit                3.0.39
protobuf                      4.21.12
psutil                        5.9.5
psycopg2                      2.9.7
ptyprocess                    0.7.0
pure-eval                     0.2.2
py-cpuinfo                    9.0.0
pyarrow                       12.0.1
pycparser                     2.21
pyerfa                        2.0.0.3
Pygments                      2.16.1
pyparsing                     3.1.1
pyproj                        3.6.1
PyQt5                         5.15.9
PyQt5-sip                     12.12.2
pyshp                         2.3.1
PySocks                       1.7.1
python-dateutil               2.8.2
python-json-logger            2.0.7
pytz                          2023.3.post1
PyWavelets                    1.4.1
PyYAML                        6.0.1
pyzmq                         25.1.1
referencing                   0.30.2
requests                      2.31.0
rfc3339-validator             0.1.4
rfc3986-validator             0.1.1
rpds-py                       0.10.4
scikit-image                  0.21.0
scikit-learn                  1.3.1
scipy                         1.11.3
seaborn                       0.13.0
Send2Trash                    1.8.2
setuptools                    68.2.2
shapely                       2.0.1
sip                           6.7.11
six                           1.16.0
sncosmo                       2.10.2
sniffio                       1.3.0
sortedcontainers              2.4.0
soupsieve                     2.5
stack-data                    0.6.2
statsmodels                   0.14.0
sympy                         1.12
tables                        3.9.2
tabulate                      0.9.0
tblib                         2.0.0
terminado                     0.17.1
threadpoolctl                 3.2.0
tifffile                      2023.9.26
tinycss2                      1.2.1
toml                          0.10.2
tomli                         2.0.1
toolz                         0.12.0
torch                         2.1.2
tornado                       6.3.3
tqdm                          4.66.1
traitlets                     5.11.2
triton                        2.1.0
types-python-dateutil         2.8.19.14
typing_extensions             4.8.0
typing-utils                  0.1.0
tzdata                        2023.3
uri-template                  1.3.0
urllib3                       2.0.6
wcwidth                       0.2.8
webcolors                     1.13
webencodings                  0.5.1
websocket-client              1.6.3
wheel                         0.41.2
widgetsnbextension            4.0.9
xyzservices                   2023.10.0
zict                          3.0.0
zipp                          3.17.0

Any guidance on how to proceed would be greatly appreciated, thanks!

hombit commented 8 months ago

Thank you, @Vernonicuhhh, for the report. I cannot investigate it right now, but I guess that something is wrong with the dataset. Have you tried another dataset?

Vernonicuhhh commented 8 months ago

Thank you so much for the response! Yes I have tried it with different datasets and have run into the same problem. When installing parsnip I had installed it in a new virtual environment with no other packages installed prior.