search

talmolab / sleap

A deep learning framework for multi-animal pose tracking.
https://sleap.ai
Other
435 stars 97 forks source link

Error when evaluating centroid model #1616

Closed ZVWan closed 11 months ago

ZVWan commented 11 months ago

Bug description

Hi! I'm currently training the SLEAP model for tracking insect activity with only one bodypart. The training went well and prediction can be saved. But when the model evaluation started, the process terminated with an warning of 'WARNING:sleap.nn.evals:Failed to compute metrics'. And no evaluation result was saved.

Expected behaviour

There should be a file with evaluation result.

Actual behaviour

INFO:sleap.nn.training:Finished training loop. [15.2 min] INFO:sleap.nn.training:Deleting visualization directory: models/personal.centroid_1/viz INFO:sleap.nn.training:Saving evaluation metrics to model folder... Predicting... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% ETA: 0:00:00 8.3 FPS /mnt/DATA2/envs/sleap/sleap/nn/evals.py:539: RuntimeWarning: Mean of empty slice "dist.avg": np.nanmean(dists), /mnt/DATA2/envs/sleap/sleap/nn/evals.py:572: RuntimeWarning: Mean of empty slice. mPCK = mPCK_parts.mean() /home/anaconda3/envs/sleap/lib/python3.7/site-packages/numpy/core/_methods.py:189: RuntimeWarning: invalid value encountered in double_scalars ret = ret.dtype.type(ret / rcount) /mnt/DATA2/envs/sleap/sleap/nn/evals.py:666: RuntimeWarning: Mean of empty slice. pair_pck = metrics["pck.pcks"].mean(axis=-1).mean(axis=-1) /home/anaconda3/envs/sleap/lib/python3.7/site-packages/numpy/core/_methods.py:182: RuntimeWarning: invalid value encountered in true_divide ret, rcount, out=ret, casting='unsafe', subok=False) /mnt/DATA2/envs/sleap/sleap/nn/evals.py:668: RuntimeWarning: Mean of empty slice. metrics["oks.mOKS"] = pair_oks.mean() WARNING:sleap.nn.evals:Failed to compute metrics. INFO:sleap.nn.evals:Saved predictions: models/personal.centroid_1/labels_pr.train.slp Predicting... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% ETA: 0:00:00 ? /mnt/DATA2/envs/sleap/sleap/nn/evals.py:539: RuntimeWarning: Mean of empty slice "dist.avg": np.nanmean(dists), /mnt/DATA2/envs/sleap/sleap/nn/evals.py:572: RuntimeWarning: Mean of empty slice. mPCK = mPCK_parts.mean() /home/anaconda3/envs/sleap/lib/python3.7/site-packages/numpy/core/_methods.py:189: RuntimeWarning: invalid value encountered in double_scalars ret = ret.dtype.type(ret / rcount) /mnt/DATA2/envs/sleap/sleap/nn/evals.py:666: RuntimeWarning: Mean of empty slice. pair_pck = metrics["pck.pcks"].mean(axis=-1).mean(axis=-1) /home/anaconda3/envs/sleap/lib/python3.7/site-packages/numpy/core/_methods.py:182: RuntimeWarning: invalid value encountered in true_divide ret, rcount, out=ret, casting='unsafe', subok=False) /mnt/DATA2/envs/sleap/sleap/nn/evals.py:668: RuntimeWarning: Mean of empty slice. metrics["oks.mOKS"] = pair_oks.mean() WARNING:sleap.nn.evals:Failed to compute metrics. INFO:sleap.nn.evals:Saved predictions: models/personal.centroid_1/labels_pr.val.slp

Your personal set up

Environment packages absl-py==1.0.0 accessible-pygments==0.0.4 alabaster==0.7.13 anyio==3.7.1 appdirs==1.4.4 argon2-cffi==23.1.0 argon2-cffi-bindings==21.2.0 astunparse==1.6.3 attrs==21.4.0 Babel==2.13.1 backcall==0.2.0 backports.zoneinfo==0.2.1 beautifulsoup4==4.12.2 black==21.6b0 bleach==6.0.0 cached-property==1.5.2 cachetools==4.2.4 cattrs @ file:///home/conda/feedstock_root/build_artifacts/cattrs_1604136207372/work certifi==2023.7.22 cffi==1.15.1 cfgv==3.3.1 charset-normalizer==2.0.9 click==8.0.4 cloudpickle @ file:///home/conda/feedstock_root/build_artifacts/cloudpickle_1674202310934/work colorama==0.4.6 comm==0.1.4 coverage==7.2.7 cryptography==41.0.5 cycler @ file:///home/conda/feedstock_root/build_artifacts/cycler_1635519461629/work cytoolz @ file:///home/conda/feedstock_root/build_artifacts/cytoolz_1657553457169/work dask @ file:///home/conda/feedstock_root/build_artifacts/dask-core_1644602974678/work debugpy==1.7.0 decorator==5.1.1 defusedxml==0.7.1 Deprecated==1.2.14 distlib==0.3.7 docutils==0.17.1 efficientnet==1.0.0 entrypoints==0.4 exceptiongroup==1.1.3 filelock==3.12.2 flatbuffers==2.0 fonttools @ file:///home/conda/feedstock_root/build_artifacts/fonttools_1666389892786/work fsspec @ file:///home/conda/feedstock_root/build_artifacts/fsspec_1674184942191/work gast==0.4.0 gitdb==4.0.11 GitPython==3.1.40 google-auth==2.3.3 google-auth-oauthlib==0.4.6 google-pasta==0.2.0 greenlet==3.0.1 grpcio==1.43.0 h5py==3.1.0 hdmf==3.5.5 identify==2.5.24 idna==3.3 image-classifiers==1.0.0 imagecodecs @ file:///home/conda/feedstock_root/build_artifacts/imagecodecs_1644819473370/work imageio @ file:///home/conda/feedstock_root/build_artifacts/imageio_1696854106455/work imagesize==1.4.1 imgaug @ file:///home/conda/feedstock_root/build_artifacts/imgaug_1640909786103/work imgstore==0.2.9 importlib-metadata==6.7.0 importlib-resources==5.12.0 iniconfig==2.0.0 ipykernel==6.16.2 ipython==7.34.0 ipython-genutils==0.2.0 ipywidgets==7.8.1 jaraco.classes==3.2.3 jedi==0.17.2 jeepney==0.8.0 Jinja2==3.1.2 joblib @ file:///home/conda/feedstock_root/build_artifacts/joblib_1691577114857/work jsmin @ file:///home/conda/feedstock_root/build_artifacts/jsmin_1642532731678/work json5==0.9.14 jsonpickle==1.2 jsonschema==4.17.3 jupyter-cache==0.4.3 jupyter-server==1.13.5 jupyter-server-mathjax==0.2.6 jupyter-sphinx==0.3.2 jupyter_client==7.4.9 jupyter_core==4.12.0 jupyterlab==3.3.4 jupyterlab-pygments==0.2.2 jupyterlab-widgets==1.1.7 jupyterlab_server==2.16.3 keras==2.7.0 Keras-Applications==1.0.8 Keras-Preprocessing==1.1.2 keyring==24.1.1 kiwisolver @ file:///home/conda/feedstock_root/build_artifacts/kiwisolver_1657953088445/work libclang==12.0.0 linkify-it-py==2.0.2 livereload==2.6.3 locket @ file:///home/conda/feedstock_root/build_artifacts/locket_1650660393415/work lxml==4.9.3 Markdown==3.3.6 markdown-it-py==1.1.0 MarkupSafe==2.1.3 matplotlib @ file:///home/conda/feedstock_root/build_artifacts/matplotlib-suite_1661439848456/work matplotlib-inline==0.1.6 mdit-py-plugins==0.2.8 mdurl @ file:///home/conda/feedstock_root/build_artifacts/mdurl_1639515908913/work mistune==0.8.4 more-itertools==9.1.0 munkres==1.1.4 mypy-extensions==1.0.0 myst-nb==0.13.2 myst-parser==0.15.2 nbclassic==0.5.6 nbclient==0.5.13 nbconvert==6.5.4 nbdime==3.2.1 nbformat==5.1.3 ndx-pose==0.1.1 nest-asyncio==1.5.8 networkx @ file:///home/conda/feedstock_root/build_artifacts/networkx_1635253012265/work nixio==1.5.3 nodeenv==1.8.0 notebook==6.5.6 notebook_shim==0.2.3 numpy==1.19.5 oauthlib==3.1.1 opencv-python==4.6.0 opencv-python-headless==4.2.0.34 opt-einsum==3.3.0 packaging==21.3 pandas==1.3.5 pandocfilters==1.5.0 parso==0.7.1 partd @ file:///home/conda/feedstock_root/build_artifacts/partd_1695667515973/work pathspec==0.11.2 patsy @ file:///home/conda/feedstock_root/build_artifacts/patsy_1665356157073/work pexpect==4.8.0 pickleshare==0.7.5 Pillow @ file:///home/conda/feedstock_root/build_artifacts/pillow_1660385854171/work pkginfo==1.9.6 pkgutil_resolve_name==1.3.10 platformdirs==3.11.0 pluggy==1.2.0 pre-commit==2.21.0 prometheus-client==0.17.1 prompt-toolkit==3.0.39 protobuf==3.19.1 psutil @ file:///home/conda/feedstock_root/build_artifacts/psutil_1666155398032/work ptyprocess==0.7.0 pyasn1==0.4.8 pyasn1-modules==0.2.8 pycparser==2.21 pydata-sphinx-theme==0.13.3 PyGithub==2.1.1 Pygments @ file:///home/conda/feedstock_root/build_artifacts/pygments_1691408637400/work PyJWT==2.8.0 pykalman==0.9.5 PyNaCl==1.5.0 pynwb==2.3.3 pyparsing==3.0.6 pyrsistent==0.19.3 PySide2==5.15.6 pytest==7.4.3 pytest-cov==3.0.0 pytest-qt==4.2.0 pytest-xvfb==3.0.0 python-dateutil @ file:///home/conda/feedstock_root/build_artifacts/python-dateutil_1626286286081/work python-rapidjson @ file:///home/conda/feedstock_root/build_artifacts/python-rapidjson_1665999896718/work pytz @ file:///home/conda/feedstock_root/build_artifacts/pytz_1693930252784/work PyVirtualDisplay==3.0 PyWavelets @ file:///home/conda/feedstock_root/build_artifacts/pywavelets_1649616401885/work PyYAML @ file:///home/conda/feedstock_root/build_artifacts/pyyaml_1648757092905/work pyzmq @ file:///home/conda/feedstock_root/build_artifacts/pyzmq_1663830492333/work qimage2ndarray==1.10.0 QtPy @ file:///home/conda/feedstock_root/build_artifacts/qtpy_1698112029416/work readme-renderer==37.3 regex==2023.10.3 requests==2.26.0 requests-oauthlib==1.3.0 requests-toolbelt==1.0.0 rfc3986==2.0.0 rich @ file:///home/conda/feedstock_root/build_artifacts/rich-split_1696096668362/work/dist rsa==4.8 ruamel.yaml==0.18.5 ruamel.yaml.clib==0.2.8 scikit-image @ file:///home/conda/feedstock_root/build_artifacts/scikit-image_1660146497293/work scikit-learn @ file:///home/conda/feedstock_root/build_artifacts/scikit-learn_1632611341839/work scikit-video==1.1.11 scipy @ file:///home/conda/feedstock_root/build_artifacts/scipy_1637806658031/work seaborn @ file:///home/conda/feedstock_root/build_artifacts/seaborn-split_1672497695270/work SecretStorage==3.3.3 segmentation-models==1.0.1 Send2Trash==1.8.2 setuptools-scm==6.3.2 Shapely @ file:///home/conda/feedstock_root/build_artifacts/shapely_1665624546039/work shiboken2==5.15.6 six @ file:///home/conda/feedstock_root/build_artifacts/six_1620240208055/work -e git+https://github.com/talmolab/sleap@cb82d364e95f910e1306483369b8f0ef89998b86#egg=sleap smmap==5.0.1 sniffio==1.3.0 snowballstemmer==2.2.0 soupsieve==2.4.1 Sphinx==4.5.0 sphinx-autobuild==2021.3.14 sphinx-book-theme==1.0.1 sphinx-copybutton==0.5.2 sphinx-togglebutton==0.3.2 sphinxcontrib-applehelp==1.0.2 sphinxcontrib-devhelp==1.0.2 sphinxcontrib-htmlhelp==2.0.0 sphinxcontrib-jsmath==1.0.1 sphinxcontrib-qthelp==1.0.3 sphinxcontrib-serializinghtml==1.1.5 SQLAlchemy==1.4.50 statsmodels @ file:///home/conda/feedstock_root/build_artifacts/statsmodels_1654787101575/work tensorboard==2.7.0 tensorboard-data-server==0.6.1 tensorboard-plugin-wit==1.8.0 tensorflow==2.7.0 tensorflow-estimator==2.7.0 tensorflow-hub @ file:///home/conda/feedstock_root/build_artifacts/tensorflow-hub_1689265971363/work/wheel_dir/tensorflow_hub-0.14.0-py2.py3-none-any.whl#sha256=b3f505c585faec74d197afb45fc183664d19d585b3c84865d933b64d30c4a339 tensorflow-io-gcs-filesystem==0.23.1 termcolor==1.1.0 terminado==0.17.1 threadpoolctl @ file:///home/conda/feedstock_root/build_artifacts/threadpoolctl_1643647933166/work tifffile @ file:///home/conda/feedstock_root/build_artifacts/tifffile_1635944860688/work tinycss2==1.2.1 toml==0.10.2 tomli==2.0.0 toolz @ file:///home/conda/feedstock_root/build_artifacts/toolz_1657485559105/work tornado==6.2 tqdm==4.66.1 traitlets==5.9.0 twine==3.3.0 typed-ast==1.5.5 typing_extensions @ file:///home/conda/feedstock_root/build_artifacts/typing_extensions_1688315532570/work tzlocal==5.1 uc-micro-py==1.0.2 unicodedata2 @ file:///home/conda/feedstock_root/build_artifacts/unicodedata2_1649111917568/work urllib3==1.26.7 virtualenv==20.24.6 wcwidth==0.2.9 webencodings==0.5.1 websocket-client==1.6.1 Werkzeug==2.0.2 widgetsnbextension==3.6.6 wrapt==1.13.3 zipp==3.6.0

How to reproduce

sleap-train baseline.centroid.json "personal.slp" --run_name "personal.centroid_1" --video-paths "Personal_data/"

talmo commented 11 months ago

Hi @ZVWan,

Thanks for the report! This may be ok as post-training evaluation can fail for all sorts of reasons unrelated to whether the model works or not.

When you use the model for inference are you getting reasonable predictions?

Cheers,

Talmo

ZVWan commented 11 months ago

Hi @talmo

Thank you for the reply! The inference seems reasonable, the accuracy of output looks acceptable, and the inferences can be successfully saved.

Besides, I also tried to generate the metric evaluation for a centered instance model I trained simutaneously with centroid model, and similar error occurred with no new file created.

Sincerely, Zhong

talmo commented 11 months ago

Thanks @ZVWan! Yep this might be because of it being a single node skeleton, or because something else funky is happening, but if you're getting good results I'd say it's safe to just ignore the metrics errors :)

Let us know if you need anything else!

Talmo