search

talmolab / sleap

A deep learning framework for multi-animal pose tracking.
https://sleap.ai
Other
435 stars 97 forks source link

trained buttomup model is not outputing all keypoints (trained from DLC data) #1662

Closed VincentCoulombe closed 10 months ago

VincentCoulombe commented 10 months ago

Hi,

I imported my DLC dataset and defined a skeleton (with the GUI) as follow: skeleton.json

Which gave me the following training config: training_config.json

I used the vanilla pretrained_buttomup.json (the one with efficientnet as the encoder). I inspected the labels and everything seems okay, it seems to have imported well.

The training went as follow: training_log.csv

I mean, until then everything went as expected. Congratulation on the nice software design, btw :)

Bug description

Because an image is worth a 1000 words, here's an example of my labels: labels

And here's an example of my predictions (obviously from the trained network): predictions

To be noted that every single prediction is missing the snout and the ears.

I am guessing that I screwed up somewhere in my training/inference configs, but I can't figure out where, so (just in case it might be an actual bug) I am leaving it here.

Thank you and have a nice day :)

Expected behaviour

Actual behaviour

Your personal set up

Environment packages absl-py==1.0.0 anyio==3.7.1 argon2-cffi==23.1.0 argon2-cffi-bindings==21.2.0 astunparse==1.6.3 attrs @ file:///home/conda/feedstock_root/build_artifacts/attrs_1640799537051/work backcall @ file:///home/conda/feedstock_root/build_artifacts/backcall_1592338393461/work backports.zoneinfo==0.2.1 beautifulsoup4==4.12.2 bleach==6.0.0 cached-property @ file:///home/conda/feedstock_root/build_artifacts/cached_property_1615209429212/work cachetools==4.2.4 cattrs @ file:///home/conda/feedstock_root/build_artifacts/cattrs_1604136207372/work certifi @ file:///home/conda/feedstock_root/build_artifacts/certifi_1700303426725/work/certifi cffi==1.15.1 charset-normalizer==2.0.9 cloudpickle @ file:///home/conda/feedstock_root/build_artifacts/cloudpickle_1674202310934/work comm==0.1.4 cycler @ file:///home/conda/feedstock_root/build_artifacts/cycler_1635519461629/work cytoolz @ file:///home/conda/feedstock_root/build_artifacts/cytoolz_1657553457169/work dask @ file:///home/conda/feedstock_root/build_artifacts/dask-core_1644602974678/work debugpy==1.7.0 decorator @ file:///home/conda/feedstock_root/build_artifacts/decorator_1641555617451/work defusedxml==0.7.1 efficientnet==1.0.0 entrypoints @ file:///home/conda/feedstock_root/build_artifacts/entrypoints_1643888246732/work exceptiongroup==1.2.0 fastjsonschema==2.19.1 flatbuffers==2.0 fonttools @ file:///home/conda/feedstock_root/build_artifacts/fonttools_1666389892786/work fsspec @ file:///home/conda/feedstock_root/build_artifacts/fsspec_1674184942191/work gast==0.4.0 google-auth==2.3.3 google-auth-oauthlib==0.4.6 google-pasta==0.2.0 grpcio==1.43.0 h5py @ file:///home/conda/feedstock_root/build_artifacts/h5py_1604753641401/work hdmf==3.6.1 idna==3.3 image-classifiers==1.0.0 imagecodecs @ file:///home/conda/feedstock_root/build_artifacts/imagecodecs_1644819473370/work imageio @ file:///home/conda/feedstock_root/build_artifacts/imageio_1702571712725/work imgaug @ file:///home/conda/feedstock_root/build_artifacts/imgaug_1640909786103/work imgstore==0.2.9 importlib-metadata==4.10.0 importlib-resources==5.12.0 ipykernel==6.16.2 ipython @ file:///home/conda/feedstock_root/build_artifacts/ipython_1651240553635/work ipython-genutils==0.2.0 ipywidgets==8.1.1 jedi @ file:///home/conda/feedstock_root/build_artifacts/jedi_1696326070614/work Jinja2==3.1.3 joblib @ file:///home/conda/feedstock_root/build_artifacts/joblib_1691577114857/work jsmin @ file:///home/conda/feedstock_root/build_artifacts/jsmin_1642532731678/work jsonpickle==1.2 jsonschema==4.17.3 jupyter==1.0.0 jupyter-console==6.6.3 jupyter-server==1.24.0 jupyter_client==7.4.9 jupyter_core==4.12.0 jupyterlab-pygments==0.2.2 jupyterlab-widgets==3.0.9 keras==2.7.0 Keras-Applications==1.0.8 Keras-Preprocessing==1.1.2 kiwisolver @ file:///home/conda/feedstock_root/build_artifacts/kiwisolver_1657953088445/work libclang==12.0.0 locket @ file:///home/conda/feedstock_root/build_artifacts/locket_1650660393415/work Markdown==3.3.6 markdown-it-py @ file:///home/conda/feedstock_root/build_artifacts/markdown-it-py_1677100944732/work MarkupSafe==2.1.3 matplotlib @ file:///home/conda/feedstock_root/build_artifacts/matplotlib-suite_1661439848456/work matplotlib-inline @ file:///home/conda/feedstock_root/build_artifacts/matplotlib-inline_1660814786464/work mdurl @ file:///home/conda/feedstock_root/build_artifacts/mdurl_1704317613764/work mistune==3.0.2 motmetrics==1.4.0 munkres==1.1.4 nbclassic==1.0.0 nbclient==0.7.4 nbconvert==7.6.0 nbformat==5.8.0 ndx-pose==0.1.1 nest-asyncio==1.5.8 networkx @ file:///home/conda/feedstock_root/build_artifacts/networkx_1635253012265/work nixio==1.5.3 notebook==6.5.6 notebook_shim==0.2.3 numpy @ file:///home/conda/feedstock_root/build_artifacts/numpy_1649806299270/work oauthlib==3.1.1 opencv-python==4.1.2.30 opencv-python-headless==4.2.0.34 opt-einsum==3.3.0 packaging==21.3 pandas==1.3.5 pandocfilters==1.5.0 parso @ file:///home/conda/feedstock_root/build_artifacts/parso_1638334955874/work partd @ file:///home/conda/feedstock_root/build_artifacts/partd_1695667515973/work patsy @ file:///home/conda/feedstock_root/build_artifacts/patsy_1704469236901/work pexpect @ file:///home/conda/feedstock_root/build_artifacts/pexpect_1667297516076/work pickleshare @ file:///home/conda/feedstock_root/build_artifacts/pickleshare_1602536217715/work Pillow @ file:///home/conda/feedstock_root/build_artifacts/pillow_1660385854171/work pkgutil_resolve_name==1.3.10 prometheus-client==0.17.1 prompt-toolkit @ file:///home/conda/feedstock_root/build_artifacts/prompt-toolkit_1702399386289/work protobuf==3.19.1 psutil @ file:///home/conda/feedstock_root/build_artifacts/psutil_1666155398032/work ptyprocess @ file:///home/conda/feedstock_root/build_artifacts/ptyprocess_1609419310487/work/dist/ptyprocess-0.7.0-py2.py3-none-any.whl pyasn1==0.4.8 pyasn1-modules==0.2.8 pycparser==2.21 Pygments @ file:///home/conda/feedstock_root/build_artifacts/pygments_1700607939962/work pykalman==0.9.5 pynwb==2.3.3 pyparsing==3.0.6 pyrsistent==0.19.3 PySide2==5.13.2 python-dateutil @ file:///home/conda/feedstock_root/build_artifacts/python-dateutil_1626286286081/work python-rapidjson @ file:///home/conda/feedstock_root/build_artifacts/python-rapidjson_1665999896718/work pytz @ file:///home/conda/feedstock_root/build_artifacts/pytz_1693930252784/work PyWavelets @ file:///home/conda/feedstock_root/build_artifacts/pywavelets_1649616401885/work PyYAML @ file:///home/conda/feedstock_root/build_artifacts/pyyaml_1648757092905/work pyzmq @ file:///home/conda/feedstock_root/build_artifacts/pyzmq_1663830492333/work qimage2ndarray==1.10.0 qtconsole==5.4.4 QtPy @ file:///home/conda/feedstock_root/build_artifacts/qtpy_1698112029416/work requests==2.26.0 requests-oauthlib==1.3.0 rich @ file:///home/conda/feedstock_root/build_artifacts/rich-split_1700160075651/work/dist rsa==4.8 ruamel.yaml==0.17.32 ruamel.yaml.clib==0.2.7 scikit-image @ file:///home/conda/feedstock_root/build_artifacts/scikit-image_1660146497293/work scikit-learn @ file:///home/conda/feedstock_root/build_artifacts/scikit-learn_1632611341839/work scikit-video==1.1.11 scipy @ file:///home/conda/feedstock_root/build_artifacts/scipy_1637806658031/work seaborn @ file:///home/conda/feedstock_root/build_artifacts/seaborn-split_1672497695270/work segmentation-models==1.0.1 Send2Trash==1.8.2 setuptools-scm==6.3.2 Shapely @ file:///home/conda/feedstock_root/build_artifacts/shapely_1665624546039/work shiboken2==5.13.2 six @ file:///home/conda/feedstock_root/build_artifacts/six_1620240208055/work sleap==1.3.3 sniffio==1.3.0 soupsieve==2.4.1 statsmodels @ file:///home/conda/feedstock_root/build_artifacts/statsmodels_1654787101575/work supervision==0.11.1 tensorboard==2.7.0 tensorboard-data-server==0.6.1 tensorboard-plugin-wit==1.8.0 tensorflow==2.7.0 tensorflow-estimator==2.7.0 tensorflow-hub @ file:///home/conda/feedstock_root/build_artifacts/tensorflow-hub_1678880940235/work/wheel_dir/tensorflow_hub-0.13.0-py2.py3-none-any.whl tensorflow-io-gcs-filesystem==0.23.1 termcolor==1.1.0 terminado==0.17.1 threadpoolctl @ file:///home/conda/feedstock_root/build_artifacts/threadpoolctl_1643647933166/work tifffile @ file:///home/conda/feedstock_root/build_artifacts/tifffile_1635944860688/work tinycss2==1.2.1 tomli==2.0.0 toolz @ file:///home/conda/feedstock_root/build_artifacts/toolz_1657485559105/work tornado @ file:///home/conda/feedstock_root/build_artifacts/tornado_1656937818679/work traitlets @ file:///home/conda/feedstock_root/build_artifacts/traitlets_1675110562325/work typing_extensions==4.0.1 tzlocal==5.0.1 unicodedata2 @ file:///home/conda/feedstock_root/build_artifacts/unicodedata2_1649111917568/work urllib3==1.26.7 wcwidth==0.1.9 webencodings==0.5.1 websocket-client==1.6.1 Werkzeug==2.0.2 widgetsnbextension==4.0.9 wrapt==1.13.3 xmltodict==0.13.0 zipp==3.15.0 ``` # paste output of `pip freeze` or `conda list` here ```
Logs ``` # paste relevant logs here, if any ```

Screenshots

How to reproduce

  1. Go to '...'
  2. Click on '....'
  3. Scroll down to '....'
  4. See error
talmo commented 10 months ago

Hi @VincentCoulombe,

Thanks for the detailed issue report!

Taking a first pass here: based on the screenshot, it looks like your skeleton might not form a tree (i.e., no node can have 2 parents).

This is an algorithmic requirement for bottom-up models. DLC relaxes this constraint at the cost of losing global optimality guarantees. An easy fix is to just edit your skeleton edges such that no node has two different source nodes.

As a quick test, you could just have all the nodes come from the centroid node as the source. This reduces dependency on lateral nodes and should give us a baseline for whether this is the source of the issue :)

Give it a spin and let us know how it goes!

Cheers,

Talmo

VincentCoulombe commented 10 months ago

Thank you for the swift response,

I re-trained with the proposed skeleton and it worked perfectly.

If you guys don't mind, I'll close the "issue".

Have a nice afternoon :)