search

talmolab / sleap

A deep learning framework for multi-animal pose tracking.
https://sleap.ai
Other
435 stars 97 forks source link

Failure to import, merge or open an analysis.h5 file in SLEAP GUI with skeleton edges #1746

Open amblypatty opened 7 months ago

amblypatty commented 7 months ago

I exported some analysis.h5 files from the SLEAP GUI from a slp file. Attempts to Import, Merge into Project or Open Project all fail to import the HDF5 file appropriately in the SLEAP GUI. While Import... SLEAP analysis HDF5 file works somewhat, the file has all the information it should need to pull the videos and generate the edges of the skeleton, but the GUI asks for the location of the video file and only keypoint positions are populated once the video is located. The other two options fail with a TypeError without an adaptor.

The expected behavior is for the SLEAP GUI to use the information from the analysis.h5 file to create a new SLEAP project file or to be able to merge into an existing SLEAP project and have a full skeleton set up. The keypoints are populated in each frame with a track but without the rest of the skeleton, even though the analysis.h5 file contains edge_inds and edge_names.

From the image below of the analysis.h5 file open in HDFView, all of the file parts are there that should be able to be used to open the file in SLEAP with the full skeleton intact and without having to locate the video file unless the video_path no longer exists. Additionally, users should be able to Merge into Project from HDF5 files if they contain all the information.

Your personal set up

Windows 10

Environment packages ``` # packages in environment at C:\ProgramData\Anaconda2\envs\sleap133: # # Name Version Build Channel absl-py 1.0.0 pypi_0 pypi anyio 3.7.1 pypi_0 pypi argon2-cffi 23.1.0 pypi_0 pypi argon2-cffi-bindings 21.2.0 pypi_0 pypi astunparse 1.6.3 pypi_0 pypi attrs 21.4.0 pyhd8ed1ab_0 conda-forge av 10.0.0 pypi_0 pypi backcall 0.2.0 pypi_0 pypi backports-zoneinfo 0.2.1 pypi_0 pypi beautifulsoup4 4.12.2 pypi_0 pypi bleach 6.0.0 pypi_0 pypi blosc 1.21.5 hdccc3a2_0 conda-forge brotli 1.0.9 hcfcfb64_9 conda-forge brotli-bin 1.0.9 hcfcfb64_9 conda-forge bzip2 1.0.8 h8ffe710_4 conda-forge c-blosc2 2.10.2 h183a6f4_0 conda-forge ca-certificates 2023.7.22 h56e8100_0 conda-forge cached-property 1.5.2 pypi_0 pypi cachetools 4.2.4 pypi_0 pypi cattrs 1.1.1 pyhd8ed1ab_0 conda-forge certifi 2023.7.22 pyhd8ed1ab_0 conda-forge cffi 1.15.1 pypi_0 pypi cfitsio 4.0.0 hd67004f_0 conda-forge charls 2.3.4 h39d44d4_0 conda-forge charset-normalizer 2.0.9 pypi_0 pypi cloudpickle 2.2.1 pyhd8ed1ab_0 conda-forge colorama 0.4.6 pypi_0 pypi cuda-nvcc 11.3.58 hb8d16a4_0 nvidia cudatoolkit 11.3.1 hf2f0253_12 conda-forge cudnn 8.2.1.32 h754d62a_0 conda-forge cycler 0.11.0 pyhd8ed1ab_0 conda-forge cytoolz 0.12.0 py37hcc03f2d_0 conda-forge dask-core 2022.2.0 pyhd8ed1ab_0 conda-forge debugpy 1.7.0 pypi_0 pypi decorator 5.1.1 pypi_0 pypi defusedxml 0.7.1 pypi_0 pypi efficientnet 1.0.0 pypi_0 pypi entrypoints 0.4 pypi_0 pypi exceptiongroup 1.2.0 pypi_0 pypi fastjsonschema 2.19.0 pypi_0 pypi flatbuffers 2.0 pypi_0 pypi fonttools 4.38.0 py37h51bd9d9_0 conda-forge freeglut 3.2.2 h63175ca_2 conda-forge freetype 2.12.1 hdaf720e_2 conda-forge fsspec 2023.1.0 pyhd8ed1ab_0 conda-forge gast 0.4.0 pypi_0 pypi geos 3.11.0 h39d44d4_0 conda-forge giflib 5.2.1 h64bf75a_3 conda-forge google-auth 2.3.3 pypi_0 pypi google-auth-oauthlib 0.4.6 pypi_0 pypi google-pasta 0.2.0 pypi_0 pypi grpcio 1.43.0 pypi_0 pypi h5py 3.1.0 pypi_0 pypi hdmf 3.6.1 pypi_0 pypi icu 69.1 h0e60522_0 conda-forge idna 3.3 pypi_0 pypi image-classifiers 1.0.0 pypi_0 pypi imagecodecs 2021.11.20 py37hfea8715_2 conda-forge imageio 2.31.1 pyh24c5eb1_0 conda-forge imageio-ffmpeg 0.4.9 pypi_0 pypi imgaug 0.4.0 pyhd8ed1ab_1 conda-forge imgstore 0.2.9 pypi_0 pypi importlib-metadata 4.2.0 pypi_0 pypi importlib-resources 5.12.0 pypi_0 pypi intel-openmp 2023.2.0 h57928b3_49503 conda-forge ipykernel 6.16.2 pypi_0 pypi ipython 7.34.0 pypi_0 pypi ipython-genutils 0.2.0 pypi_0 pypi jasper 2.0.33 hc2e4405_1 conda-forge jedi 0.19.1 pypi_0 pypi jinja2 3.1.2 pypi_0 pypi joblib 1.3.2 pyhd8ed1ab_0 conda-forge jpeg 9e hcfcfb64_3 conda-forge jsmin 3.0.1 pyhd8ed1ab_0 conda-forge jsonpickle 1.2 py_0 conda-forge jsonschema 4.17.3 pypi_0 pypi jupyter-client 7.4.9 pypi_0 pypi jupyter-core 4.12.0 pypi_0 pypi jupyter-server 1.24.0 pypi_0 pypi jupyterlab-pygments 0.2.2 pypi_0 pypi jxrlib 1.1 h8ffe710_2 conda-forge keras 2.7.0 pypi_0 pypi keras-applications 1.0.8 pypi_0 pypi keras-preprocessing 1.1.2 pypi_0 pypi kiwisolver 1.4.4 py37h8c56517_0 conda-forge krb5 1.20.1 h6609f42_0 conda-forge lcms2 2.14 h90d422f_0 conda-forge lerc 3.0 h0e60522_0 conda-forge libaec 1.0.6 h63175ca_1 conda-forge libblas 3.9.0 18_win64_mkl conda-forge libbrotlicommon 1.0.9 hcfcfb64_9 conda-forge libbrotlidec 1.0.9 hcfcfb64_9 conda-forge libbrotlienc 1.0.9 hcfcfb64_9 conda-forge libcblas 3.9.0 18_win64_mkl conda-forge libclang 12.0.0 pypi_0 pypi libcurl 8.1.2 h68f0423_0 conda-forge libdeflate 1.10 h8ffe710_0 conda-forge libhwloc 2.9.1 h51c2c0f_0 conda-forge libiconv 1.17 h8ffe710_0 conda-forge liblapack 3.9.0 18_win64_mkl conda-forge liblapacke 3.9.0 18_win64_mkl conda-forge libopencv 4.5.1 py37ha0199f4_0 conda-forge libpng 1.6.39 h19919ed_0 conda-forge libprotobuf 3.21.8 h12be248_0 conda-forge libsodium 1.0.18 h8d14728_1 conda-forge libsqlite 3.43.0 hcfcfb64_0 conda-forge libssh2 1.10.0 h680486a_3 conda-forge libtiff 4.4.0 hc4061b1_0 conda-forge libwebp-base 1.3.2 hcfcfb64_0 conda-forge libxcb 1.13 hcd874cb_1004 conda-forge libxml2 2.10.4 hc3477c8_0 conda-forge libxslt 1.1.37 h0192164_0 conda-forge libzlib 1.2.13 hcfcfb64_5 conda-forge libzopfli 1.0.3 h0e60522_0 conda-forge locket 1.0.0 pyhd8ed1ab_0 conda-forge lz4-c 1.9.3 h8ffe710_1 conda-forge m2w64-gcc-libgfortran 5.3.0 6 conda-forge m2w64-gcc-libs 5.3.0 7 conda-forge m2w64-gcc-libs-core 5.3.0 7 conda-forge m2w64-gmp 6.1.0 2 conda-forge m2w64-libwinpthread-git 5.0.0.4634.697f757 2 conda-forge markdown 3.3.6 pypi_0 pypi markdown-it-py 2.2.0 pyhd8ed1ab_0 conda-forge markupsafe 2.1.3 pypi_0 pypi matplotlib-base 3.5.3 py37hbaab90a_2 conda-forge matplotlib-inline 0.1.6 pypi_0 pypi mdurl 0.1.0 pyhd8ed1ab_0 conda-forge mistune 3.0.2 pypi_0 pypi mkl 2022.1.0 h6a75c08_874 conda-forge msys2-conda-epoch 20160418 1 conda-forge munkres 1.1.4 pyh9f0ad1d_0 conda-forge nbclassic 1.0.0 pypi_0 pypi nbclient 0.7.4 pypi_0 pypi nbconvert 7.6.0 pypi_0 pypi nbformat 5.8.0 pypi_0 pypi ndx-pose 0.1.1 pypi_0 pypi nest-asyncio 1.5.8 pypi_0 pypi networkx 2.6.3 pyhd8ed1ab_1 conda-forge nixio 1.5.3 pypi_0 pypi notebook 6.5.6 pypi_0 pypi notebook-shim 0.2.3 pypi_0 pypi numpy 1.19.5 pypi_0 pypi oauthlib 3.1.1 pypi_0 pypi opencv 4.5.1 py37h03978a9_0 conda-forge opencv-python-headless 4.2.0.34 pypi_0 pypi openjpeg 2.5.0 hc9384bd_1 conda-forge openssl 1.1.1w hcfcfb64_0 conda-forge opt-einsum 3.3.0 pypi_0 pypi packaging 21.3 pypi_0 pypi pandas 1.3.5 py37h9386db6_0 conda-forge pandocfilters 1.5.0 pypi_0 pypi parso 0.8.3 pypi_0 pypi partd 1.4.0 pyhd8ed1ab_1 conda-forge patsy 0.5.3 pyhd8ed1ab_0 conda-forge pickleshare 0.7.5 pypi_0 pypi pillow 9.2.0 py37h42a8222_2 conda-forge pip 23.2.1 pyhd8ed1ab_0 conda-forge pkgutil-resolve-name 1.3.10 pypi_0 pypi prometheus-client 0.17.1 pypi_0 pypi prompt-toolkit 3.0.41 pypi_0 pypi protobuf 3.19.1 pypi_0 pypi psutil 5.9.3 py37h51bd9d9_0 conda-forge pthread-stubs 0.4 hcd874cb_1001 conda-forge pthreads-win32 2.9.1 hfa6e2cd_3 conda-forge py-opencv 4.5.1 py37heaed05f_0 conda-forge pyasn1 0.4.8 pypi_0 pypi pyasn1-modules 0.2.8 pypi_0 pypi pycparser 2.21 pypi_0 pypi pygments 2.16.1 pyhd8ed1ab_0 conda-forge pykalman 0.9.5 py_1 conda-forge pynwb 2.3.3 pypi_0 pypi pyparsing 3.0.6 pypi_0 pypi pyrsistent 0.19.3 pypi_0 pypi pyside2 5.13.2 py37h760f651_8 conda-forge python 3.7.12 h7840368_100_cpython conda-forge python-dateutil 2.8.2 pyhd8ed1ab_0 conda-forge python-rapidjson 1.9 py37h7f67f24_0 conda-forge python_abi 3.7 4_cp37m conda-forge pytz 2023.3.post1 pyhd8ed1ab_0 conda-forge pywavelets 1.3.0 py37h3a130e4_1 conda-forge pywin32 306 pypi_0 pypi pywinpty 2.0.10 pypi_0 pypi pyyaml 6.0 py37hcc03f2d_4 conda-forge pyzmq 24.0.1 py37h7347f05_0 conda-forge qimage2ndarray 1.10.0 pypi_0 pypi qt 5.12.9 h556501e_6 conda-forge qtpy 2.4.0 pyhd8ed1ab_0 conda-forge requests 2.26.0 pypi_0 pypi requests-oauthlib 1.3.0 pypi_0 pypi rich 13.5.3 pyhd8ed1ab_0 conda-forge ruamel-yaml 0.17.32 pypi_0 pypi ruamel-yaml-clib 0.2.7 pypi_0 pypi scikit-image 0.19.3 py37h3182a2c_1 conda-forge scikit-learn 1.0 py37ha78be43_1 conda-forge scikit-video 1.1.11 pyh24bf2e0_0 conda-forge scipy 1.7.3 py37hb6553fb_0 conda-forge seaborn 0.12.2 hd8ed1ab_0 conda-forge seaborn-base 0.12.2 pyhd8ed1ab_0 conda-forge segmentation-models 1.0.1 pypi_0 pypi send2trash 1.8.2 pypi_0 pypi setuptools 59.8.0 py37h03978a9_1 conda-forge setuptools-scm 6.3.2 pypi_0 pypi shapely 1.8.5 py37h475e9a0_0 conda-forge simplejson 3.19.2 pypi_0 pypi six 1.16.0 pyh6c4a22f_0 conda-forge sleap 1.3.3 pypi_0 pypi sleap-io 0.0.13 pypi_0 pypi snappy 1.1.10 hfb803bf_0 conda-forge sniffio 1.3.0 pypi_0 pypi soupsieve 2.4.1 pypi_0 pypi sqlite 3.43.0 hcfcfb64_0 conda-forge statsmodels 0.13.2 py37h3a130e4_0 conda-forge tbb 2021.9.0 h91493d7_0 conda-forge tensorboard 2.7.0 pypi_0 pypi tensorboard-data-server 0.6.1 pypi_0 pypi tensorboard-plugin-wit 1.8.0 pypi_0 pypi tensorflow 2.7.0 pypi_0 pypi tensorflow-estimator 2.7.0 pypi_0 pypi tensorflow-hub 0.12.0 pyhca92ed8_0 conda-forge tensorflow-io-gcs-filesystem 0.23.1 pypi_0 pypi termcolor 1.1.0 pypi_0 pypi terminado 0.17.1 pypi_0 pypi threadpoolctl 3.1.0 pyh8a188c0_0 conda-forge tifffile 2021.11.2 pyhd8ed1ab_0 conda-forge tinycss2 1.2.1 pypi_0 pypi tk 8.6.13 hcfcfb64_0 conda-forge tomli 2.0.0 pypi_0 pypi toolz 0.12.0 pyhd8ed1ab_0 conda-forge tornado 6.2 pypi_0 pypi traitlets 5.9.0 pypi_0 pypi typing-extensions 4.0.1 pypi_0 pypi typing_extensions 4.7.1 pyha770c72_0 conda-forge tzdata 2023.3 pypi_0 pypi tzlocal 5.0.1 pypi_0 pypi ucrt 10.0.22621.0 h57928b3_0 conda-forge unicodedata2 14.0.0 py37hcc03f2d_1 conda-forge urllib3 1.26.7 pypi_0 pypi vc 14.3 h64f974e_17 conda-forge vc14_runtime 14.36.32532 hdcecf7f_17 conda-forge vs2015_runtime 14.36.32532 h05e6639_17 conda-forge wcwidth 0.2.12 pypi_0 pypi webencodings 0.5.1 pypi_0 pypi websocket-client 1.6.1 pypi_0 pypi werkzeug 2.0.2 pypi_0 pypi wheel 0.41.2 pyhd8ed1ab_0 conda-forge wrapt 1.13.3 pypi_0 pypi xorg-libxau 1.0.11 hcd874cb_0 conda-forge xorg-libxdmcp 1.1.3 hcd874cb_0 conda-forge xz 5.2.6 h8d14728_0 conda-forge yaml 0.2.5 h8ffe710_2 conda-forge zeromq 4.3.4 h0e60522_1 conda-forge zfp 0.5.5 h0e60522_8 conda-forge zipp 3.15.0 pypi_0 pypi zlib 1.2.13 hcfcfb64_5 conda-forge zlib-ng 2.0.7 hcfcfb64_0 conda-forge zstd 1.5.5 h12be248_0 conda-forge ```
Logs ``` C:/Users/psych/SLEAP-Data/Analysis_h5s/Runway_experiment_1.analysis.h5 doesn't match ext for json or json.zip Traceback (most recent call last): File "C:\ProgramData\Anaconda2\envs\sleap133\lib\site-packages\sleap\gui\commands.py", line 289, in openProject self.execute(OpenProject, filename=filename, first_open=first_open) File "C:\ProgramData\Anaconda2\envs\sleap133\lib\site-packages\sleap\gui\commands.py", line 242, in execute command().execute(context=self, params=kwargs) File "C:\ProgramData\Anaconda2\envs\sleap133\lib\site-packages\sleap\gui\commands.py", line 138, in execute self.do_with_signal(context, params) File "C:\ProgramData\Anaconda2\envs\sleap133\lib\site-packages\sleap\gui\commands.py", line 162, in do_with_signal cls.do_action(context, params) File "C:\ProgramData\Anaconda2\envs\sleap133\lib\site-packages\sleap\gui\commands.py", line 727, in do_action context.loadProjectFile(filename) File "C:\ProgramData\Anaconda2\envs\sleap133\lib\site-packages\sleap\gui\commands.py", line 274, in loadProjectFile self.execute(LoadProjectFile, filename=filename) File "C:\ProgramData\Anaconda2\envs\sleap133\lib\site-packages\sleap\gui\commands.py", line 242, in execute command().execute(context=self, params=kwargs) File "C:\ProgramData\Anaconda2\envs\sleap133\lib\site-packages\sleap\gui\commands.py", line 136, in execute okay = self.ask(context, params) File "C:\ProgramData\Anaconda2\envs\sleap133\lib\site-packages\sleap\gui\commands.py", line 698, in ask labels = Labels.load_file(filename, video_search=gui_video_callback) File "C:\ProgramData\Anaconda2\envs\sleap133\lib\site-packages\sleap\io\dataset.py", line 1969, in load_file filename, for_object="labels", video_search=video_search, *args, **kwargs File "C:\ProgramData\Anaconda2\envs\sleap133\lib\site-packages\sleap\io\format\main.py", line 113, in read return disp.read(filename, *args, **kwargs) File "C:\ProgramData\Anaconda2\envs\sleap133\lib\site-packages\sleap\io\format\dispatch.py", line 59, in read raise TypeError("No file format adaptor could read this file.") TypeError: No file format adaptor could read this file. ``` Open Project... above ^ Merge into Project... ``` Happy SLEAPing! :) C:/Users/psych/SLEAP-Data/Analysis_h5s/Runway_experiment_1.analysis.h5 doesn't match ext for json or json.zip Traceback (most recent call last): File "C:\ProgramData\Anaconda2\envs\sleap133\lib\site-packages\sleap\gui\commands.py", line 606, in mergeProject self.execute(MergeProject, filenames=filenames) File "C:\ProgramData\Anaconda2\envs\sleap133\lib\site-packages\sleap\gui\commands.py", line 242, in execute command().execute(context=self, params=kwargs) File "C:\ProgramData\Anaconda2\envs\sleap133\lib\site-packages\sleap\gui\commands.py", line 134, in execute self.ask_and_do(context, params) File "C:\ProgramData\Anaconda2\envs\sleap133\lib\site-packages\sleap\gui\commands.py", line 2843, in ask_and_do new_labels = Labels.load_file(filename, video_search=gui_video_callback) File "C:\ProgramData\Anaconda2\envs\sleap133\lib\site-packages\sleap\io\dataset.py", line 1969, in load_file filename, for_object="labels", video_search=video_search, *args, **kwargs File "C:\ProgramData\Anaconda2\envs\sleap133\lib\site-packages\sleap\io\format\main.py", line 113, in read return disp.read(filename, *args, **kwargs) File "C:\ProgramData\Anaconda2\envs\sleap133\lib\site-packages\sleap\io\format\dispatch.py", line 59, in read raise TypeError("No file format adaptor could read this file.") TypeError: No file format adaptor could read this file. ```

Screenshots

Image of the HDF5 file contents: image

Image of SLEAP asking to find the video even though the HDF5 file contents has the video_path: image

Image of the imported HDF5 file with only keypoints showing. While I cannot screen capture it, the setting for Show Edges is enabled with a check mark:

image

The skeleton shows no symmetry or edges. But the HDF5 file contains the information needed to create the full skeleton from edge_inds, edge_names, and node_names. Currently, it seems that only node_names is utilized.

How to reproduce

  1. Go to 'sleap-label in sleap conda environment...'
  2. Click on 'Import...SLEAP analysis HDF5 or Merge Project... or Open Project...'
  3. Locate the analysis.h5 file.
  4. Either locate the video file from the pop-up GUI prompt or see TypeError traceback in conda-CLI.
  5. See partial skeleton without edges if importing works
roomrys commented 7 months ago

The Open and Merge into project have historically not been intended to be used to handle analysis files. In general, the lack of support around loading in analysis files is because these were intended to be used as the last step of working with SLEAP (once analysis was exported, you would not come back and make changes to the analysis). Although, by this issue and the other related issues, it seems there is some interest in modifying the data in the analysis and then piping the modified data back into SLEAP.

amblypatty commented 7 months ago

Indeed, using the analysis files to edit mismatched tracks (at least when there should be only one track, but SLEAP created one for the user-labeled instances and a separate one for the predicted instances) is far easier than using the GUI to go through each video to swap them. Cleaning and retracking did not work either, so editing the analysis files was my workaround.