search

talmolab / sleap

A deep learning framework for multi-animal pose tracking.
https://sleap.ai
Other
435 stars 97 forks source link

Crashing during training on mac OS 14 (Sonoma) #1602

Closed hl8536 closed 7 months ago

hl8536 commented 1 year ago

Bug description

SLEAP crashing during training with the following error output.

Your personal set up

Environment packages ``` # Name Version Build Channel abseil-cpp 20211102.0 he4e09e4_3 conda-forge absl-py 1.4.0 pypi_0 pypi aiohttp 3.8.6 py39h17cfd9d_1 conda-forge aiosignal 1.3.1 pyhd8ed1ab_0 conda-forge aom 3.5.0 h7ea286d_0 conda-forge astunparse 1.6.3 pyhd8ed1ab_0 conda-forge async-timeout 4.0.3 pyhd8ed1ab_0 conda-forge attrs 23.1.0 pyh71513ae_1 conda-forge blinker 1.7.0 pyhd8ed1ab_0 conda-forge blosc 1.21.5 hc338f07_0 conda-forge brotli 1.0.9 h1a8c8d9_9 conda-forge brotli-bin 1.0.9 h1a8c8d9_9 conda-forge brotli-python 1.0.9 py39h23fbdae_9 conda-forge brunsli 0.1 h9f76cd9_0 conda-forge bzip2 1.0.8 h3422bc3_4 conda-forge c-ares 1.21.0 h93a5062_0 conda-forge c-blosc2 2.10.5 h8eb3132_0 conda-forge ca-certificates 2023.7.22 hf0a4a13_0 conda-forge cached-property 1.5.2 hd8ed1ab_1 conda-forge cached_property 1.5.2 pyha770c72_1 conda-forge cachetools 5.3.1 pypi_0 pypi cairo 1.16.0 h73a0509_1014 conda-forge cattrs 1.1.1 pyhd8ed1ab_0 conda-forge certifi 2023.7.22 pyhd8ed1ab_0 conda-forge cffi 1.16.0 py39he153c15_0 conda-forge cfitsio 4.2.0 h2f961c4_0 conda-forge charls 2.3.4 hbdafb3b_0 conda-forge charset-normalizer 3.2.0 pypi_0 pypi click 8.1.7 unix_pyh707e725_0 conda-forge contourpy 1.1.1 py39hbd775c9_1 conda-forge cryptography 39.0.0 py39haa0b8cc_0 conda-forge cycler 0.12.1 pyhd8ed1ab_0 conda-forge dav1d 1.2.1 hb547adb_0 conda-forge efficientnet 1.0.0 pypi_0 pypi expat 2.5.0 hb7217d7_1 conda-forge ffmpeg 4.4.2 gpl_hf318d42_112 conda-forge font-ttf-dejavu-sans-mono 2.37 hab24e00_0 conda-forge font-ttf-inconsolata 3.000 h77eed37_0 conda-forge font-ttf-source-code-pro 2.038 h77eed37_0 conda-forge font-ttf-ubuntu 0.83 hab24e00_0 conda-forge fontconfig 2.14.2 h82840c6_0 conda-forge fonts-conda-ecosystem 1 0 conda-forge fonts-conda-forge 1 0 conda-forge fonttools 4.43.1 py39h17cfd9d_0 conda-forge freetype 2.12.1 hadb7bae_2 conda-forge frozenlist 1.4.0 py39h0f82c59_1 conda-forge gast 0.4.0 pyh9f0ad1d_0 conda-forge geos 3.12.0 h13dd4ca_0 conda-forge gettext 0.21.1 h0186832_0 conda-forge giflib 5.2.1 h1a8c8d9_3 conda-forge glib 2.78.0 ha614eb4_0 conda-forge glib-tools 2.78.0 ha614eb4_0 conda-forge gmp 6.2.1 h9f76cd9_0 conda-forge gnutls 3.7.8 h9f1a10d_0 conda-forge google-auth 2.23.0 pypi_0 pypi google-auth-oauthlib 0.4.6 pyhd8ed1ab_0 conda-forge google-pasta 0.2.0 pyh8c360ce_0 conda-forge graphite2 1.3.13 h9f76cd9_1001 conda-forge grpc-cpp 1.46.4 hcaf9be7_3 conda-forge grpcio 1.58.0 pypi_0 pypi gst-plugins-base 1.22.6 h27255cc_2 conda-forge gstreamer 1.22.6 h8c52bba_2 conda-forge h5py 3.8.0 nompi_py39hc9149d8_100 conda-forge harfbuzz 5.3.0 hddbc195_0 conda-forge hdf5 1.12.2 nompi_h55deafc_101 conda-forge hdmf 3.9.0 pypi_0 pypi icu 70.1 h6b3803e_0 conda-forge idna 3.4 pyhd8ed1ab_0 conda-forge image-classifiers 1.0.0 pypi_0 pypi imagecodecs 2022.9.26 py39hd7f743f_4 conda-forge imageio 2.31.5 pyh8c1a49c_0 conda-forge imgaug 0.4.0 pyhd8ed1ab_1 conda-forge imgstore 0.2.9 pypi_0 pypi importlib-metadata 6.8.0 pyha770c72_0 conda-forge importlib-resources 6.1.0 pyhd8ed1ab_0 conda-forge importlib_resources 6.1.0 pyhd8ed1ab_0 conda-forge jasper 2.0.33 hc3cd1e9_1 conda-forge joblib 1.3.2 pyhd8ed1ab_0 conda-forge jpeg 9e h1a8c8d9_3 conda-forge jsmin 3.0.1 pyhd8ed1ab_0 conda-forge jsonpickle 1.2 py_0 conda-forge jsonschema 4.19.0 pypi_0 pypi jsonschema-specifications 2023.7.1 pypi_0 pypi jxrlib 1.1 h27ca646_2 conda-forge keras 2.9.0 pyhd8ed1ab_0 conda-forge keras-applications 1.0.8 pypi_0 pypi keras-preprocessing 1.1.2 pyhd8ed1ab_0 conda-forge kiwisolver 1.4.5 py39hbd775c9_1 conda-forge krb5 1.20.1 h127bd45_0 conda-forge lame 3.100 h1a8c8d9_1003 conda-forge lazy_loader 0.3 pyhd8ed1ab_0 conda-forge lcms2 2.14 h8193b64_0 conda-forge lerc 4.0.0 h9a09cb3_0 conda-forge libabseil 20211102.0 cxx17_h28b99d4_3 conda-forge libaec 1.1.2 h13dd4ca_1 conda-forge libavif 0.11.1 h9f83d30_2 conda-forge libblas 3.9.0 19_osxarm64_openblas conda-forge libbrotlicommon 1.0.9 h1a8c8d9_9 conda-forge libbrotlidec 1.0.9 h1a8c8d9_9 conda-forge libbrotlienc 1.0.9 h1a8c8d9_9 conda-forge libcblas 3.9.0 19_osxarm64_openblas conda-forge libclang 16.0.6 pypi_0 pypi libclang13 14.0.6 default_hc7183e1_1 conda-forge libcurl 7.87.0 hbe9bab4_0 conda-forge libcxx 16.0.6 h4653b0c_0 conda-forge libdeflate 1.14 h1a8c8d9_0 conda-forge libedit 3.1.20191231 hc8eb9b7_2 conda-forge libev 4.33 h642e427_1 conda-forge libexpat 2.5.0 hb7217d7_1 conda-forge libffi 3.4.2 h3422bc3_5 conda-forge libgfortran 5.0.0 13_2_0_hd922786_1 conda-forge libgfortran5 13.2.0 hf226fd6_1 conda-forge libglib 2.78.0 h24e9cb9_0 conda-forge libiconv 1.17 he4db4b2_0 conda-forge libidn2 2.3.4 h1a8c8d9_0 conda-forge liblapack 3.9.0 19_osxarm64_openblas conda-forge liblapacke 3.9.0 19_osxarm64_openblas conda-forge libllvm14 14.0.6 hd1a9a77_4 conda-forge libnghttp2 1.51.0 hd184df1_0 conda-forge libogg 1.3.4 h27ca646_1 conda-forge libopenblas 0.3.24 openmp_hd76b1f2_0 conda-forge libopencv 4.6.0 py39he1c1adf_3 conda-forge libopus 1.3.1 h27ca646_1 conda-forge libpng 1.6.39 h76d750c_0 conda-forge libpq 15.1 hbce9e56_3 conda-forge libprotobuf 3.20.3 hb5ab8b9_0 conda-forge libsodium 1.0.18 h27ca646_1 conda-forge libsqlite 3.44.0 h091b4b1_0 conda-forge libssh2 1.10.0 hb80f160_3 conda-forge libtasn1 4.19.0 h1a8c8d9_0 conda-forge libtiff 4.4.0 heb92581_5 conda-forge libunistring 0.9.10 h3422bc3_0 conda-forge libvorbis 1.3.7 h9f76cd9_0 conda-forge libvpx 1.11.0 hc470f4d_3 conda-forge libwebp-base 1.3.2 hb547adb_0 conda-forge libxcb 1.13 h9b22ae9_1004 conda-forge libxml2 2.10.3 h67585b2_4 conda-forge libxslt 1.1.37 h1bd8bc4_0 conda-forge libzlib 1.2.13 h53f4e23_5 conda-forge libzopfli 1.0.3 h9f76cd9_0 conda-forge llvm-openmp 17.0.4 hcd81f8e_0 conda-forge lz4-c 1.9.4 hb7217d7_0 conda-forge markdown 3.4.4 pypi_0 pypi markdown-it-py 3.0.0 pyhd8ed1ab_0 conda-forge markupsafe 2.1.3 py39h0f82c59_1 conda-forge matplotlib-base 3.8.1 py39h1a09f3e_0 conda-forge mdurl 0.1.0 pyhd8ed1ab_0 conda-forge multidict 6.0.4 py39h02fc5c5_1 conda-forge munkres 1.1.4 pyh9f0ad1d_0 conda-forge mysql-common 8.0.32 hab468bb_0 conda-forge mysql-libs 8.0.32 hea58576_0 conda-forge ncurses 6.4 h463b476_2 conda-forge ndx-pose 0.1.1 pypi_0 pypi nettle 3.8.1 h63371fa_1 conda-forge networkx 3.2.1 pyhd8ed1ab_0 conda-forge nixio 1.5.3 pypi_0 pypi nspr 4.35 hb7217d7_0 conda-forge nss 3.94 hc6b9969_0 conda-forge numpy 1.22.4 py39h7df2422_0 conda-forge oauthlib 3.2.2 pyhd8ed1ab_0 conda-forge opencv 4.6.0 py39hdf13c20_3 conda-forge openh264 2.3.1 hb7217d7_2 conda-forge openjpeg 2.5.0 h5d4e404_1 conda-forge openssl 1.1.1w h53f4e23_0 conda-forge opt_einsum 3.3.0 pyhc1e730c_2 conda-forge p11-kit 0.24.1 h29577a5_0 conda-forge packaging 23.2 pyhd8ed1ab_0 conda-forge pandas 2.1.2 py39hf8cecc8_0 conda-forge patsy 0.5.3 pyhd8ed1ab_0 conda-forge pcre2 10.40 hb34f9b4_0 conda-forge pillow 9.2.0 py39h139752e_3 conda-forge pip 23.3.1 pyhd8ed1ab_0 conda-forge pixman 0.42.2 h13dd4ca_0 conda-forge protobuf 3.19.6 pypi_0 pypi psutil 5.9.5 py39h0f82c59_1 conda-forge pthread-stubs 0.4 h27ca646_1001 conda-forge py-opencv 4.6.0 py39hfa6204d_3 conda-forge pyasn1 0.5.0 pyhd8ed1ab_0 conda-forge pyasn1-modules 0.3.0 pyhd8ed1ab_0 conda-forge pycparser 2.21 pyhd8ed1ab_0 conda-forge pygments 2.16.1 pyhd8ed1ab_0 conda-forge pyjwt 2.8.0 pyhd8ed1ab_0 conda-forge pykalman 0.9.5 py_1 conda-forge pynwb 2.5.0 pypi_0 pypi pyopenssl 23.2.0 pyhd8ed1ab_1 conda-forge pyparsing 3.1.1 pyhd8ed1ab_0 conda-forge pyside2 5.15.8 py39h0adaba8_2 conda-forge pysocks 1.7.1 pyha2e5f31_6 conda-forge python 3.9.15 h2d96c93_0_cpython conda-forge python-dateutil 2.8.2 pyhd8ed1ab_0 conda-forge python-flatbuffers 1.12 pyhd8ed1ab_1 conda-forge python-rapidjson 1.13 py39h4ce5507_0 conda-forge python-tzdata 2023.3 pyhd8ed1ab_0 conda-forge python_abi 3.9 4_cp39 conda-forge pytz 2023.3.post1 pyhd8ed1ab_0 conda-forge pyu2f 0.1.5 pyhd8ed1ab_0 conda-forge pywavelets 1.4.1 py39hf4a74a7_1 conda-forge pyyaml 6.0.1 py39h0f82c59_1 conda-forge pyzmq 25.1.1 py39he0a3c8b_2 conda-forge qimage2ndarray 1.10.0 pypi_0 pypi qt-main 5.15.8 hfe8d25c_6 conda-forge qtpy 2.4.1 pyhd8ed1ab_0 conda-forge re2 2022.06.01 h9a09cb3_1 conda-forge readline 8.2 h92ec313_1 conda-forge referencing 0.30.2 pypi_0 pypi requests 2.31.0 pyhd8ed1ab_0 conda-forge requests-oauthlib 1.3.1 pyhd8ed1ab_0 conda-forge rich 13.6.0 pyhd8ed1ab_0 conda-forge rpds-py 0.10.3 pypi_0 pypi rsa 4.9 pyhd8ed1ab_0 conda-forge ruamel-yaml 0.17.32 pypi_0 pypi ruamel-yaml-clib 0.2.7 pypi_0 pypi scikit-image 0.22.0 py39hf8cecc8_2 conda-forge scikit-learn 1.0 py39h12ba089_1 conda-forge scikit-video 1.1.11 pyh24bf2e0_0 conda-forge scipy 1.9.0 py39h14896cb_0 conda-forge seaborn 0.13.0 hd8ed1ab_0 conda-forge seaborn-base 0.13.0 pyhd8ed1ab_0 conda-forge segmentation-models 1.0.1 pypi_0 pypi setuptools 68.2.2 pyhd8ed1ab_0 conda-forge shapely 2.0.2 py39h4b213b6_0 conda-forge six 1.15.0 pypi_0 pypi sleap 1.3.3 pypi_0 pypi snappy 1.1.10 h17c5cce_0 conda-forge sqlite 3.44.0 hf2abe2d_0 conda-forge statsmodels 0.14.0 py39h373d45f_2 conda-forge svt-av1 1.4.1 h7ea286d_0 conda-forge tensorboard 2.9.1 pypi_0 pypi tensorboard-data-server 0.6.1 py39haa0b8cc_4 conda-forge tensorboard-plugin-wit 1.8.1 pyhd8ed1ab_0 conda-forge tensorflow 2.9.1 cpu_py39h2839aeb_0 conda-forge tensorflow-base 2.9.1 cpu_py39ha1ad4ae_0 conda-forge tensorflow-estimator 2.9.1 cpu_py39h7b621ec_0 conda-forge tensorflow-hub 0.12.0 pyhca92ed8_0 conda-forge tensorflow-macos 2.9.2 pypi_0 pypi tensorflow-metal 0.5.0 pypi_0 pypi termcolor 2.3.0 pyhd8ed1ab_0 conda-forge threadpoolctl 3.2.0 pyha21a80b_0 conda-forge tifffile 2022.10.10 pyhd8ed1ab_0 conda-forge tk 8.6.13 hb31c410_0 conda-forge typing-extensions 4.8.0 hd8ed1ab_0 conda-forge typing_extensions 4.8.0 pyha770c72_0 conda-forge tzdata 2023c h71feb2d_0 conda-forge tzlocal 5.0.1 pypi_0 pypi unicodedata2 15.1.0 py39h0f82c59_0 conda-forge urllib3 1.26.16 pypi_0 pypi werkzeug 2.3.7 pypi_0 pypi wheel 0.41.3 pyhd8ed1ab_0 conda-forge wrapt 1.15.0 py39h0f82c59_1 conda-forge x264 1!164.3095 h57fd34a_2 conda-forge x265 3.5 hbc6ce65_3 conda-forge xorg-libxau 1.0.11 hb547adb_0 conda-forge xorg-libxdmcp 1.1.3 h27ca646_0 conda-forge xz 5.2.6 h57fd34a_0 conda-forge yaml 0.2.5 h3422bc3_2 conda-forge yarl 1.9.2 py39h17cfd9d_1 conda-forge zeromq 4.3.5 h965bd2d_0 conda-forge zfp 1.0.0 h82938aa_4 conda-forge zipp 3.17.0 pyhd8ed1ab_0 conda-forge zlib 1.2.13 h53f4e23_5 conda-forge zlib-ng 2.0.7 h1a8c8d9_0 conda-forge zstd 1.5.5 h4f39d0f_0 conda-forge ```
Logs ``` # paste relevant logs here, if any ```

Screenshots

Screenshot 2023-11-16 at 3 50 02 PM

How to reproduce

Run training from GUI via Predict -> Run Training... -> Run using single animal pipeline

talmo commented 11 months ago

Hi @hl8536,

Apologies for the delay! It looks like this is a bug related to Mac OS Sonoma unfortunately breaking the TensorFlow 2.9 support.

We'll investigate, but unfortunately this might take a little while since it'll require a rework of our dependencies.

In the meantime, you could try using Google Colab or downgrading to Mac OS Ventura if possible.

If you want to try fixing it yourself, the thing to try would be to install from source and tweak the versions in requirements.txt:

https://github.com/talmolab/sleap/blob/2d24296c2149402c7ee07601e4d0272fdd308441/requirements.txt#L9-L10

And possibly related dependencies in environment_mac.yml.

Sorry about that! Let us know if you have any questions!

Talmo

mjamagon commented 10 months ago

@talmo setting tensorflow-macos==2.8.0 worked for me when installing from source (apple silicon + Sonoma)

talmo commented 7 months ago

We are now using macOS Sonoma + tensorflow-macos==2.9.2 in testing as of #1734.

So far, so good, but it's not clear if this specific bug would be caught there. This bug might still come up in regular use, but we should have added some fixes to how it's installed which will change the environment relative to the reported one here (namely, not pulling in conda-forge tensorflow). These fixes will be available in SLEAP v1.4.0.

I'm closing for now, but if needed, it sounds like tensorflow-macos=2.8.0 might be a workaround.

Related issue: #1547