search

talmolab / sleap

A deep learning framework for multi-animal pose tracking.
https://sleap.ai
Other
425 stars 97 forks source link

Crashing during inference on Apple M2 Max #1647

Open uklibaite opened 8 months ago

uklibaite commented 8 months ago

sleap crashes during inference of entire video (even short video) but runs inference on a small number of suggested frames. Samples during training and these small inference batches look good. Inference starts and the count of inferred frames increases (then sometimes the speed of inference appears to double shortly before a crash). The popup says there is probably more info in the command line (posted below).

Expected behaviour

Inference running normally (oddly it did run once all the way through a movie that was 180000 frames). If relevant - this happens both when tracking a simple movie with a single animal or when tracking about 5 individuals. Training runs normally.

Actual behaviour

Command line outputs for training and inference below.

Your personal set up

I installed using the mamba instructions for apple silicon (M2 Max). I use conda activate sleap to load the environment.

Environment packages ``` conda list # packages in environment at /Users/ugne/mambaforge3/envs/sleap: # # Name Version Build Channel abseil-cpp 20211102.0 he4e09e4_3 conda-forge absl-py 1.4.0 pypi_0 pypi aiohttp 3.9.1 py39h17cfd9d_0 conda-forge aiosignal 1.3.1 pyhd8ed1ab_0 conda-forge aom 3.5.0 h7ea286d_0 conda-forge astunparse 1.6.3 pyhd8ed1ab_0 conda-forge async-timeout 4.0.3 pyhd8ed1ab_0 conda-forge attrs 23.1.0 pyh71513ae_1 conda-forge blinker 1.7.0 pyhd8ed1ab_0 conda-forge blosc 1.21.5 hc338f07_0 conda-forge brotli 1.0.9 h1a8c8d9_9 conda-forge brotli-bin 1.0.9 h1a8c8d9_9 conda-forge brotli-python 1.0.9 py39h23fbdae_9 conda-forge brunsli 0.1 h9f76cd9_0 conda-forge bzip2 1.0.8 h93a5062_5 conda-forge c-ares 1.24.0 h93a5062_0 conda-forge c-blosc2 2.11.3 h8eb3132_0 conda-forge ca-certificates 2023.11.17 hf0a4a13_0 conda-forge cached-property 1.5.2 hd8ed1ab_1 conda-forge cached_property 1.5.2 pyha770c72_1 conda-forge cachetools 5.3.1 pypi_0 pypi cairo 1.16.0 h73a0509_1014 conda-forge cattrs 1.1.1 pyhd8ed1ab_0 conda-forge certifi 2023.11.17 pyhd8ed1ab_0 conda-forge cffi 1.16.0 py39he153c15_0 conda-forge cfitsio 4.2.0 h2f961c4_0 conda-forge charls 2.3.4 hbdafb3b_0 conda-forge charset-normalizer 3.2.0 pypi_0 pypi click 8.1.7 unix_pyh707e725_0 conda-forge contourpy 1.2.0 py39he9de807_0 conda-forge cryptography 39.0.0 py39haa0b8cc_0 conda-forge cycler 0.12.1 pyhd8ed1ab_0 conda-forge dav1d 1.2.1 hb547adb_0 conda-forge efficientnet 1.0.0 pypi_0 pypi expat 2.5.0 hb7217d7_1 conda-forge ffmpeg 4.4.2 gpl_hf318d42_112 conda-forge font-ttf-dejavu-sans-mono 2.37 hab24e00_0 conda-forge font-ttf-inconsolata 3.000 h77eed37_0 conda-forge font-ttf-source-code-pro 2.038 h77eed37_0 conda-forge font-ttf-ubuntu 0.83 h77eed37_1 conda-forge fontconfig 2.14.2 h82840c6_0 conda-forge fonts-conda-ecosystem 1 0 conda-forge fonts-conda-forge 1 0 conda-forge fonttools 4.47.0 py39h17cfd9d_0 conda-forge freetype 2.12.1 hadb7bae_2 conda-forge frozenlist 1.4.1 py39h17cfd9d_0 conda-forge gast 0.4.0 pyh9f0ad1d_0 conda-forge geos 3.12.1 h965bd2d_0 conda-forge gettext 0.21.1 h0186832_0 conda-forge giflib 5.2.1 h1a8c8d9_3 conda-forge glib 2.78.3 h9e231a4_0 conda-forge glib-tools 2.78.3 h9e231a4_0 conda-forge gmp 6.3.0 h965bd2d_0 conda-forge gnutls 3.7.9 hd26332c_0 conda-forge google-auth 2.23.0 pypi_0 pypi google-auth-oauthlib 0.4.6 pyhd8ed1ab_0 conda-forge google-pasta 0.2.0 pyh8c360ce_0 conda-forge graphite2 1.3.13 h9f76cd9_1001 conda-forge grpc-cpp 1.46.4 hcaf9be7_3 conda-forge grpcio 1.58.0 pypi_0 pypi gst-plugins-base 1.22.8 h09b4b5e_0 conda-forge gstreamer 1.22.8 h551c6ff_0 conda-forge h5py 3.8.0 nompi_py39hc9149d8_100 conda-forge harfbuzz 5.3.0 hddbc195_0 conda-forge hdf5 1.12.2 nompi_h55deafc_101 conda-forge hdmf 3.9.0 pypi_0 pypi icu 70.1 h6b3803e_0 conda-forge idna 3.4 pypi_0 pypi image-classifiers 1.0.0 pypi_0 pypi imagecodecs 2022.9.26 py39hd7f743f_4 conda-forge imageio 2.33.1 pyh8c1a49c_0 conda-forge imgaug 0.4.0 pyhd8ed1ab_1 conda-forge imgstore 0.2.9 pypi_0 pypi importlib-metadata 7.0.0 pyha770c72_0 conda-forge importlib-resources 6.1.1 pyhd8ed1ab_0 conda-forge importlib_resources 6.1.1 pyhd8ed1ab_0 conda-forge jasper 2.0.33 hc3cd1e9_1 conda-forge joblib 1.3.2 pyhd8ed1ab_0 conda-forge jpeg 9e h1a8c8d9_3 conda-forge jsmin 3.0.1 pyhd8ed1ab_0 conda-forge jsonpickle 1.2 py_0 conda-forge jsonschema 4.19.0 pypi_0 pypi jsonschema-specifications 2023.7.1 pypi_0 pypi jxrlib 1.1 h27ca646_2 conda-forge keras 2.9.0 pyhd8ed1ab_0 conda-forge keras-applications 1.0.8 pypi_0 pypi keras-preprocessing 1.1.2 pyhd8ed1ab_0 conda-forge kiwisolver 1.4.5 py39hbd775c9_1 conda-forge krb5 1.20.1 h127bd45_0 conda-forge lame 3.100 h1a8c8d9_1003 conda-forge lazy_loader 0.3 pyhd8ed1ab_0 conda-forge lcms2 2.14 h8193b64_0 conda-forge lerc 4.0.0 h9a09cb3_0 conda-forge libabseil 20211102.0 cxx17_h28b99d4_3 conda-forge libaec 1.1.2 h13dd4ca_1 conda-forge libavif 0.11.1 h9f83d30_2 conda-forge libblas 3.9.0 20_osxarm64_openblas conda-forge libbrotlicommon 1.0.9 h1a8c8d9_9 conda-forge libbrotlidec 1.0.9 h1a8c8d9_9 conda-forge libbrotlienc 1.0.9 h1a8c8d9_9 conda-forge libcblas 3.9.0 20_osxarm64_openblas conda-forge libclang 16.0.6 pypi_0 pypi libclang13 14.0.6 default_hc7183e1_1 conda-forge libcurl 7.87.0 hbe9bab4_0 conda-forge libcxx 16.0.6 h4653b0c_0 conda-forge libdeflate 1.14 h1a8c8d9_0 conda-forge libedit 3.1.20191231 hc8eb9b7_2 conda-forge libev 4.33 h93a5062_2 conda-forge libexpat 2.5.0 hb7217d7_1 conda-forge libffi 3.4.2 h3422bc3_5 conda-forge libgfortran 5.0.0 13_2_0_hd922786_1 conda-forge libgfortran5 13.2.0 hf226fd6_1 conda-forge libglib 2.78.3 hb438215_0 conda-forge libiconv 1.17 h0d3ecfb_2 conda-forge libidn2 2.3.4 h1a8c8d9_0 conda-forge liblapack 3.9.0 20_osxarm64_openblas conda-forge liblapacke 3.9.0 20_osxarm64_openblas conda-forge libllvm14 14.0.6 hd1a9a77_4 conda-forge libnghttp2 1.51.0 hd184df1_0 conda-forge libogg 1.3.4 h27ca646_1 conda-forge libopenblas 0.3.25 openmp_h6c19121_0 conda-forge libopencv 4.6.0 py39he1c1adf_3 conda-forge libopus 1.3.1 h27ca646_1 conda-forge libpng 1.6.39 h76d750c_0 conda-forge libpq 15.1 hbce9e56_3 conda-forge libprotobuf 3.20.3 hb5ab8b9_0 conda-forge libsodium 1.0.18 h27ca646_1 conda-forge libsqlite 3.44.2 h091b4b1_0 conda-forge libssh2 1.10.0 hb80f160_3 conda-forge libtasn1 4.19.0 h1a8c8d9_0 conda-forge libtiff 4.4.0 heb92581_5 conda-forge libunistring 0.9.10 h3422bc3_0 conda-forge libvorbis 1.3.7 h9f76cd9_0 conda-forge libvpx 1.11.0 hc470f4d_3 conda-forge libwebp-base 1.3.2 hb547adb_0 conda-forge libxcb 1.13 h9b22ae9_1004 conda-forge libxml2 2.10.3 h67585b2_4 conda-forge libxslt 1.1.37 h1bd8bc4_0 conda-forge libzlib 1.2.13 h53f4e23_5 conda-forge libzopfli 1.0.3 h9f76cd9_0 conda-forge llvm-openmp 17.0.6 hcd81f8e_0 conda-forge lz4-c 1.9.4 hb7217d7_0 conda-forge markdown 3.4.4 pypi_0 pypi markdown-it-py 3.0.0 pyhd8ed1ab_0 conda-forge markupsafe 2.1.3 py39h0f82c59_1 conda-forge matplotlib-base 3.8.2 py39h1a09f3e_0 conda-forge mdurl 0.1.0 pyhd8ed1ab_0 conda-forge multidict 6.0.4 py39h02fc5c5_1 conda-forge munkres 1.1.4 pyh9f0ad1d_0 conda-forge mysql-common 8.0.32 hab468bb_0 conda-forge mysql-libs 8.0.32 hea58576_0 conda-forge ncurses 6.4 h463b476_2 conda-forge ndx-pose 0.1.1 pypi_0 pypi nettle 3.9.1 h40ed0f5_0 conda-forge networkx 3.2.1 pyhd8ed1ab_0 conda-forge nixio 1.5.3 pypi_0 pypi nspr 4.35 hb7217d7_0 conda-forge nss 3.96 h5ce2875_0 conda-forge numpy 1.22.4 py39h7df2422_0 conda-forge oauthlib 3.2.2 pyhd8ed1ab_0 conda-forge opencv 4.6.0 py39hdf13c20_3 conda-forge openh264 2.3.1 hb7217d7_2 conda-forge openjpeg 2.5.0 h5d4e404_1 conda-forge openssl 1.1.1w h53f4e23_0 conda-forge opt_einsum 3.3.0 pyhc1e730c_2 conda-forge p11-kit 0.24.1 h29577a5_0 conda-forge packaging 23.2 pyhd8ed1ab_0 conda-forge pandas 2.1.4 py39hf8cecc8_0 conda-forge patsy 0.5.4 pyhd8ed1ab_0 conda-forge pcre2 10.42 h26f9a81_0 conda-forge pillow 9.2.0 py39h139752e_3 conda-forge pip 23.3.2 pyhd8ed1ab_0 conda-forge pixman 0.42.2 h13dd4ca_0 conda-forge protobuf 3.19.6 pypi_0 pypi psutil 5.9.7 py39h17cfd9d_0 conda-forge pthread-stubs 0.4 h27ca646_1001 conda-forge py-opencv 4.6.0 py39hfa6204d_3 conda-forge pyasn1 0.5.0 pypi_0 pypi pyasn1-modules 0.3.0 pyhd8ed1ab_0 conda-forge pycparser 2.21 pyhd8ed1ab_0 conda-forge pygments 2.17.2 pyhd8ed1ab_0 conda-forge pyjwt 2.8.0 pyhd8ed1ab_0 conda-forge pykalman 0.9.5 py_1 conda-forge pynwb 2.5.0 pypi_0 pypi pyopenssl 23.2.0 pyhd8ed1ab_1 conda-forge pyparsing 3.1.1 pyhd8ed1ab_0 conda-forge pyside2 5.15.8 py39h0adaba8_2 conda-forge pysocks 1.7.1 pyha2e5f31_6 conda-forge python 3.9.15 h2d96c93_0_cpython conda-forge python-dateutil 2.8.2 pyhd8ed1ab_0 conda-forge python-flatbuffers 1.12 pyhd8ed1ab_1 conda-forge python-rapidjson 1.14 py39hf3050f2_0 conda-forge python-tzdata 2023.3 pyhd8ed1ab_0 conda-forge python_abi 3.9 4_cp39 conda-forge pytz 2023.3.post1 pyhd8ed1ab_0 conda-forge pyu2f 0.1.5 pyhd8ed1ab_0 conda-forge pywavelets 1.4.1 py39hf4a74a7_1 conda-forge pyyaml 6.0.1 py39h0f82c59_1 conda-forge pyzmq 25.1.2 py39he1e2164_0 conda-forge qimage2ndarray 1.10.0 pypi_0 pypi qt-main 5.15.8 hfe8d25c_6 conda-forge qtpy 2.4.1 pyhd8ed1ab_0 conda-forge re2 2022.06.01 h9a09cb3_1 conda-forge readline 8.2 h92ec313_1 conda-forge referencing 0.30.2 pypi_0 pypi requests 2.31.0 pyhd8ed1ab_0 conda-forge requests-oauthlib 1.3.1 pyhd8ed1ab_0 conda-forge rich 13.7.0 pyhd8ed1ab_0 conda-forge rpds-py 0.10.3 pypi_0 pypi rsa 4.9 pyhd8ed1ab_0 conda-forge ruamel-yaml 0.17.32 pypi_0 pypi ruamel-yaml-clib 0.2.7 pypi_0 pypi scikit-image 0.22.0 py39hf8cecc8_2 conda-forge scikit-learn 1.0 py39h12ba089_1 conda-forge scikit-video 1.1.11 pyh24bf2e0_0 conda-forge scipy 1.9.0 py39h14896cb_0 conda-forge seaborn 0.13.0 hd8ed1ab_0 conda-forge seaborn-base 0.13.0 pyhd8ed1ab_0 conda-forge segmentation-models 1.0.1 pypi_0 pypi setuptools 68.2.2 pyhd8ed1ab_0 conda-forge shapely 2.0.2 py39ha70ab96_1 conda-forge six 1.15.0 pypi_0 pypi sleap 1.3.3 pypi_0 pypi snappy 1.1.10 h17c5cce_0 conda-forge sqlite 3.44.2 hf2abe2d_0 conda-forge statsmodels 0.14.1 py39h373d45f_0 conda-forge svt-av1 1.4.1 h7ea286d_0 conda-forge tensorboard 2.9.1 pypi_0 pypi tensorboard-data-server 0.6.1 py39haa0b8cc_4 conda-forge tensorboard-plugin-wit 1.8.1 pyhd8ed1ab_0 conda-forge tensorflow 2.9.1 cpu_py39h2839aeb_0 conda-forge tensorflow-base 2.9.1 cpu_py39ha1ad4ae_0 conda-forge tensorflow-estimator 2.9.1 cpu_py39h7b621ec_0 conda-forge tensorflow-hub 0.12.0 pyhca92ed8_0 conda-forge tensorflow-macos 2.9.2 pypi_0 pypi tensorflow-metal 0.5.0 pypi_0 pypi termcolor 2.3.0 pyhd8ed1ab_0 conda-forge threadpoolctl 3.2.0 pyha21a80b_0 conda-forge tifffile 2022.10.10 pyhd8ed1ab_0 conda-forge tk 8.6.13 h5083fa2_1 conda-forge typing-extensions 4.9.0 hd8ed1ab_0 conda-forge typing_extensions 4.9.0 pyha770c72_0 conda-forge tzdata 2023c h71feb2d_0 conda-forge tzlocal 5.0.1 pypi_0 pypi unicodedata2 15.1.0 py39h0f82c59_0 conda-forge urllib3 1.26.16 pypi_0 pypi werkzeug 2.3.7 pypi_0 pypi wheel 0.42.0 pyhd8ed1ab_0 conda-forge wrapt 1.15.0 pypi_0 pypi x264 1!164.3095 h57fd34a_2 conda-forge x265 3.5 hbc6ce65_3 conda-forge xorg-libxau 1.0.11 hb547adb_0 conda-forge xorg-libxdmcp 1.1.3 h27ca646_0 conda-forge xz 5.2.6 h57fd34a_0 conda-forge yaml 0.2.5 h3422bc3_2 conda-forge yarl 1.9.3 py39h17cfd9d_0 conda-forge zeromq 4.3.5 h965bd2d_0 conda-forge zfp 1.0.1 ha8f4885_0 conda-forge zipp 3.17.0 pyhd8ed1ab_0 conda-forge zlib 1.2.13 h53f4e23_5 conda-forge zlib-ng 2.0.7 h1a8c8d9_0 conda-forge zstd 1.5.5 h4f39d0f_0 conda-forge ```
Logs ``` Command line call: sleap-track /Users/ugne/Dropbox/pregnant/labels.v001_pupx1.slp --only-suggested-frames -m /Users/ugne/Dropbox/pregnant/models/231223_213850.centroid.n=8 -m /Users/ugne/Dropbox/pregnant/models/231223_214910.centered_instance.n=8 --max_instances 1 -o /Users/ugne/Dropbox/pregnant/predictions/labels.v001_pupx1.slp.231223_220359.predictions.slp --verbosity json --no-empty-frames Started inference at: 2023-12-23 22:04:03.074597 Args: { │ 'data_path': '/Users/ugne/Dropbox/pregnant/labels.v001_pupx1.slp', │ 'models': [ │ │ '/Users/ugne/Dropbox/pregnant/models/231223_213850.centroid.n=8', │ │ '/Users/ugne/Dropbox/pregnant/models/231223_214910.centered_instance.n=8' │ ], │ 'frames': '', │ 'only_labeled_frames': False, │ 'only_suggested_frames': True, │ 'output': '/Users/ugne/Dropbox/pregnant/predictions/labels.v001_pupx1.slp.231223_220359.predictions.slp', 2023-12-23 22:04:03.684157: I tensorflow/core/common_runtime/pluggable_device/pluggable_device_factory.cc:305] Could not identify NUMA node of platform GPU ID 0, defaulting to 0. Your kernel may not have been built with NUMA support. 2023-12-23 22:04:03.684311: I tensorflow/core/common_runtime/pluggable_device/pluggable_device_factory.cc:271] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 0 MB memory) -> physical PluggableDevice (device: 0, name: METAL, pci bus id: ) │ 'no_empty_frames': True, │ 'verbosity': 'json', │ 'video.dataset': None, │ 'video.input_format': 'channels_last', │ 'video.index': '', │ 'cpu': False, │ 'first_gpu': False, │ 'last_gpu': False, │ 'gpu': 'auto', │ 'max_edge_length_ratio': 0.25, │ 'dist_penalty_weight': 1.0, │ 'batch_size': 4, │ 'open_in_gui': False, │ 'peak_threshold': 0.2, │ 'max_instances': 1, │ 'tracking.tracker': None, │ 'tracking.max_tracking': None, │ 'tracking.max_tracks': None, 2023-12-23 22:04:04.592476: W tensorflow/core/platform/profile_utils/cpu_utils.cc:128] Failed to get CPU frequency: 0 Hz │ 'tracking.target_instance_count': None, │ 'tracking.pre_cull_to_target': None, │ 'tracking.pre_cull_iou_threshold': None, │ 'tracking.post_connect_single_breaks': None, │ 'tracking.clean_instance_count': None, │ 'tracking.clean_iou_threshold': None, │ 'tracking.similarity': None, │ 'tracking.match': None, │ 'tracking.robust': None, │ 'tracking.track_window': None, │ 'tracking.min_new_track_points': None, │ 'tracking.min_match_points': None, │ 'tracking.img_scale': None, │ 'tracking.of_window_size': None, │ 'tracking.of_max_levels': None, │ 'tracking.save_shifted_instances': None, │ 'tracking.kf_node_indices': None, │ 'tracking.kf_init_frame_count': None } INFO:sleap.nn.inference:Failed to query GPU memory from nvidia-smi. Defaulting to first GPU. Metal device set to: Apple M2 Max 2023-12-23 22:04:06.796696: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:113] Plugin optimizer for device_type GPU is enabled. 2023-12-23 22:04:06.874764: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: -47 } dim { size: -48 } dim { size: -49 } dim { size: 1 } } } inputs { dtype: DT_FLOAT shape { dim { size: -15 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -15 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 12 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -15 } dim { size: -50 } dim { size: -51 } dim { size: 1 } } } 2023-12-23 22:04:06.875581: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_UINT8 } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_UINT8 shape { dim { size: 4 } dim { size: 1200 } dim { size: 1920 } dim { size: 3 } } } inputs { dtype: DT_FLOAT shape { dim { size: -16 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -16 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 12 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -16 } dim { size: -63 } dim { size: -64 } dim { size: 3 } } } 2023-12-23 22:04:06.878681: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: -131 } dim { size: -132 } dim { size: -133 } dim { size: 1 } } } inputs { dtype: DT_FLOAT shape { dim { size: -22 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -22 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 12 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -22 } dim { size: -135 } dim { size: -136 } dim { size: 1 } } } loc("mps_select"("(mpsFileLoc): /AppleInternal/Library/BuildRoots/0032d1ee-80fd-11ee-8227-6aecfccc70fe/Library/Caches/com.apple.xbs/Sources/MetalPerformanceShadersGraph/mpsgraph/MetalPerformanceShadersGraph/Core/Files/MPSGraphUtilities.mm":294:0)): error: 'anec.gain_offset_control' op result #0 must be 4D/5D memref of 16-bit float or 8-bit signed integer or 8-bit unsigned integer values, but got 'memref<4x3x1x1xi1>' Versions: SLEAP: 1.3.3 TensorFlow: 2.9.2 Numpy: 1.22.4 Python: 3.9.15 OS: macOS-14.2.1-arm64-arm-64bit System: GPUs: 1/1 available Device: /physical_device:GPU:0 Available: True 2023-12-23 22:04:08.017111: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:113] Plugin optimizer for device_type GPU is enabled. Initalized: False Memory growth: True 2023-12-23 22:04:08.096753: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: -58 } dim { size: -59 } dim { size: -60 } dim { size: 1 } } } inputs { dtype: DT_FLOAT shape { dim { size: -15 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -15 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 12 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -15 } dim { size: -61 } dim { size: -62 } dim { size: 1 } } } 2023-12-23 22:04:08.097681: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_UINT8 } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_UINT8 shape { dim { size: -25 } dim { size: 1200 } dim { size: 1920 } dim { size: 3 } } } inputs { dtype: DT_FLOAT shape { dim { size: -16 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -16 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 12 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -16 } dim { size: -74 } dim { size: -75 } dim { size: 3 } } } 2023-12-23 22:04:08.100799: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: -142 } dim { size: -143 } dim { size: -144 } dim { size: 1 } } } inputs { dtype: DT_FLOAT shape { dim { size: -22 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -22 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 12 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -22 } dim { size: -146 } dim { size: -147 } dim { size: 1 } } } loc("mps_select"("(mpsFileLoc): /AppleInternal/Library/BuildRoots/0032d1ee-80fd-11ee-8227-6aecfccc70fe/Library/Caches/com.apple.xbs/Sources/MetalPerformanceShadersGraph/mpsgraph/MetalPerformanceShadersGraph/Core/Files/MPSGraphUtilities.mm":294:0)): error: 'anec.gain_offset_control' op result #0 must be 4D/5D memref of 16-bit float or 8-bit signed integer or 8-bit unsigned integer values, but got 'memref<1x3x1x1xi1>' Finished inference at: 2023-12-23 22:04:08.754709 Total runtime: 5.680121898651123 secs Predicted frames: 13/13 Provenance: { │ 'model_paths': [ │ │ '/Users/ugne/Dropbox/pregnant/models/231223_213850.centroid.n=8/training_config.json', Process return code: 0 Using already trained model for centroid: /Users/ugne/Dropbox/pregnant/models/231223_213850.centroid.n=8/training_config.json Using already trained model for centered_instance: /Users/ugne/Dropbox/pregnant/models/231223_214910.centered_instance.n=8/training_config.json Command line call: sleap-track /Users/ugne/Dropbox/pregnant/labels.v001_pupx1.slp --video.index 0 --frames 0,-14999 -m /Users/ugne/Dropbox/pregnant/models/231223_213850.centroid.n=8/training_config.json -m /Users/ugne/Dropbox/pregnant/models/231223_214910.centered_instance.n=8/training_config.json --tracking.tracker none --max_instances 1 -o /Users/ugne/Dropbox/pregnant/predictions/labels.v001_pupx1.slp.231223_220450.predictions.slp --verbosity json --no-empty-frames Started inference at: 2023-12-23 22:04:54.576367 Args: { │ 'data_path': '/Users/ugne/Dropbox/pregnant/labels.v001_pupx1.slp', │ 'models': [ │ │ '/Users/ugne/Dropbox/pregnant/models/231223_213850.centroid.n=8/training_config.json', │ │ '/Users/ugne/Dropbox/pregnant/models/231223_214910.centered_instance.n=8/training_config.json' │ ], │ 'frames': '0,-14999', │ 'only_labeled_frames': False, │ 'only_suggested_frames': False, │ 'output': '/Users/ugne/Dropbox/pregnant/predictions/labels.v001_pupx1.slp.231223_220450.predictions.slp', 2023-12-23 22:04:55.185517: I tensorflow/core/common_runtime/pluggable_device/pluggable_device_factory.cc:305] Could not identify NUMA node of platform GPU ID 0, defaulting to 0. Your kernel may not have been built with NUMA support. 2023-12-23 22:04:55.185676: I tensorflow/core/common_runtime/pluggable_device/pluggable_device_factory.cc:271] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 0 MB memory) -> physical PluggableDevice (device: 0, name: METAL, pci bus id: ) │ 'no_empty_frames': True, │ 'verbosity': 'json', │ 'video.dataset': None, │ 'video.input_format': 'channels_last', │ 'video.index': '0', │ 'cpu': False, │ 'first_gpu': False, │ 'last_gpu': False, │ 'gpu': 'auto', │ 'max_edge_length_ratio': 0.25, │ 'dist_penalty_weight': 1.0, │ 'batch_size': 4, │ 'open_in_gui': False, │ 'peak_threshold': 0.2, │ 'max_instances': 1, │ 'tracking.tracker': 'none', │ 'tracking.max_tracking': None, │ 'tracking.max_tracks': None, 2023-12-23 22:04:56.094568: W tensorflow/core/platform/profile_utils/cpu_utils.cc:128] Failed to get CPU frequency: 0 Hz │ 'tracking.target_instance_count': None, │ 'tracking.pre_cull_to_target': None, │ 'tracking.pre_cull_iou_threshold': None, │ 'tracking.post_connect_single_breaks': None, │ 'tracking.clean_instance_count': None, │ 'tracking.clean_iou_threshold': None, │ 'tracking.similarity': None, │ 'tracking.match': None, │ 'tracking.robust': None, │ 'tracking.track_window': None, │ 'tracking.min_new_track_points': None, │ 'tracking.min_match_points': None, │ 'tracking.img_scale': None, │ 'tracking.of_window_size': None, │ 'tracking.of_max_levels': None, │ 'tracking.save_shifted_instances': None, │ 'tracking.kf_node_indices': None, │ 'tracking.kf_init_frame_count': None } INFO:sleap.nn.inference:Failed to query GPU memory from nvidia-smi. Defaulting to first GPU. Metal device set to: Apple M2 Max 2023-12-23 22:04:57.975830: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:113] Plugin optimizer for device_type GPU is enabled. 2023-12-23 22:04:58.054578: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: -47 } dim { size: -48 } dim { size: -49 } dim { size: 1 } } } inputs { dtype: DT_FLOAT shape { dim { size: -15 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -15 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 12 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -15 } dim { size: -50 } dim { size: -51 } dim { size: 1 } } } 2023-12-23 22:04:58.055451: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_UINT8 } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_UINT8 shape { dim { size: 4 } dim { size: 1200 } dim { size: 1920 } dim { size: 3 } } } inputs { dtype: DT_FLOAT shape { dim { size: -16 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -16 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 12 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -16 } dim { size: -63 } dim { size: -64 } dim { size: 3 } } } 2023-12-23 22:04:58.058697: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: -131 } dim { size: -132 } dim { size: -133 } dim { size: 1 } } } inputs { dtype: DT_FLOAT shape { dim { size: -22 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -22 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 12 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -22 } dim { size: -135 } dim { size: -136 } dim { size: 1 } } } loc("mps_select"("(mpsFileLoc): /AppleInternal/Library/BuildRoots/0032d1ee-80fd-11ee-8227-6aecfccc70fe/Library/Caches/com.apple.xbs/Sources/MetalPerformanceShadersGraph/mpsgraph/MetalPerformanceShadersGraph/Core/Files/MPSGraphUtilities.mm":294:0)): error: 'anec.gain_offset_control' op result #0 must be 4D/5D memref of 16-bit float or 8-bit signed integer or 8-bit unsigned integer values, but got 'memref<4x3x1x1xi1>' Versions: SLEAP: 1.3.3 TensorFlow: 2.9.2 Numpy: 1.22.4 Python: 3.9.15 OS: macOS-14.2.1-arm64-arm-64bit System: GPUs: 1/1 available Device: /physical_device:GPU:0 Available: True Initalized: False Memory growth: True Traceback (most recent call last): File "/Users/ugne/mambaforge3/envs/sleap/bin/sleap-track", line 33, in sys.exit(load_entry_point('sleap==1.3.3', 'console_scripts', 'sleap-track')()) File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/sleap/nn/inference.py", line 5424, in main labels_pr = predictor.predict(provider) File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/sleap/nn/inference.py", line 526, in predict self._make_labeled_frames_from_generator(generator, data) File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/sleap/nn/inference.py", line 2633, in _make_labeled_frames_from_generator for ex in generator: File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/sleap/nn/inference.py", line 457, in _predict_generator ex = process_batch(ex) File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/sleap/nn/inference.py", line 399, in process_batch preds = self.inference_model.predict_on_batch(ex, numpy=True) File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/sleap/nn/inference.py", line 1069, in predict_on_batch outs = super().predict_on_batch(data, **kwargs) File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/keras/engine/training.py", line 2230, in predict_on_batch outputs = self.predict_function(iterator) File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/tensorflow/python/util/traceback_utils.py", line 153, in error_handler raise e.with_traceback(filtered_tb) from None File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/tensorflow/python/eager/execute.py", line 54, in quick_execute tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name, tensorflow.python.framework.errors_impl.InternalError: Graph execution error: Detected at node 'top_down_inference_model/find_instance_peaks/model/stack0_dec0_s16_to_s8_interp_bilinear/resize/ResizeBilinear' defined at (most recent call last): File "/Users/ugne/mambaforge3/envs/sleap/bin/sleap-track", line 33, in sys.exit(load_entry_point('sleap==1.3.3', 'console_scripts', 'sleap-track')()) File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/sleap/nn/inference.py", line 5424, in main labels_pr = predictor.predict(provider) File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/sleap/nn/inference.py", line 526, in predict self._make_labeled_frames_from_generator(generator, data) File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/sleap/nn/inference.py", line 2633, in _make_labeled_frames_from_generator for ex in generator: File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/sleap/nn/inference.py", line 457, in _predict_generator ex = process_batch(ex) File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/sleap/nn/inference.py", line 399, in process_batch preds = self.inference_model.predict_on_batch(ex, numpy=True) File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/sleap/nn/inference.py", line 1069, in predict_on_batch outs = super().predict_on_batch(data, **kwargs) File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/keras/engine/training.py", line 2230, in predict_on_batch outputs = self.predict_function(iterator) File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/keras/engine/training.py", line 1845, in predict_function return step_function(self, iterator) File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/keras/engine/training.py", line 1834, in step_function outputs = model.distribute_strategy.run(run_step, args=(data,)) File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/keras/engine/training.py", line 1823, in run_step outputs = model.predict_step(data) File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/keras/engine/training.py", line 1791, in predict_step return self(x, training=False) File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler return fn(*args, **kwargs) File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/keras/engine/training.py", line 490, in __call__ return super().__call__(*args, **kwargs) File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler return fn(*args, **kwargs) File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/keras/engine/base_layer.py", line 1014, in __call__ outputs = call_fn(inputs, *args, **kwargs) File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler return fn(*args, **kwargs) File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/sleap/nn/inference.py", line 2256, in call if isinstance(self.instance_peaks, FindInstancePeaksGroundTruth): File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/sleap/nn/inference.py", line 2265, in call peaks_output = self.instance_peaks(crop_output) File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler return fn(*args, **kwargs) File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/keras/engine/base_layer.py", line 1014, in __call__ outputs = call_fn(inputs, *args, **kwargs) File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler return fn(*args, **kwargs) File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/sleap/nn/inference.py", line 2088, in call out = self.keras_model(crops) File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler return fn(*args, **kwargs) File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/keras/engine/training.py", line 490, in __call__ return super().__call__(*args, **kwargs) File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler return fn(*args, **kwargs) File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/keras/engine/base_layer.py", line 1014, in __call__ outputs = call_fn(inputs, *args, **kwargs) File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler return fn(*args, **kwargs) File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/keras/engine/functional.py", line 458, in call return self._run_internal_graph( File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/keras/engine/functional.py", line 596, in _run_internal_graph outputs = node.layer(*args, **kwargs) File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler return fn(*args, **kwargs) File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/keras/engine/base_layer.py", line 1014, in __call__ outputs = call_fn(inputs, *args, **kwargs) File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler return fn(*args, **kwargs) File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/keras/layers/reshaping/up_sampling2d.py", line 129, in call return backend.resize_images( File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/keras/backend.py", line 3432, in resize_images x = tf.image.resize(x, new_shape, method=interpolations[interpolation]) Node: 'top_down_inference_model/find_instance_peaks/model/stack0_dec0_s16_to_s8_interp_bilinear/resize/ResizeBilinear' Detected at node 'top_down_inference_model/find_instance_peaks/model/stack0_dec0_s16_to_s8_interp_bilinear/resize/ResizeBilinear' defined at (most recent call last): File "/Users/ugne/mambaforge3/envs/sleap/bin/sleap-track", line 33, in sys.exit(load_entry_point('sleap==1.3.3', 'console_scripts', 'sleap-track')()) File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/sleap/nn/inference.py", line 5424, in main labels_pr = predictor.predict(provider) File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/sleap/nn/inference.py", line 526, in predict self._make_labeled_frames_from_generator(generator, data) File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/sleap/nn/inference.py", line 2633, in _make_labeled_frames_from_generator for ex in generator: File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/sleap/nn/inference.py", line 457, in _predict_generator ex = process_batch(ex) File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/sleap/nn/inference.py", line 399, in process_batch preds = self.inference_model.predict_on_batch(ex, numpy=True) File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/sleap/nn/inference.py", line 1069, in predict_on_batch outs = super().predict_on_batch(data, **kwargs) File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/keras/engine/training.py", line 2230, in predict_on_batch outputs = self.predict_function(iterator) File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/keras/engine/training.py", line 1845, in predict_function return step_function(self, iterator) File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/keras/engine/training.py", line 1834, in step_function outputs = model.distribute_strategy.run(run_step, args=(data,)) File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/keras/engine/training.py", line 1823, in run_step outputs = model.predict_step(data) File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/keras/engine/training.py", line 1791, in predict_step return self(x, training=False) File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler return fn(*args, **kwargs) File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/keras/engine/training.py", line 490, in __call__ return super().__call__(*args, **kwargs) File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler return fn(*args, **kwargs) File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/keras/engine/base_layer.py", line 1014, in __call__ outputs = call_fn(inputs, *args, **kwargs) File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler return fn(*args, **kwargs) File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/sleap/nn/inference.py", line 2256, in call if isinstance(self.instance_peaks, FindInstancePeaksGroundTruth): File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/sleap/nn/inference.py", line 2265, in call peaks_output = self.instance_peaks(crop_output) File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler return fn(*args, **kwargs) File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/keras/engine/base_layer.py", line 1014, in __call__ outputs = call_fn(inputs, *args, **kwargs) File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler return fn(*args, **kwargs) File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/sleap/nn/inference.py", line 2088, in call out = self.keras_model(crops) File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler return fn(*args, **kwargs) File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/keras/engine/training.py", line 490, in __call__ return super().__call__(*args, **kwargs) File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler return fn(*args, **kwargs) File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/keras/engine/base_layer.py", line 1014, in __call__ outputs = call_fn(inputs, *args, **kwargs) File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler return fn(*args, **kwargs) File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/keras/engine/functional.py", line 458, in call return self._run_internal_graph( File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/keras/engine/functional.py", line 596, in _run_internal_graph outputs = node.layer(*args, **kwargs) File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler return fn(*args, **kwargs) File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/keras/engine/base_layer.py", line 1014, in __call__ outputs = call_fn(inputs, *args, **kwargs) File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler return fn(*args, **kwargs) File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/keras/layers/reshaping/up_sampling2d.py", line 129, in call return backend.resize_images( File "/Users/ugne/mambaforge3/envs/sleap/lib/python3.9/site-packages/keras/backend.py", line 3432, in resize_images x = tf.image.resize(x, new_shape, method=interpolations[interpolation]) Node: 'top_down_inference_model/find_instance_peaks/model/stack0_dec0_s16_to_s8_interp_bilinear/resize/ResizeBilinear' 2 root error(s) found. (0) INTERNAL: Missing 0-th output from {{node top_down_inference_model/find_instance_peaks/model/stack0_dec0_s16_to_s8_interp_bilinear/resize/ResizeBilinear}} [[top_down_inference_model/find_instance_peaks/PartitionedCall/cond/else/_82/cond/add_1/_402]] (1) INTERNAL: Missing 0-th output from {{node top_down_inference_model/find_instance_peaks/model/stack0_dec0_s16_to_s8_interp_bilinear/resize/ResizeBilinear}} 0 successful operations. 0 derived errors ignored. [Op:__inference_predict_function_4953] systemMemory: 64.00 GB maxCacheSize: 24.00 GB Process return code: 1 ```

Screenshots

Screenshot 2023-12-23 at 10 21 44 PM

How to reproduce

  1. Go to '...'
  2. Click on '....'
  3. Scroll down to '....'
  4. See error
talmo commented 8 months ago

Hi @uklibaite,

It looks like this is the main error:

Node: 'top_down_inference_model/find_instance_peaks/model/stack0_dec0_s16_to_s8_interp_bilinear/resize/ResizeBilinear'
2 root error(s) found.
  (0) INTERNAL:  Missing 0-th output from {{node top_down_inference_model/find_instance_peaks/model/stack0_dec0_s16_to_s8_interp_bilinear/resize/ResizeBilinear}}
     [[top_down_inference_model/find_instance_peaks/PartitionedCall/cond/else/_82/cond/add_1/_402]]
  (1) INTERNAL:  Missing 0-th output from {{node top_down_inference_model/find_instance_peaks/model/stack0_dec0_s16_to_s8_interp_bilinear/resize/ResizeBilinear}}

This is currently a bug in the Mac TensorFlow working differently than Windows/Linux in how it handles empty tensors.

We have an issue tracking this (https://github.com/talmolab/sleap/issues/1100#issuecomment-1756560300) and an in-progress PR to fix it (#1547).

We'll keep you updated once the fix is ready for testing!

Talmo

shiladityaneuron commented 2 months ago

Hi Following the post.

toniig commented 4 days ago

Hi! I am having the same error using a Appl M2 pro! Is there any solution? Thank you!!!

talmo commented 2 days ago

Hi folks,

Just to update with a quick comment: we will revisit this after the v1.4 release (one PR left!) which will also update TensorFlow versions and change the underlying behavior.

We'll take another pass at this after v1.4, but if the fix requires a big redesign of the TensorFlow backend (which is what present evidence points to), it might need to wait until we fully switch over to our new PyTorch backend in the coming months.

We'll keep everyone updated here and this issue pinned until either path is clear.

Sorry about that!

Talmo