When training on a Mac (using a .pkg.slp), the centroid Training Model plot of losses at each epoch is not updated past "Preparing to run training". However, the heat-map visualization is updated and the training seems to be running fine. Also strange is that the centered-instance Training Plot is updated as expected.
Bug description
When training on a Mac (using a .pkg.slp), the centroid Training Model plot of losses at each epoch is not updated past "Preparing to run training". However, the heat-map visualization is updated and the training seems to be running fine. Also strange is that the centered-instance Training Plot is updated as expected.
Expected behaviour
The Training Model plot is updated.
Your personal set up
micromamba list
``` List of packages in environment: "/Users/liezlmaree/micromamba/envs/sa0" Name Version Build Channel ──────────────────────────────────────────────────────────────────────────────── abseil-cpp 20211102.0 he4e09e4_3 conda-forge absl-py 2.0.0 pyhd8ed1ab_0 conda-forge aiohttp 3.8.5 py39h0f82c59_0 conda-forge aiosignal 1.3.1 pyhd8ed1ab_0 conda-forge aom 3.5.0 h7ea286d_0 conda-forge astunparse 1.6.3 pyhd8ed1ab_0 conda-forge async-timeout 4.0.3 pyhd8ed1ab_0 conda-forge attrs 23.1.0 pyh71513ae_1 conda-forge blas 2.118 openblas conda-forge blas-devel 3.9.0 18_osxarm64_openblas conda-forge blinker 1.6.2 pyhd8ed1ab_0 conda-forge blosc 1.21.5 hc338f07_0 conda-forge brotli 1.0.9 h1a8c8d9_9 conda-forge brotli-bin 1.0.9 h1a8c8d9_9 conda-forge brotli-python 1.0.9 py39h23fbdae_9 conda-forge brunsli 0.1 h9f76cd9_0 conda-forge bzip2 1.0.8 h3422bc3_4 conda-forge c-ares 1.19.1 hb547adb_0 conda-forge c-blosc2 2.10.2 h068da5f_0 conda-forge ca-certificates 2023.7.22 hf0a4a13_0 conda-forge cached-property 1.5.2 hd8ed1ab_1 conda-forge cached_property 1.5.2 pyha770c72_1 conda-forge cachetools 5.3.1 pyhd8ed1ab_0 conda-forge cairo 1.16.0 h73a0509_1014 conda-forge cattrs 1.1.1 pyhd8ed1ab_0 conda-forge certifi 2023.7.22 pyhd8ed1ab_0 conda-forge cffi 1.15.1 py39he153c15_5 conda-forge cfitsio 4.2.0 h2f961c4_0 conda-forge charls 2.3.4 hbdafb3b_0 conda-forge charset-normalizer 3.2.0 pyhd8ed1ab_0 conda-forge click 8.1.7 unix_pyh707e725_0 conda-forge cloudpickle 2.2.1 pyhd8ed1ab_0 conda-forge contourpy 1.1.1 py39hbd775c9_1 conda-forge cryptography 39.0.0 py39haa0b8cc_0 conda-forge cycler 0.11.0 pyhd8ed1ab_0 conda-forge cytoolz 0.12.2 py39h0f82c59_1 conda-forge dask-core 2023.9.2 pyhd8ed1ab_0 conda-forge dav1d 1.2.1 hb547adb_0 conda-forge expat 2.5.0 hb7217d7_1 conda-forge ffmpeg 4.4.2 gpl_hf318d42_112 conda-forge font-ttf-dejavu-sans-mono 2.37 hab24e00_0 conda-forge font-ttf-inconsolata 3.000 h77eed37_0 conda-forge font-ttf-source-code-pro 2.038 h77eed37_0 conda-forge font-ttf-ubuntu 0.83 hab24e00_0 conda-forge fontconfig 2.14.2 h82840c6_0 conda-forge fonts-conda-ecosystem 1 0 conda-forge fonts-conda-forge 1 0 conda-forge fonttools 4.42.1 py39h0f82c59_0 conda-forge freetype 2.12.1 hadb7bae_2 conda-forge frozenlist 1.4.0 py39h0f82c59_1 conda-forge fsspec 2023.9.2 pyh1a96a4e_0 conda-forge gast 0.4.0 pyh9f0ad1d_0 conda-forge geos 3.11.2 hb7217d7_0 conda-forge gettext 0.21.1 h0186832_0 conda-forge giflib 5.2.1 h1a8c8d9_3 conda-forge glib 2.78.0 ha614eb4_0 conda-forge glib-tools 2.78.0 ha614eb4_0 conda-forge gmp 6.2.1 h9f76cd9_0 conda-forge gnutls 3.7.8 h9f1a10d_0 conda-forge google-auth 2.23.0 pyh1a96a4e_1 conda-forge google-auth-oauthlib 0.4.6 pyhd8ed1ab_0 conda-forge google-pasta 0.2.0 pyh8c360ce_0 conda-forge graphite2 1.3.13 h9f76cd9_1001 conda-forge grpc-cpp 1.46.4 hcaf9be7_3 conda-forge grpcio 1.46.4 py39hb9dff5f_3 conda-forge gst-plugins-base 1.21.3 h8b7775e_1 conda-forge gstreamer 1.21.3 hcb7b3dd_1 conda-forge h5py 3.8.0 nompi_py39hc9149d8_100 conda-forge harfbuzz 5.3.0 hddbc195_0 conda-forge hdf5 1.12.2 nompi_h8968d4b_100 conda-forge icu 70.1 h6b3803e_0 conda-forge idna 3.4 pyhd8ed1ab_0 conda-forge imagecodecs 2022.9.26 py39hd7f743f_4 conda-forge imageio 2.31.1 pyh24c5eb1_0 conda-forge imgaug 0.4.0 pyhd8ed1ab_1 conda-forge importlib-metadata 6.8.0 pyha770c72_0 conda-forge importlib-resources 6.1.0 pyhd8ed1ab_0 conda-forge importlib_metadata 6.8.0 hd8ed1ab_0 conda-forge importlib_resources 6.1.0 pyhd8ed1ab_0 conda-forge jasper 2.0.33 hc3cd1e9_1 conda-forge joblib 1.3.2 pyhd8ed1ab_0 conda-forge jpeg 9e h1a8c8d9_3 conda-forge jsmin 3.0.1 pyhd8ed1ab_0 conda-forge jsonpickle 1.2 py_0 conda-forge jxrlib 1.1 h27ca646_2 conda-forge keras 2.9.0 pyhd8ed1ab_0 conda-forge keras-preprocessing 1.1.2 pyhd8ed1ab_0 conda-forge kiwisolver 1.4.5 py39hbd775c9_1 conda-forge krb5 1.19.3 hf9b2bbe_0 conda-forge lame 3.100 h1a8c8d9_1003 conda-forge lazy_loader 0.3 pyhd8ed1ab_0 conda-forge lcms2 2.14 h8193b64_0 conda-forge lerc 4.0.0 h9a09cb3_0 conda-forge libabseil 20211102.0 cxx17_h28b99d4_3 conda-forge libaec 1.0.6 hb7217d7_1 conda-forge libavif 0.11.1 h9f83d30_2 conda-forge libblas 3.9.0 18_osxarm64_openblas conda-forge libbrotlicommon 1.0.9 h1a8c8d9_9 conda-forge libbrotlidec 1.0.9 h1a8c8d9_9 conda-forge libbrotlienc 1.0.9 h1a8c8d9_9 conda-forge libcblas 3.9.0 18_osxarm64_openblas conda-forge libclang 15.0.7 default_h5dc8d65_3 conda-forge libclang13 15.0.7 default_hc7183e1_3 conda-forge libcurl 7.86.0 hd538317_1 conda-forge libcxx 16.0.6 h4653b0c_0 conda-forge libdeflate 1.14 h1a8c8d9_0 conda-forge libedit 3.1.20191231 hc8eb9b7_2 conda-forge libev 4.33 h642e427_1 conda-forge libexpat 2.5.0 hb7217d7_1 conda-forge libffi 3.4.2 h3422bc3_5 conda-forge libgfortran 5.0.0 13_2_0_hd922786_1 conda-forge libgfortran5 13.2.0 hf226fd6_1 conda-forge libglib 2.78.0 h24e9cb9_0 conda-forge libiconv 1.17 he4db4b2_0 conda-forge libidn2 2.3.4 h1a8c8d9_0 conda-forge liblapack 3.9.0 18_osxarm64_openblas conda-forge liblapacke 3.9.0 18_osxarm64_openblas conda-forge libllvm15 15.0.7 h62b9111_1 conda-forge libnghttp2 1.51.0 hd184df1_0 conda-forge libogg 1.3.4 h27ca646_1 conda-forge libopenblas 0.3.24 openmp_hd76b1f2_0 conda-forge libopencv 4.6.0 py39he1c1adf_3 conda-forge libopus 1.3.1 h27ca646_1 conda-forge libpng 1.6.39 h76d750c_0 conda-forge libpq 15.1 hb650857_1 conda-forge libprotobuf 3.20.3 hb5ab8b9_0 conda-forge libsodium 1.0.18 h27ca646_1 conda-forge libsqlite 3.43.0 hb31c410_0 conda-forge libssh2 1.10.0 hb80f160_3 conda-forge libtasn1 4.19.0 h1a8c8d9_0 conda-forge libtiff 4.4.0 heb92581_5 conda-forge libunistring 0.9.10 h3422bc3_0 conda-forge libvorbis 1.3.7 h9f76cd9_0 conda-forge libvpx 1.11.0 hc470f4d_3 conda-forge libwebp-base 1.3.2 hb547adb_0 conda-forge libxcb 1.13 h9b22ae9_1004 conda-forge libxml2 2.10.3 h67585b2_4 conda-forge libxslt 1.1.37 h1bd8bc4_0 conda-forge libzlib 1.2.13 h53f4e23_5 conda-forge libzopfli 1.0.3 h9f76cd9_0 conda-forge llvm-openmp 16.0.6 h1c12783_0 conda-forge locket 1.0.0 pyhd8ed1ab_0 conda-forge lz4-c 1.9.4 hb7217d7_0 conda-forge markdown 3.4.4 pyhd8ed1ab_0 conda-forge markdown-it-py 3.0.0 pyhd8ed1ab_0 conda-forge markupsafe 2.1.3 py39h0f82c59_1 conda-forge matplotlib-base 3.7.2 py39h70c6c4c_0 conda-forge mdurl 0.1.0 pyhd8ed1ab_0 conda-forge multidict 6.0.4 py39h02fc5c5_0 conda-forge munkres 1.1.4 pyh9f0ad1d_0 conda-forge mysql-common 8.0.32 hab468bb_0 conda-forge mysql-libs 8.0.32 hea58576_0 conda-forge ncurses 6.4 h7ea286d_0 conda-forge nettle 3.8.1 h63371fa_1 conda-forge networkx 3.1 pyhd8ed1ab_0 conda-forge nspr 4.35 hb7217d7_0 conda-forge nss 3.92 hc6b9969_0 conda-forge numpy 1.22.3 py39h25ab29e_0 anaconda numpy-base 1.22.3 py39h974a1f5_0 anaconda oauthlib 3.2.2 pyhd8ed1ab_0 conda-forge openblas 0.3.24 openmp_hce3e5ba_0 conda-forge opencv 4.6.0 py39hdf13c20_3 conda-forge openh264 2.3.1 hb7217d7_2 conda-forge openjpeg 2.5.0 h5d4e404_1 conda-forge openssl 1.1.1w h53f4e23_0 conda-forge opt_einsum 3.3.0 pyhd8ed1ab_1 conda-forge p11-kit 0.24.1 h29577a5_0 conda-forge packaging 23.1 pyhd8ed1ab_0 conda-forge pandas 2.0.3 py39h6b13a34_1 conda-forge partd 1.4.1 pyhd8ed1ab_0 conda-forge patsy 0.5.3 pyhd8ed1ab_0 conda-forge pcre2 10.40 hb34f9b4_0 conda-forge pillow 9.2.0 py39h139752e_3 conda-forge pip 23.2.1 pyhd8ed1ab_0 conda-forge pixman 0.40.0 h27ca646_0 conda-forge protobuf 3.20.3 py39h23fbdae_1 conda-forge psutil 5.9.5 py39h0f82c59_1 conda-forge pthread-stubs 0.4 h27ca646_1001 conda-forge py-opencv 4.6.0 py39hfa6204d_3 conda-forge pyasn1 0.5.0 pyhd8ed1ab_0 conda-forge pyasn1-modules 0.3.0 pyhd8ed1ab_0 conda-forge pycparser 2.21 pyhd8ed1ab_0 conda-forge pygments 2.16.1 pyhd8ed1ab_0 conda-forge pyjwt 2.8.0 pyhd8ed1ab_0 conda-forge pykalman 0.9.5 py_1 conda-forge pyopenssl 23.2.0 pyhd8ed1ab_1 conda-forge pyparsing 3.0.9 pyhd8ed1ab_0 conda-forge pyside2 5.15.8 py39h0adaba8_2 conda-forge pysocks 1.7.1 pyha2e5f31_6 conda-forge python 3.9.15 h2d96c93_0_cpython conda-forge python-dateutil 2.8.2 pyhd8ed1ab_0 conda-forge python-flatbuffers 1.12 pyhd8ed1ab_1 conda-forge python-rapidjson 1.11 py39hb198ff7_1 conda-forge python-tzdata 2023.3 pyhd8ed1ab_0 conda-forge python_abi 3.9 4_cp39 conda-forge pytz 2023.3.post1 pyhd8ed1ab_0 conda-forge pyu2f 0.1.5 pyhd8ed1ab_0 conda-forge pywavelets 1.4.1 py39h4d8bf0d_0 conda-forge pyyaml 6.0.1 py39h0f82c59_1 conda-forge pyzmq 25.1.1 py39h1e134f0_1 conda-forge qt-main 5.15.6 ha0e8b26_4 conda-forge qtpy 2.4.0 pyhd8ed1ab_0 conda-forge re2 2022.06.01 h9a09cb3_1 conda-forge readline 8.2 h92ec313_1 conda-forge requests 2.31.0 pyhd8ed1ab_0 conda-forge requests-oauthlib 1.3.1 pyhd8ed1ab_0 conda-forge rich 13.5.3 pyhd8ed1ab_0 conda-forge rsa 4.9 pyhd8ed1ab_0 conda-forge scikit-image 0.20.0 py39hd28f0be_1 conda-forge scikit-learn 1.0 py39h12ba089_1 conda-forge scikit-video 1.1.11 pyh24bf2e0_0 conda-forge scipy 1.9.0 py39h14896cb_0 conda-forge seaborn 0.12.2 hd8ed1ab_0 conda-forge seaborn-base 0.12.2 pyhd8ed1ab_0 conda-forge setuptools 68.2.2 pyhd8ed1ab_0 conda-forge shapely 2.0.1 py39h3010c4d_1 conda-forge six 1.16.0 pyh6c4a22f_0 conda-forge snappy 1.1.10 h17c5cce_0 conda-forge sqlite 3.43.0 h203b68d_0 conda-forge statsmodels 0.14.0 py39h8a366b7_1 conda-forge svt-av1 1.4.1 h7ea286d_0 conda-forge tensorboard 2.9.0 pyhd8ed1ab_0 conda-forge tensorboard-data-server 0.6.1 py39haa0b8cc_4 conda-forge tensorboard-plugin-wit 1.8.1 pyhd8ed1ab_0 conda-forge tensorflow 2.9.1 cpu_py39h2839aeb_0 conda-forge tensorflow-base 2.9.1 cpu_py39ha1ad4ae_0 conda-forge tensorflow-estimator 2.9.1 cpu_py39h7b621ec_0 conda-forge tensorflow-hub 0.12.0 pyhca92ed8_0 conda-forge termcolor 2.3.0 pyhd8ed1ab_0 conda-forge threadpoolctl 3.2.0 pyha21a80b_0 conda-forge tifffile 2022.10.10 pyhd8ed1ab_0 conda-forge tk 8.6.13 hb31c410_0 conda-forge toolz 0.12.0 pyhd8ed1ab_0 conda-forge typing-extensions 4.8.0 hd8ed1ab_0 conda-forge typing_extensions 4.8.0 pyha770c72_0 conda-forge tzdata 2023c h71feb2d_0 conda-forge unicodedata2 15.0.0 py39h0f82c59_1 conda-forge urllib3 1.26.16 pyhd8ed1ab_0 conda-forge werkzeug 2.3.7 pyhd8ed1ab_0 conda-forge wheel 0.41.2 pyhd8ed1ab_0 conda-forge wrapt 1.15.0 py39h0f82c59_1 conda-forge x264 1!164.3095 h57fd34a_2 conda-forge x265 3.5 hbc6ce65_3 conda-forge xorg-libxau 1.0.11 hb547adb_0 conda-forge xorg-libxdmcp 1.1.3 h27ca646_0 conda-forge xz 5.2.6 h57fd34a_0 conda-forge yaml 0.2.5 h3422bc3_2 conda-forge yarl 1.9.2 py39h0f82c59_0 conda-forge zeromq 4.3.4 hbdafb3b_1 conda-forge zfp 1.0.0 hb6e4faa_3 conda-forge zipp 3.17.0 pyhd8ed1ab_0 conda-forge zlib 1.2.13 h53f4e23_5 conda-forge zlib-ng 2.0.7 h1a8c8d9_0 conda-forge zstd 1.5.5 h4f39d0f_0 conda-forge ```pip freeze
``` absl-py @ file:///home/conda/feedstock_root/build_artifacts/absl-py_1695154477682/work accessible-pygments==0.0.4 aiohttp @ file:///Users/runner/miniforge3/conda-bld/aiohttp_1689805247820/work aiosignal @ file:///home/conda/feedstock_root/build_artifacts/aiosignal_1667935791922/work alabaster==0.7.13 aniposelib==0.4.3 anyio==3.7.1 appdirs==1.4.4 appnope==0.1.3 argon2-cffi==23.1.0 argon2-cffi-bindings==21.2.0 asttokens==2.4.0 astunparse @ file:///home/conda/feedstock_root/build_artifacts/astunparse_1610696312422/work async-timeout @ file:///home/conda/feedstock_root/build_artifacts/async-timeout_1691763562544/work attrs==21.4.0 Babel==2.12.1 backcall==0.2.0 beautifulsoup4==4.12.2 black==21.6b0 bleach==6.0.0 blinker @ file:///home/conda/feedstock_root/build_artifacts/blinker_1681349778161/work Brotli @ file:///Users/runner/miniforge3/conda-bld/brotli-split_1687884292427/work cached-property @ file:///home/conda/feedstock_root/build_artifacts/cached_property_1615209429212/work cachetools @ file:///home/conda/feedstock_root/build_artifacts/cachetools_1688227447901/work cattrs @ file:///home/conda/feedstock_root/build_artifacts/cattrs_1604136207372/work certifi==2023.7.22 cffi @ file:///Users/runner/miniforge3/conda-bld/cffi_1695367189446/work cfgv==3.4.0 charset-normalizer @ file:///home/conda/feedstock_root/build_artifacts/charset-normalizer_1688813409104/work click==8.0.4 cloudpickle @ file:///home/conda/feedstock_root/build_artifacts/cloudpickle_1674202310934/work colorama==0.4.6 comm==0.1.4 contourpy @ file:///Users/runner/miniforge3/conda-bld/contourpy_1695554288975/work coverage==7.3.1 cryptography @ file:///Users/runner/miniforge3/conda-bld/cryptography-split_1672673285537/work cycler @ file:///home/conda/feedstock_root/build_artifacts/cycler_1635519461629/work cytoolz @ file:///Users/runner/miniforge3/conda-bld/cytoolz_1695545219492/work dask @ file:///home/conda/feedstock_root/build_artifacts/dask-core_1694811846293/work debugpy==1.8.0 decorator==5.1.1 defusedxml==0.7.1 Deprecated==1.2.14 distlib==0.3.7 docutils==0.17.1 efficientnet==1.0.0 entrypoints==0.4 exceptiongroup==1.1.3 executing==1.2.0 filelock==3.12.4 flatbuffers @ file:///home/conda/feedstock_root/build_artifacts/python-flatbuffers_1617723079010/work fonttools @ file:///Users/runner/miniforge3/conda-bld/fonttools_1692542726085/work frozenlist @ file:///Users/runner/miniforge3/conda-bld/frozenlist_1695377682617/work fsspec @ file:///home/conda/feedstock_root/build_artifacts/fsspec_1695417469930/work gast @ file:///home/conda/feedstock_root/build_artifacts/gast_1596839682936/work gitdb==4.0.10 GitPython==3.1.37 google-auth @ file:///home/conda/feedstock_root/build_artifacts/google-auth_1694862673644/work google-auth-oauthlib @ file:///home/conda/feedstock_root/build_artifacts/google-auth-oauthlib_1630497468950/work google-pasta==0.2.0 grpcio @ file:///Users/runner/miniforge3/conda-bld/grpc-split_1660663644663/work h5py @ file:///Users/runner/miniforge3/conda-bld/h5py_1674499003063/work hdmf==3.9.0 identify==2.5.29 idna @ file:///home/conda/feedstock_root/build_artifacts/idna_1663625384323/work image-classifiers==1.0.0 imagecodecs @ file:///Users/runner/miniforge3/conda-bld/imagecodecs_1668867616820/work imageio @ file:///home/conda/feedstock_root/build_artifacts/imageio_1686552404404/work imageio-ffmpeg==0.4.9 imagesize==1.4.1 imgaug @ file:///home/conda/feedstock_root/build_artifacts/imgaug_1640909786103/work imgstore==0.2.9 importlib-metadata @ file:///home/conda/feedstock_root/build_artifacts/importlib-metadata_1688754491823/work importlib-resources @ file:///home/conda/feedstock_root/build_artifacts/importlib_resources_1695414790617/work iniconfig==2.0.0 ipykernel==6.25.2 ipython==8.15.0 ipython-genutils==0.2.0 ipywidgets==7.8.1 jaraco.classes==3.3.0 jedi==0.17.2 Jinja2==3.1.2 joblib @ file:///home/conda/feedstock_root/build_artifacts/joblib_1691577114857/work jsmin @ file:///home/conda/feedstock_root/build_artifacts/jsmin_1642532731678/work json5==0.9.14 jsonpickle==1.2 jsonschema==4.17.3 jupyter-cache==0.4.3 jupyter-server==1.13.5 jupyter-server-mathjax==0.2.6 jupyter-sphinx==0.3.2 jupyter_client==7.4.9 jupyter_core==5.3.1 jupyterlab==3.3.4 jupyterlab-pygments==0.2.2 jupyterlab-widgets==1.1.7 jupyterlab_server==2.16.3 keras @ file:///home/conda/feedstock_root/build_artifacts/keras_1652925255303/work/keras-2.9.0-py2.py3-none-any.whl Keras-Applications==1.0.8 Keras-Preprocessing @ file:///home/conda/feedstock_root/build_artifacts/keras-preprocessing_1610713559828/work keyring==24.2.0 kiwisolver @ file:///Users/runner/miniforge3/conda-bld/kiwisolver_1695380068083/work lazy_loader @ file:///home/conda/feedstock_root/build_artifacts/lazy_loader_1692295373316/work libclang==16.0.6 linkify-it-py==2.0.2 livereload==2.6.3 llvmlite==0.41.0 locket @ file:///home/conda/feedstock_root/build_artifacts/locket_1650660393415/work lxml==4.9.3 Markdown @ file:///home/conda/feedstock_root/build_artifacts/markdown_1690307387991/work markdown-it-py==1.1.0 MarkupSafe @ file:///Users/runner/miniforge3/conda-bld/markupsafe_1695367493730/work matplotlib @ file:///Users/runner/miniforge3/conda-bld/matplotlib-suite_1688685044808/work matplotlib-inline==0.1.6 mdit-py-plugins==0.2.8 mdurl @ file:///home/conda/feedstock_root/build_artifacts/mdurl_1639515908913/work mistune==0.8.4 more-itertools==10.1.0 multidict @ file:///Users/runner/miniforge3/conda-bld/multidict_1672339583149/work munkres==1.1.4 mypy-extensions==1.0.0 myst-nb==0.13.2 myst-parser==0.15.2 nbclassic==0.5.6 nbclient==0.5.13 nbconvert==6.5.4 nbdime==3.2.1 nbformat==5.1.3 ndx-pose==0.1.1 nest-asyncio==1.5.8 networkx @ file:///home/conda/feedstock_root/build_artifacts/networkx_1680692919326/work nh3==0.2.14 nixio==1.5.3 nodeenv==1.8.0 notebook==6.5.6 notebook_shim==0.2.3 numba==0.58.0 numpy @ file:///private/var/folders/nz/j6p8yfhx1mv_0grj5xl4650h0000gp/T/abs_ba5c3eaa-be93-4f42-ae24-fa720a3c6627m_sp7nbj/croots/recipe/numpy_and_numpy_base_1652801688716/work oauthlib @ file:///home/conda/feedstock_root/build_artifacts/oauthlib_1666056362788/work opencv-contrib-python==4.6.0.66 opencv-python==4.6.0 opt-einsum @ file:///home/conda/feedstock_root/build_artifacts/opt_einsum_1617859230218/work packaging @ file:///home/conda/feedstock_root/build_artifacts/packaging_1681337016113/work pandas @ file:///Users/runner/miniforge3/conda-bld/pandas_1688740575873/work pandocfilters==1.5.0 parso==0.7.1 partd @ file:///home/conda/feedstock_root/build_artifacts/partd_1695667515973/work pathspec==0.11.2 patsy @ file:///home/conda/feedstock_root/build_artifacts/patsy_1665356157073/work pexpect==4.8.0 pickleshare==0.7.5 Pillow @ file:///Users/runner/miniforge3/conda-bld/pillow_1666920692505/work pkginfo==1.9.6 platformdirs==3.10.0 pluggy==1.3.0 pre-commit==3.4.0 prometheus-client==0.17.1 prompt-toolkit==3.0.39 protobuf==3.19.6 psutil @ file:///Users/runner/miniforge3/conda-bld/psutil_1695367129654/work ptyprocess==0.7.0 pure-eval==0.2.2 pyasn1 @ file:///home/conda/feedstock_root/build_artifacts/pyasn1_1694615621498/work pyasn1-modules @ file:///home/conda/feedstock_root/build_artifacts/pyasn1-modules_1695107857548/work pycparser @ file:///home/conda/feedstock_root/build_artifacts/pycparser_1636257122734/work pydata-sphinx-theme==0.13.3 PyGithub==1.59.1 Pygments @ file:///home/conda/feedstock_root/build_artifacts/pygments_1691408637400/work PyJWT @ file:///home/conda/feedstock_root/build_artifacts/pyjwt_1689721553971/work pykalman==0.9.5 PyNaCl==1.5.0 pynwb==2.5.0 pyOpenSSL @ file:///home/conda/feedstock_root/build_artifacts/pyopenssl_1685514481738/work pyparsing @ file:///home/conda/feedstock_root/build_artifacts/pyparsing_1652235407899/work pyrsistent==0.19.3 PySide2==5.15.8 PySocks @ file:///home/conda/feedstock_root/build_artifacts/pysocks_1661604839144/work pytest==7.4.2 pytest-cov==3.0.0 pytest-qt==4.2.0 pytest-xvfb==3.0.0 python-dateutil @ file:///home/conda/feedstock_root/build_artifacts/python-dateutil_1626286286081/work python-rapidjson @ file:///Users/runner/miniforge3/conda-bld/python-rapidjson_1695615758828/work pytz @ file:///home/conda/feedstock_root/build_artifacts/pytz_1693930252784/work pyu2f @ file:///home/conda/feedstock_root/build_artifacts/pyu2f_1604248910016/work PyVirtualDisplay==3.0 PyWavelets @ file:///Users/runner/miniforge3/conda-bld/pywavelets_1673082473214/work PyYAML @ file:///Users/runner/miniforge3/conda-bld/pyyaml_1695373497981/work pyzmq==24.0.1 qimage2ndarray==1.10.0 QtPy @ file:///home/conda/feedstock_root/build_artifacts/qtpy_1693347765905/work readme-renderer==42.0 regex==2023.8.8 requests @ file:///home/conda/feedstock_root/build_artifacts/requests_1684774241324/work requests-oauthlib @ file:///home/conda/feedstock_root/build_artifacts/requests-oauthlib_1643557462909/work requests-toolbelt==1.0.0 rfc3986==2.0.0 rich @ file:///home/conda/feedstock_root/build_artifacts/rich-split_1694974539951/work/dist rsa @ file:///home/conda/feedstock_root/build_artifacts/rsa_1658328885051/work ruamel.yaml==0.17.32 ruamel.yaml.clib==0.2.7 scikit-image @ file:///Users/runner/miniforge3/conda-bld/scikit-image_1681313573054/work/dist/scikit_image-0.20.0-cp39-cp39-macosx_11_0_arm64.whl scikit-learn @ file:///Users/runner/miniforge3/conda-bld/scikit-learn_1632611447519/work scikit-video==1.1.11 scipy==1.9.0 seaborn @ file:///home/conda/feedstock_root/build_artifacts/seaborn-split_1672497695270/work segmentation-models==1.0.1 Send2Trash==1.8.2 shapely @ file:///Users/runner/miniforge3/conda-bld/shapely_1679082009014/work shiboken2==5.15.8 six==1.15.0 -e git+https://github.com/talmolab/sleap@6b14bcab93b8c1953913dcfbabd6508f78f77e8a#egg=sleap sleap-anipose==0.1.7 smmap==5.0.1 sniffio==1.3.0 snowballstemmer==2.2.0 soupsieve==2.5 Sphinx==4.5.0 sphinx-autobuild==2021.3.14 sphinx-book-theme==1.0.1 sphinx-copybutton==0.5.2 sphinx-togglebutton==0.3.2 sphinxcontrib-applehelp==1.0.4 sphinxcontrib-devhelp==1.0.2 sphinxcontrib-htmlhelp==2.0.1 sphinxcontrib-jsmath==1.0.1 sphinxcontrib-qthelp==1.0.3 sphinxcontrib-serializinghtml==1.1.5 SQLAlchemy==1.4.49 stack-data==0.6.2 statsmodels @ file:///Users/runner/miniforge3/conda-bld/statsmodels_1683305774790/work tensorboard @ file:///home/conda/feedstock_root/build_artifacts/tensorboard_1651519171059/work/tensorboard-2.9.0-py3-none-any.whl tensorboard-data-server @ file:///Users/runner/miniforge3/conda-bld/tensorboard-data-server_1670043861058/work/tensorboard_data_server-0.6.1-py3-none-macosx_11_0_arm64.whl tensorboard-plugin-wit @ file:///home/conda/feedstock_root/build_artifacts/tensorboard-plugin-wit_1641458951060/work/tensorboard_plugin_wit-1.8.1-py3-none-any.whl tensorflow @ file:///Users/ngam/Repos/tensorflow-feedstock/miniforge3/conda-bld/tensorflow-split_1658102176686/work/tensorflow_pkg/tensorflow-2.9.1-cp39-cp39-macosx_11_0_arm64.whl tensorflow-estimator @ file:///Users/ngam/Repos/tensorflow-feedstock/miniforge3/conda-bld/tensorflow-split_1658102176686/work/tensorflow-estimator/wheel_dir/tensorflow_estimator-2.9.0-py2.py3-none-any.whl tensorflow-hub @ file:///home/conda/feedstock_root/build_artifacts/tensorflow-hub_1618768305670/work/wheel_dir/tensorflow_hub-0.12.0-py2.py3-none-any.whl tensorflow-macos==2.9.2 tensorflow-metal==0.5.0 termcolor @ file:///home/conda/feedstock_root/build_artifacts/termcolor_1682317048417/work terminado==0.17.1 threadpoolctl @ file:///home/conda/feedstock_root/build_artifacts/threadpoolctl_1689261241048/work tifffile @ file:///home/conda/feedstock_root/build_artifacts/tifffile_1665588749940/work tinycss2==1.2.1 toml==0.10.2 tomli==2.0.1 toolz @ file:///home/conda/feedstock_root/build_artifacts/toolz_1657485559105/work tornado==6.3.3 tqdm==4.66.1 traitlets==5.10.1 twine==3.3.0 typing_extensions @ file:///home/conda/feedstock_root/build_artifacts/typing_extensions_1695040754690/work tzdata @ file:///home/conda/feedstock_root/build_artifacts/python-tzdata_1680081134351/work tzlocal==5.0.1 uc-micro-py==1.0.2 unicodedata2 @ file:///Users/runner/miniforge3/conda-bld/unicodedata2_1695405755547/work urllib3 @ file:///home/conda/feedstock_root/build_artifacts/urllib3_1695649846578/work virtualenv==20.24.5 wcwidth==0.2.6 webencodings==0.5.1 websocket-client==1.6.3 Werkzeug @ file:///home/conda/feedstock_root/build_artifacts/werkzeug_1692105915497/work widgetsnbextension==3.6.6 wrapt @ file:///Users/runner/miniforge3/conda-bld/wrapt_1695409130439/work yarl @ file:///Users/runner/miniforge3/conda-bld/yarl_1685191803109/work zipp @ file:///home/conda/feedstock_root/build_artifacts/zipp_1695255097490/work ```Logs
GUI shenanigans
``` (sa0) liezlmaree:~$sleap-label $dsdmc Saving config: /Users/liezlmaree/.sleap/1.3.3/preferences.yaml Restoring GUI state... Software versions: SLEAP: 1.3.3 TensorFlow: 2.9.2 Numpy: 1.22.3 Python: 3.9.15 OS: macOS-13.5-arm64-arm-64bit Happy SLEAPing! :) 2023-10-16 06:29:32.212 python[30835:1868817] +[CATransaction synchronize] called within transaction 2023-10-16 06:29:55.283 python[30835:1868817] +[CATransaction synchronize] called within transaction Restoring GUI state... 2023-10-16 06:30:17.615 python[30835:1868817] +[CATransaction synchronize] called within transaction Resetting monitor window. Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png Start training centroid... ['sleap-train', '/var/folders/64/rjln6zpx7tlgwf8cqgvhm7fr0000gn/T/tmp3zbfmn4v/231016_063130_training_job.json', '/Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/courtship_labels.pkg.slp', '--zmq', '--save_viz'] INFO:sleap.nn.training:Versions: SLEAP: 1.3.3 TensorFlow: 2.9.2 Numpy: 1.22.3 Python: 3.9.15 OS: macOS-13.5-arm64-arm-64bit ```centorid set-up
``` INFO:sleap.nn.training:Training labels file: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/courtship_labels.pkg.slp INFO:sleap.nn.training:Training profile: /var/folders/64/rjln6zpx7tlgwf8cqgvhm7fr0000gn/T/tmp3zbfmn4v/231016_063130_training_job.json INFO:sleap.nn.training: INFO:sleap.nn.training:Arguments: INFO:sleap.nn.training:{ "training_job_path": "/var/folders/64/rjln6zpx7tlgwf8cqgvhm7fr0000gn/T/tmp3zbfmn4v/231016_063130_training_job.json", "labels_path": "/Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/courtship_labels.pkg.slp", "video_paths": [ "" ], "val_labels": null, "test_labels": null, "base_checkpoint": null, "tensorboard": false, "save_viz": true, "zmq": true, "run_name": "", "prefix": "", "suffix": "", "cpu": false, "first_gpu": false, "last_gpu": false, "gpu": "auto" } INFO:sleap.nn.training: INFO:sleap.nn.training:Training job: INFO:sleap.nn.training:{ "data": { "labels": { "training_labels": null, "validation_labels": null, "validation_fraction": 0.1, "test_labels": null, "split_by_inds": false, "training_inds": null, "validation_inds": null, "test_inds": null, "search_path_hints": [], "skeletons": [] }, "preprocessing": { "ensure_rgb": false, "ensure_grayscale": false, "imagenet_mode": null, "input_scaling": 0.5, "pad_to_stride": null, "resize_and_pad_to_target": true, "target_height": null, "target_width": null }, "instance_cropping": { "center_on_part": "thorax", "crop_size": null, "crop_size_detection_padding": 16 } }, "model": { "backbone": { "leap": null, "unet": { "stem_stride": null, "max_stride": 16, "output_stride": 2, "filters": 16, "filters_rate": 2.0, "middle_block": true, "up_interpolate": true, "stacks": 1 }, "hourglass": null, "resnet": null, "pretrained_encoder": null }, "heads": { "single_instance": null, "centroid": { "anchor_part": "thorax", "sigma": 2.5, "output_stride": 2, "loss_weight": 1.0, "offset_refinement": false }, "centered_instance": null, "multi_instance": null, "multi_class_bottomup": null, "multi_class_topdown": null }, "base_checkpoint": null }, "optimization": { "preload_data": true, "augmentation_config": { "rotate": true, "rotation_min_angle": -180.0, "rotation_max_angle": 180.0, "translate": false, "translate_min": -5, "translate_max": 5, "scale": false, "scale_min": 0.9, "scale_max": 1.1, "uniform_noise": false, "uniform_noise_min_val": 0.0, "uniform_noise_max_val": 10.0, "gaussian_noise": false, "gaussian_noise_mean": 5.0, "gaussian_noise_stddev": 1.0, "contrast": false, "contrast_min_gamma": 0.5, "contrast_max_gamma": 2.0, "brightness": false, "brightness_min_val": 0.0, "brightness_max_val": 10.0, "random_crop": false, "random_crop_height": 256, "random_crop_width": 256, "random_flip": true, "flip_horizontal": false }, "online_shuffling": true, "shuffle_buffer_size": 128, "prefetch": true, "batch_size": 4, "batches_per_epoch": null, "min_batches_per_epoch": 200, "val_batches_per_epoch": null, "min_val_batches_per_epoch": 10, "epochs": 200, "optimizer": "adam", "initial_learning_rate": 0.0001, "learning_rate_schedule": { "reduce_on_plateau": true, "reduction_factor": 0.5, "plateau_min_delta": 1e-06, "plateau_patience": 5, "plateau_cooldown": 3, "min_learning_rate": 1e-08 }, "hard_keypoint_mining": { "online_mining": false, "hard_to_easy_ratio": 2.0, "min_hard_keypoints": 2, "max_hard_keypoints": null, "loss_scale": 5.0 }, "early_stopping": { "stop_training_on_plateau": true, "plateau_min_delta": 1e-08, "plateau_patience": 20 } }, "outputs": { "save_outputs": true, "run_name": "231016_063130.centroid.n=149", "run_name_prefix": "", "run_name_suffix": "", "runs_folder": "/Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models", "tags": [ "" ], "save_visualizations": true, "delete_viz_images": true, "zip_outputs": false, "log_to_csv": true, "checkpointing": { "initial_model": false, "best_model": true, "every_epoch": false, "latest_model": false, "final_model": false }, "tensorboard": { "write_logs": false, "loss_frequency": "epoch", "architecture_graph": false, "profile_graph": false, "visualizations": true }, "zmq": { "subscribe_to_controller": true, "controller_address": "tcp://127.0.0.1:9000", "controller_polling_timeout": 10, "publish_updates": true, "publish_address": "tcp://127.0.0.1:9001" } }, "name": "", "description": "", "sleap_version": "1.3.3", "filename": "/var/folders/64/rjln6zpx7tlgwf8cqgvhm7fr0000gn/T/tmp3zbfmn4v/231016_063130_training_job.json" } INFO:sleap.nn.training: INFO:sleap.nn.training:Failed to query GPU memory from nvidia-smi. Defaulting to first GPU. INFO:sleap.nn.training:Using GPU 0 for acceleration. INFO:sleap.nn.training:Disabled GPU memory pre-allocation. INFO:sleap.nn.training:System: GPUs: 1/1 available Device: /physical_device:GPU:0 Available: True Initalized: False Memory growth: True INFO:sleap.nn.training: INFO:sleap.nn.training:Initializing trainer... INFO:sleap.nn.training:Loading training labels from: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/courtship_labels.pkg.slp INFO:sleap.nn.training:Creating training and validation splits from validation fraction: 0.1 INFO:sleap.nn.training: Splits: Training = 134 / Validation = 15. INFO:sleap.nn.training:Setting up for training... INFO:sleap.nn.training:Setting up pipeline builders... INFO:sleap.nn.training:Setting up model... INFO:sleap.nn.training:Building test pipeline... Metal device set to: Apple M2 Pro systemMemory: 16.00 GB maxCacheSize: 5.33 GB 2023-10-16 06:31:34.872132: I tensorflow/core/common_runtime/pluggable_device/pluggable_device_factory.cc:305] Could not identify NUMA node of platform GPU ID 0, defaulting to 0. Your kernel may not have been built with NUMA support. 2023-10-16 06:31:34.872288: I tensorflow/core/common_runtime/pluggable_device/pluggable_device_factory.cc:271] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 0 MB memory) -> physical PluggableDevice (device: 0, name: METAL, pci bus id:centroid epochs
``` Epoch 1/200 2023-10-16 06:31:42.576707: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:113] Plugin optimizer for device_type GPU is enabled. 2023-10-16 06:32:18.788127: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:113] Plugin optimizer for device_type GPU is enabled. 200/200 - 39s - loss: 1.8696e-04 - val_loss: 7.9245e-05 - lr: 1.0000e-04 - 39s/epoch - 194ms/step Epoch 2/200 Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png 200/200 - 36s - loss: 5.5605e-05 - val_loss: 4.3189e-05 - lr: 1.0000e-04 - 36s/epoch - 182ms/step Epoch 3/200 Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png 200/200 - 36s - loss: 3.8371e-05 - val_loss: 3.1048e-05 - lr: 1.0000e-04 - 36s/epoch - 181ms/step Epoch 4/200 Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png 200/200 - 36s - loss: 3.3563e-05 - val_loss: 2.5386e-05 - lr: 1.0000e-04 - 36s/epoch - 181ms/step Epoch 5/200 Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png 200/200 - 36s - loss: 3.2097e-05 - val_loss: 2.8539e-05 - lr: 1.0000e-04 - 36s/epoch - 181ms/step Epoch 6/200 Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png 200/200 - 36s - loss: 2.9414e-05 - val_loss: 2.6226e-05 - lr: 1.0000e-04 - 36s/epoch - 181ms/step Epoch 7/200 Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png 200/200 - 36s - loss: 2.8567e-05 - val_loss: 2.5540e-05 - lr: 1.0000e-04 - 36s/epoch - 181ms/step Epoch 8/200 Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png 200/200 - 36s - loss: 2.7141e-05 - val_loss: 2.2127e-05 - lr: 1.0000e-04 - 36s/epoch - 180ms/step Epoch 9/200 Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png 200/200 - 36s - loss: 2.6605e-05 - val_loss: 2.4407e-05 - lr: 1.0000e-04 - 36s/epoch - 180ms/step Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png Epoch 10/200 200/200 - 36s - loss: 2.5723e-05 - val_loss: 2.1715e-05 - lr: 1.0000e-04 - 36s/epoch - 181ms/step Epoch 11/200 Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png 200/200 - 36s - loss: 2.4948e-05 - val_loss: 2.0652e-05 - lr: 1.0000e-04 - 36s/epoch - 181ms/step Epoch 12/200 Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png 200/200 - 36s - loss: 2.3232e-05 - val_loss: 1.8315e-05 - lr: 1.0000e-04 - 36s/epoch - 182ms/step Epoch 13/200 Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png 200/200 - 36s - loss: 2.4580e-05 - val_loss: 2.1906e-05 - lr: 1.0000e-04 - 36s/epoch - 181ms/step Epoch 14/200 Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png 200/200 - 36s - loss: 2.3585e-05 - val_loss: 2.5078e-05 - lr: 1.0000e-04 - 36s/epoch - 181ms/step Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png Epoch 15/200 200/200 - 36s - loss: 2.3469e-05 - val_loss: 2.0460e-05 - lr: 1.0000e-04 - 36s/epoch - 180ms/step Epoch 16/200 Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png 200/200 - 36s - loss: 2.3363e-05 - val_loss: 1.9228e-05 - lr: 1.0000e-04 - 36s/epoch - 182ms/step Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png Epoch 17/200 Epoch 17: ReduceLROnPlateau reducing learning rate to 4.999999873689376e-05. 200/200 - 36s - loss: 2.2882e-05 - val_loss: 1.8767e-05 - lr: 1.0000e-04 - 36s/epoch - 182ms/step Epoch 18/200 Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png 200/200 - 36s - loss: 2.0425e-05 - val_loss: 1.7738e-05 - lr: 5.0000e-05 - 36s/epoch - 182ms/step Epoch 19/200 Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png 200/200 - 36s - loss: 1.9551e-05 - val_loss: 1.8845e-05 - lr: 5.0000e-05 - 36s/epoch - 180ms/step Epoch 20/200 Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png 200/200 - 36s - loss: 2.0076e-05 - val_loss: 1.7287e-05 - lr: 5.0000e-05 - 36s/epoch - 181ms/step Epoch 21/200 Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png 200/200 - 36s - loss: 1.9783e-05 - val_loss: 1.7333e-05 - lr: 5.0000e-05 - 36s/epoch - 180ms/step Epoch 22/200 Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png 200/200 - 36s - loss: 1.9134e-05 - val_loss: 1.5902e-05 - lr: 5.0000e-05 - 36s/epoch - 181ms/step Epoch 23/200 Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png 200/200 - 36s - loss: 1.9437e-05 - val_loss: 1.9749e-05 - lr: 5.0000e-05 - 36s/epoch - 181ms/step Epoch 24/200 Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png 200/200 - 36s - loss: 1.8544e-05 - val_loss: 1.6604e-05 - lr: 5.0000e-05 - 36s/epoch - 181ms/step Epoch 25/200 Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png 200/200 - 36s - loss: 1.9098e-05 - val_loss: 1.7043e-05 - lr: 5.0000e-05 - 36s/epoch - 181ms/step Epoch 26/200 Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png 200/200 - 36s - loss: 1.8231e-05 - val_loss: 1.6614e-05 - lr: 5.0000e-05 - 36s/epoch - 181ms/step Epoch 27/200 Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png Epoch 27: ReduceLROnPlateau reducing learning rate to 2.499999936844688e-05. 200/200 - 36s - loss: 1.8520e-05 - val_loss: 1.5647e-05 - lr: 5.0000e-05 - 36s/epoch - 181ms/step Epoch 28/200 Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png 200/200 - 36s - loss: 1.7531e-05 - val_loss: 1.7952e-05 - lr: 2.5000e-05 - 36s/epoch - 182ms/step Epoch 29/200 Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png 200/200 - 36s - loss: 1.7436e-05 - val_loss: 1.5768e-05 - lr: 2.5000e-05 - 36s/epoch - 182ms/step Epoch 30/200 Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png ```centered-instance set-up
``` "scale_min": 0.9, "scale_max": 1.1, "uniform_noise": false, "uniform_noise_min_val": 0.0, "uniform_noise_max_val": 10.0, "gaussian_noise": false, "gaussian_noise_mean": 5.0, "gaussian_noise_stddev": 1.0, "contrast": false, "contrast_min_gamma": 0.5, "contrast_max_gamma": 2.0, "brightness": false, "brightness_min_val": 0.0, "brightness_max_val": 10.0, "random_crop": false, "random_crop_height": 256, "random_crop_width": 256, "random_flip": true, "flip_horizontal": false }, "online_shuffling": true, "shuffle_buffer_size": 128, "prefetch": true, "batch_size": 4, "batches_per_epoch": null, "min_batches_per_epoch": 200, "val_batches_per_epoch": null, "min_val_batches_per_epoch": 10, "epochs": 200, "optimizer": "adam", "initial_learning_rate": 0.0001, "learning_rate_schedule": { "reduce_on_plateau": true, "reduction_factor": 0.5, "plateau_min_delta": 1e-06, "plateau_patience": 5, "plateau_cooldown": 3, "min_learning_rate": 1e-08 }, "hard_keypoint_mining": { "online_mining": false, "hard_to_easy_ratio": 2.0, "min_hard_keypoints": 2, "max_hard_keypoints": null, "loss_scale": 5.0 }, "early_stopping": { "stop_training_on_plateau": true, "plateau_min_delta": 1e-08, "plateau_patience": 10 } }, "outputs": { "save_outputs": true, "run_name": "231016_070843.centered_instance.n=149", "run_name_prefix": "", "run_name_suffix": "", "runs_folder": "/Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models", "tags": [ "" ], "save_visualizations": true, "delete_viz_images": true, "zip_outputs": false, "log_to_csv": true, "checkpointing": { "initial_model": false, "best_model": true, "every_epoch": false, "latest_model": false, "final_model": false }, "tensorboard": { "write_logs": false, "loss_frequency": "epoch", "architecture_graph": false, "profile_graph": false, "visualizations": true }, "zmq": { "subscribe_to_controller": true, "controller_address": "tcp://127.0.0.1:9000", "controller_polling_timeout": 10, "publish_updates": true, "publish_address": "tcp://127.0.0.1:9001" } }, "name": "", "description": "", "sleap_version": "1.3.3", "filename": "/var/folders/64/rjln6zpx7tlgwf8cqgvhm7fr0000gn/T/tmpuot85hlo/231016_070843_training_job.json" } INFO:sleap.nn.training: INFO:sleap.nn.training:Failed to query GPU memory from nvidia-smi. Defaulting to first GPU. INFO:sleap.nn.training:Using GPU 0 for acceleration. INFO:sleap.nn.training:Disabled GPU memory pre-allocation. INFO:sleap.nn.training:System: GPUs: 1/1 available Device: /physical_device:GPU:0 Available: True Initalized: False Memory growth: True INFO:sleap.nn.training: INFO:sleap.nn.training:Initializing trainer... INFO:sleap.nn.training:Loading training labels from: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/courtship_labels.pkg.slp INFO:sleap.nn.training:Creating training and validation splits from validation fraction: 0.1 INFO:sleap.nn.training: Splits: Training = 134 / Validation = 15. INFO:sleap.nn.training:Setting up for training... INFO:sleap.nn.training:Setting up pipeline builders... INFO:sleap.nn.training:Setting up model... INFO:sleap.nn.training:Building test pipeline... Metal device set to: Apple M2 Pro systemMemory: 16.00 GB maxCacheSize: 5.33 GB 2023-10-16 07:08:48.695045: I tensorflow/core/common_runtime/pluggable_device/pluggable_device_factory.cc:305] Could not identify NUMA node of platform GPU ID 0, defaulting to 0. Your kernel may not have been built with NUMA support. 2023-10-16 07:08:48.695183: I tensorflow/core/common_runtime/pluggable_device/pluggable_device_factory.cc:271] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 0 MB memory) -> physical PluggableDevice (device: 0, name: METAL, pci bus id:centered-instance epochs
``` 2023-10-16 07:08:56.464164: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: 1 } dim { size: 1024 } dim { size: 1024 } dim { size: 3 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 10 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 144 } dim { size: 144 } dim { size: 3 } } } Epoch 1/200 2023-10-16 07:08:57.322132: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:113] Plugin optimizer for device_type GPU is enabled. 2023-10-16 07:09:09.114265: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: 1 } dim { size: 1024 } dim { size: 1024 } dim { size: 3 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 10 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 144 } dim { size: 144 } dim { size: 3 } } } 2023-10-16 07:09:09.463435: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:113] Plugin optimizer for device_type GPU is enabled. 2023-10-16 07:09:10.410374: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:113] Plugin optimizer for device_type GPU is enabled. 2023-10-16 07:09:10.420501: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: 13 } dim { size: 36 } dim { size: 36 } dim { size: 1 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 10 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: -5 } dim { size: -6 } dim { size: 1 } } } 200/200 - 14s - loss: 0.0109 - head: 0.0070 - thorax: 0.0062 - abdomen: 0.0112 - wingL: 0.0125 - wingR: 0.0127 - forelegL4: 0.0112 - forelegR4: 0.0109 - midlegL4: 0.0131 - midlegR4: 0.0132 - hindlegL4: 0.0131 - hindlegR4: 0.0130 - eyeL: 0.0091 - eyeR: 0.0090 - val_loss: 0.0090 - val_head: 0.0033 - val_thorax: 0.0036 - val_abdomen: 0.0076 - val_wingL: 0.0086 - val_wingR: 0.0093 - val_forelegL4: 0.0101 - val_forelegR4: 0.0106 - val_midlegL4: 0.0122 - val_midlegR4: 0.0125 - val_hindlegL4: 0.0128 - val_hindlegR4: 0.0118 - val_eyeL: 0.0070 - val_eyeR: 0.0074 - lr: 1.0000e-04 - 14s/epoch - 72ms/step Epoch 2/200 Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_070843.centered_instance.n=149/viz/validation.*.png 2023-10-16 07:09:22.245987: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: 1 } dim { size: 1024 } dim { size: 1024 } dim { size: 3 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 10 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 144 } dim { size: 144 } dim { size: 3 } } } 200/200 - 12s - loss: 0.0081 - head: 0.0020 - thorax: 0.0028 - abdomen: 0.0061 - wingL: 0.0069 - wingR: 0.0078 - forelegL4: 0.0097 - forelegR4: 0.0093 - midlegL4: 0.0119 - midlegR4: 0.0122 - hindlegL4: 0.0121 - hindlegR4: 0.0117 - eyeL: 0.0068 - eyeR: 0.0066 - val_loss: 0.0075 - val_head: 0.0013 - val_thorax: 0.0026 - val_abdomen: 0.0042 - val_wingL: 0.0056 - val_wingR: 0.0059 - val_forelegL4: 0.0086 - val_forelegR4: 0.0100 - val_midlegL4: 0.0117 - val_midlegR4: 0.0123 - val_hindlegL4: 0.0114 - val_hindlegR4: 0.0114 - val_eyeL: 0.0062 - val_eyeR: 0.0062 - lr: 1.0000e-04 - 12s/epoch - 60ms/step Epoch 3/200 Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_070843.centered_instance.n=149/viz/validation.*.png 2023-10-16 07:09:34.223959: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: 1 } dim { size: 1024 } dim { size: 1024 } dim { size: 3 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 10 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 144 } dim { size: 144 } dim { size: 3 } } } 200/200 - 12s - loss: 0.0073 - head: 0.0011 - thorax: 0.0019 - abdomen: 0.0039 - wingL: 0.0048 - wingR: 0.0061 - forelegL4: 0.0094 - forelegR4: 0.0090 - midlegL4: 0.0116 - midlegR4: 0.0117 - hindlegL4: 0.0113 - hindlegR4: 0.0114 - eyeL: 0.0063 - eyeR: 0.0061 - val_loss: 0.0073 - val_head: 8.8589e-04 - val_thorax: 0.0018 - val_abdomen: 0.0035 - val_wingL: 0.0046 - val_wingR: 0.0049 - val_forelegL4: 0.0097 - val_forelegR4: 0.0097 - val_midlegL4: 0.0117 - val_midlegR4: 0.0121 - val_hindlegL4: 0.0118 - val_hindlegR4: 0.0115 - val_eyeL: 0.0060 - val_eyeR: 0.0066 - lr: 1.0000e-04 - 12s/epoch - 60ms/step Epoch 4/200 Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_070843.centered_instance.n=149/viz/validation.*.png 2023-10-16 07:09:46.088737: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: 1 } dim { size: 1024 } dim { size: 1024 } dim { size: 3 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 10 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 144 } dim { size: 144 } dim { size: 3 } } } 200/200 - 12s - loss: 0.0069 - head: 7.7567e-04 - thorax: 0.0014 - abdomen: 0.0031 - wingL: 0.0042 - wingR: 0.0053 - forelegL4: 0.0093 - forelegR4: 0.0088 - midlegL4: 0.0113 - midlegR4: 0.0114 - hindlegL4: 0.0111 - hindlegR4: 0.0110 - eyeL: 0.0061 - eyeR: 0.0060 - val_loss: 0.0071 - val_head: 5.7390e-04 - val_thorax: 0.0015 - val_abdomen: 0.0035 - val_wingL: 0.0041 - val_wingR: 0.0043 - val_forelegL4: 0.0094 - val_forelegR4: 0.0099 - val_midlegL4: 0.0114 - val_midlegR4: 0.0120 - val_hindlegL4: 0.0116 - val_hindlegR4: 0.0110 - val_eyeL: 0.0061 - val_eyeR: 0.0064 - lr: 1.0000e-04 - 12s/epoch - 59ms/step Epoch 5/200 Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_070843.centered_instance.n=149/viz/validation.*.png 2023-10-16 07:09:58.092878: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: 1 } dim { size: 1024 } dim { size: 1024 } dim { size: 3 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 10 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 144 } dim { size: 144 } dim { size: 3 } } } 200/200 - 12s - loss: 0.0067 - head: 7.3147e-04 - thorax: 0.0011 - abdomen: 0.0027 - wingL: 0.0038 - wingR: 0.0050 - forelegL4: 0.0091 - forelegR4: 0.0086 - midlegL4: 0.0109 - midlegR4: 0.0109 - hindlegL4: 0.0110 - hindlegR4: 0.0107 - eyeL: 0.0060 - eyeR: 0.0059 - val_loss: 0.0070 - val_head: 7.0433e-04 - val_thorax: 0.0011 - val_abdomen: 0.0031 - val_wingL: 0.0046 - val_wingR: 0.0048 - val_forelegL4: 0.0090 - val_forelegR4: 0.0099 - val_midlegL4: 0.0113 - val_midlegR4: 0.0120 - val_hindlegL4: 0.0118 - val_hindlegR4: 0.0111 - val_eyeL: 0.0060 - val_eyeR: 0.0063 - lr: 1.0000e-04 - 12s/epoch - 59ms/step Epoch 6/200 Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_070843.centered_instance.n=149/viz/validation.*.png 2023-10-16 07:10:09.993887: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: 1 } dim { size: 1024 } dim { size: 1024 } dim { size: 3 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 10 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 144 } dim { size: 144 } dim { size: 3 } } } 200/200 - 12s - loss: 0.0064 - head: 6.7371e-04 - thorax: 9.1523e-04 - abdomen: 0.0024 - wingL: 0.0036 - wingR: 0.0048 - forelegL4: 0.0089 - forelegR4: 0.0084 - midlegL4: 0.0106 - midlegR4: 0.0105 - hindlegL4: 0.0107 - hindlegR4: 0.0104 - eyeL: 0.0060 - eyeR: 0.0058 - val_loss: 0.0066 - val_head: 6.7408e-04 - val_thorax: 0.0012 - val_abdomen: 0.0027 - val_wingL: 0.0041 - val_wingR: 0.0042 - val_forelegL4: 0.0090 - val_forelegR4: 0.0093 - val_midlegL4: 0.0102 - val_midlegR4: 0.0115 - val_hindlegL4: 0.0108 - val_hindlegR4: 0.0108 - val_eyeL: 0.0057 - val_eyeR: 0.0062 - lr: 1.0000e-04 - 12s/epoch - 60ms/step Epoch 7/200 Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_070843.centered_instance.n=149/viz/validation.*.png 2023-10-16 07:10:22.083929: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: 1 } dim { size: 1024 } dim { size: 1024 } dim { size: 3 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 10 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 144 } dim { size: 144 } dim { size: 3 } } } 200/200 - 12s - loss: 0.0063 - head: 5.7252e-04 - thorax: 8.0404e-04 - abdomen: 0.0024 - wingL: 0.0035 - wingR: 0.0046 - forelegL4: 0.0087 - forelegR4: 0.0083 - midlegL4: 0.0103 - midlegR4: 0.0101 - hindlegL4: 0.0106 - hindlegR4: 0.0100 - eyeL: 0.0059 - eyeR: 0.0058 - val_loss: 0.0066 - val_head: 7.2597e-04 - val_thorax: 0.0011 - val_abdomen: 0.0031 - val_wingL: 0.0047 - val_wingR: 0.0047 - val_forelegL4: 0.0091 - val_forelegR4: 0.0086 - val_midlegL4: 0.0098 - val_midlegR4: 0.0112 - val_hindlegL4: 0.0107 - val_hindlegR4: 0.0102 - val_eyeL: 0.0057 - val_eyeR: 0.0059 - lr: 1.0000e-04 - 12s/epoch - 60ms/step Epoch 8/200 Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_070843.centered_instance.n=149/viz/validation.*.png 2023-10-16 07:10:34.155638: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: 1 } dim { size: 1024 } dim { size: 1024 } dim { size: 3 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 10 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 144 } dim { size: 144 } dim { size: 3 } } } 200/200 - 12s - loss: 0.0061 - head: 5.7429e-04 - thorax: 7.5300e-04 - abdomen: 0.0022 - wingL: 0.0032 - wingR: 0.0046 - forelegL4: 0.0084 - forelegR4: 0.0081 - midlegL4: 0.0099 - midlegR4: 0.0098 - hindlegL4: 0.0104 - hindlegR4: 0.0096 - eyeL: 0.0058 - eyeR: 0.0058 - val_loss: 0.0063 - val_head: 5.3473e-04 - val_thorax: 7.9216e-04 - val_abdomen: 0.0021 - val_wingL: 0.0038 - val_wingR: 0.0040 - val_forelegL4: 0.0089 - val_forelegR4: 0.0083 - val_midlegL4: 0.0103 - val_midlegR4: 0.0104 - val_hindlegL4: 0.0109 - val_hindlegR4: 0.0102 - val_eyeL: 0.0057 - val_eyeR: 0.0061 - lr: 1.0000e-04 - 12s/epoch - 60ms/step Epoch 9/200 Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_070843.centered_instance.n=149/viz/validation.*.png 2023-10-16 07:10:46.228547: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: 1 } dim { size: 1024 } dim { size: 1024 } dim { size: 3 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 10 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 144 } dim { size: 144 } dim { size: 3 } } } 200/200 - 12s - loss: 0.0060 - head: 5.5613e-04 - thorax: 7.6087e-04 - abdomen: 0.0020 - wingL: 0.0033 - wingR: 0.0045 - forelegL4: 0.0082 - forelegR4: 0.0079 - midlegL4: 0.0096 - midlegR4: 0.0096 - hindlegL4: 0.0102 - hindlegR4: 0.0095 - eyeL: 0.0058 - eyeR: 0.0058 - val_loss: 0.0061 - val_head: 3.8938e-04 - val_thorax: 7.1305e-04 - val_abdomen: 0.0016 - val_wingL: 0.0046 - val_wingR: 0.0040 - val_forelegL4: 0.0081 - val_forelegR4: 0.0085 - val_midlegL4: 0.0095 - val_midlegR4: 0.0103 - val_hindlegL4: 0.0103 - val_hindlegR4: 0.0100 - val_eyeL: 0.0057 - val_eyeR: 0.0060 - lr: 1.0000e-04 - 12s/epoch - 60ms/step Epoch 10/200 Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_070843.centered_instance.n=149/viz/validation.*.png 2023-10-16 07:10:58.254701: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: 1 } dim { size: 1024 } dim { size: 1024 } dim { size: 3 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 10 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 144 } dim { size: 144 } dim { size: 3 } } } 200/200 - 12s - loss: 0.0058 - head: 4.8496e-04 - thorax: 6.5602e-04 - abdomen: 0.0018 - wingL: 0.0031 - wingR: 0.0043 - forelegL4: 0.0080 - forelegR4: 0.0078 - midlegL4: 0.0095 - midlegR4: 0.0094 - hindlegL4: 0.0100 - hindlegR4: 0.0091 - eyeL: 0.0057 - eyeR: 0.0057 - val_loss: 0.0061 - val_head: 4.1555e-04 - val_thorax: 6.8679e-04 - val_abdomen: 0.0019 - val_wingL: 0.0046 - val_wingR: 0.0040 - val_forelegL4: 0.0081 - val_forelegR4: 0.0084 - val_midlegL4: 0.0091 - val_midlegR4: 0.0107 - val_hindlegL4: 0.0099 - val_hindlegR4: 0.0099 - val_eyeL: 0.0053 - val_eyeR: 0.0061 - lr: 1.0000e-04 - 12s/epoch - 60ms/step Epoch 11/200 Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_070843.centered_instance.n=149/viz/validation.*.png 2023-10-16 07:11:10.393284: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: 1 } dim { size: 1024 } dim { size: 1024 } dim { size: 3 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 10 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 144 } dim { size: 144 } dim { size: 3 } } } 200/200 - 12s - loss: 0.0057 - head: 4.7110e-04 - thorax: 6.2252e-04 - abdomen: 0.0017 - wingL: 0.0031 - wingR: 0.0044 - forelegL4: 0.0079 - forelegR4: 0.0076 - midlegL4: 0.0093 - midlegR4: 0.0093 - hindlegL4: 0.0098 - hindlegR4: 0.0089 - eyeL: 0.0058 - eyeR: 0.0057 - val_loss: 0.0061 - val_head: 4.9512e-04 - val_thorax: 9.8132e-04 - val_abdomen: 0.0019 - val_wingL: 0.0041 - val_wingR: 0.0035 - val_forelegL4: 0.0082 - val_forelegR4: 0.0083 - val_midlegL4: 0.0098 - val_midlegR4: 0.0099 - val_hindlegL4: 0.0103 - val_hindlegR4: 0.0101 - val_eyeL: 0.0060 - val_eyeR: 0.0063 - lr: 1.0000e-04 - 12s/epoch - 60ms/step Epoch 12/200 Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_070843.centered_instance.n=149/viz/validation.*.png 2023-10-16 07:11:22.584388: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: 1 } dim { size: 1024 } dim { size: 1024 } dim { size: 3 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 10 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 144 } dim { size: 144 } dim { size: 3 } } } 200/200 - 12s - loss: 0.0056 - head: 4.3117e-04 - thorax: 5.9805e-04 - abdomen: 0.0016 - wingL: 0.0029 - wingR: 0.0043 - forelegL4: 0.0077 - forelegR4: 0.0075 - midlegL4: 0.0092 - midlegR4: 0.0091 - hindlegL4: 0.0095 - hindlegR4: 0.0087 - eyeL: 0.0057 - eyeR: 0.0057 - val_loss: 0.0061 - val_head: 4.1128e-04 - val_thorax: 6.8850e-04 - val_abdomen: 0.0019 - val_wingL: 0.0042 - val_wingR: 0.0045 - val_forelegL4: 0.0081 - val_forelegR4: 0.0082 - val_midlegL4: 0.0096 - val_midlegR4: 0.0101 - val_hindlegL4: 0.0101 - val_hindlegR4: 0.0096 - val_eyeL: 0.0056 - val_eyeR: 0.0063 - lr: 1.0000e-04 - 12s/epoch - 61ms/step Epoch 13/200 Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_070843.centered_instance.n=149/viz/validation.*.png 2023-10-16 07:11:34.902982: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: 1 } dim { size: 1024 } dim { size: 1024 } dim { size: 3 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 10 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 144 } dim { size: 144 } dim { size: 3 } } } 200/200 - 12s - loss: 0.0056 - head: 4.3588e-04 - thorax: 5.8300e-04 - abdomen: 0.0015 - wingL: 0.0029 - wingR: 0.0042 - forelegL4: 0.0077 - forelegR4: 0.0075 - midlegL4: 0.0090 - midlegR4: 0.0091 - hindlegL4: 0.0094 - hindlegR4: 0.0085 - eyeL: 0.0057 - eyeR: 0.0057 - val_loss: 0.0059 - val_head: 4.6890e-04 - val_thorax: 6.5690e-04 - val_abdomen: 0.0015 - val_wingL: 0.0040 - val_wingR: 0.0042 - val_forelegL4: 0.0080 - val_forelegR4: 0.0080 - val_midlegL4: 0.0089 - val_midlegR4: 0.0100 - val_hindlegL4: 0.0104 - val_hindlegR4: 0.0095 - val_eyeL: 0.0056 - val_eyeR: 0.0060 - lr: 1.0000e-04 - 12s/epoch - 62ms/step Epoch 14/200 Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_070843.centered_instance.n=149/viz/validation.*.png 2023-10-16 07:11:47.214435: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: 1 } dim { size: 1024 } dim { size: 1024 } dim { size: 3 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 10 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 144 } dim { size: 144 } dim { size: 3 } } } 200/200 - 12s - loss: 0.0055 - head: 4.3798e-04 - thorax: 5.6456e-04 - abdomen: 0.0014 - wingL: 0.0028 - wingR: 0.0041 - forelegL4: 0.0076 - forelegR4: 0.0074 - midlegL4: 0.0089 - midlegR4: 0.0090 - hindlegL4: 0.0093 - hindlegR4: 0.0084 - eyeL: 0.0058 - eyeR: 0.0057 - val_loss: 0.0059 - val_head: 4.2385e-04 - val_thorax: 5.2439e-04 - val_abdomen: 0.0014 - val_wingL: 0.0033 - val_wingR: 0.0037 - val_forelegL4: 0.0087 - val_forelegR4: 0.0078 - val_midlegL4: 0.0095 - val_midlegR4: 0.0099 - val_hindlegL4: 0.0104 - val_hindlegR4: 0.0095 - val_eyeL: 0.0059 - val_eyeR: 0.0059 - lr: 1.0000e-04 - 12s/epoch - 61ms/step Epoch 15/200 Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_070843.centered_instance.n=149/viz/validation.*.png 2023-10-16 07:11:59.501349: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: 1 } dim { size: 1024 } dim { size: 1024 } dim { size: 3 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 10 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 144 } dim { size: 144 } dim { size: 3 } } } 200/200 - 12s - loss: 0.0054 - head: 4.2132e-04 - thorax: 5.4735e-04 - abdomen: 0.0015 - wingL: 0.0028 - wingR: 0.0041 - forelegL4: 0.0075 - forelegR4: 0.0073 - midlegL4: 0.0088 - midlegR4: 0.0089 - hindlegL4: 0.0091 - hindlegR4: 0.0083 - eyeL: 0.0057 - eyeR: 0.0057 - val_loss: 0.0059 - val_head: 4.4959e-04 - val_thorax: 8.1589e-04 - val_abdomen: 0.0019 - val_wingL: 0.0039 - val_wingR: 0.0038 - val_forelegL4: 0.0084 - val_forelegR4: 0.0076 - val_midlegL4: 0.0092 - val_midlegR4: 0.0092 - val_hindlegL4: 0.0100 - val_hindlegR4: 0.0092 - val_eyeL: 0.0056 - val_eyeR: 0.0060 - lr: 1.0000e-04 - 12s/epoch - 61ms/step Epoch 16/200 Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_070843.centered_instance.n=149/viz/validation.*.png 2023-10-16 07:12:11.712830: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: 1 } dim { size: 1024 } dim { size: 1024 } dim { size: 3 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 10 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 144 } dim { size: 144 } dim { size: 3 } } } 200/200 - 12s - loss: 0.0054 - head: 4.2216e-04 - thorax: 5.4495e-04 - abdomen: 0.0014 - wingL: 0.0028 - wingR: 0.0040 - forelegL4: 0.0075 - forelegR4: 0.0072 - midlegL4: 0.0088 - midlegR4: 0.0087 - hindlegL4: 0.0088 - hindlegR4: 0.0081 - eyeL: 0.0057 - eyeR: 0.0057 - val_loss: 0.0060 - val_head: 4.4052e-04 - val_thorax: 6.4149e-04 - val_abdomen: 0.0017 - val_wingL: 0.0049 - val_wingR: 0.0037 - val_forelegL4: 0.0079 - val_forelegR4: 0.0083 - val_midlegL4: 0.0096 - val_midlegR4: 0.0100 - val_hindlegL4: 0.0097 - val_hindlegR4: 0.0089 - val_eyeL: 0.0057 - val_eyeR: 0.0062 - lr: 1.0000e-04 - 12s/epoch - 61ms/step Epoch 17/200 Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_070843.centered_instance.n=149/viz/validation.*.png 2023-10-16 07:12:23.892308: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: 1 } dim { size: 1024 } dim { size: 1024 } dim { size: 3 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 10 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 144 } dim { size: 144 } dim { size: 3 } } } 200/200 - 12s - loss: 0.0053 - head: 3.9981e-04 - thorax: 5.1178e-04 - abdomen: 0.0013 - wingL: 0.0026 - wingR: 0.0038 - forelegL4: 0.0074 - forelegR4: 0.0072 - midlegL4: 0.0086 - midlegR4: 0.0087 - hindlegL4: 0.0087 - hindlegR4: 0.0080 - eyeL: 0.0057 - eyeR: 0.0057 - val_loss: 0.0057 - val_head: 4.6381e-04 - val_thorax: 5.2619e-04 - val_abdomen: 0.0014 - val_wingL: 0.0043 - val_wingR: 0.0037 - val_forelegL4: 0.0073 - val_forelegR4: 0.0080 - val_midlegL4: 0.0087 - val_midlegR4: 0.0097 - val_hindlegL4: 0.0093 - val_hindlegR4: 0.0094 - val_eyeL: 0.0058 - val_eyeR: 0.0058 - lr: 1.0000e-04 - 12s/epoch - 61ms/step Epoch 18/200 Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_070843.centered_instance.n=149/viz/validation.*.png 2023-10-16 07:12:36.195365: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: 1 } dim { size: 1024 } dim { size: 1024 } dim { size: 3 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 10 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 144 } dim { size: 144 } dim { size: 3 } } } 200/200 - 12s - loss: 0.0052 - head: 4.1838e-04 - thorax: 5.1378e-04 - abdomen: 0.0014 - wingL: 0.0026 - wingR: 0.0037 - forelegL4: 0.0073 - forelegR4: 0.0072 - midlegL4: 0.0086 - midlegR4: 0.0086 - hindlegL4: 0.0086 - hindlegR4: 0.0078 - eyeL: 0.0057 - eyeR: 0.0057 - val_loss: 0.0058 - val_head: 4.9995e-04 - val_thorax: 6.1695e-04 - val_abdomen: 0.0015 - val_wingL: 0.0039 - val_wingR: 0.0037 - val_forelegL4: 0.0078 - val_forelegR4: 0.0084 - val_midlegL4: 0.0081 - val_midlegR4: 0.0104 - val_hindlegL4: 0.0094 - val_hindlegR4: 0.0088 - val_eyeL: 0.0056 - val_eyeR: 0.0061 - lr: 1.0000e-04 - 12s/epoch - 61ms/step Epoch 19/200 Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_070843.centered_instance.n=149/viz/validation.*.png 2023-10-16 07:12:48.376745: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: 1 } dim { size: 1024 } dim { size: 1024 } dim { size: 3 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 10 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 144 } dim { size: 144 } dim { size: 3 } } } 200/200 - 12s - loss: 0.0051 - head: 4.2751e-04 - thorax: 5.1252e-04 - abdomen: 0.0013 - wingL: 0.0024 - wingR: 0.0035 - forelegL4: 0.0073 - forelegR4: 0.0071 - midlegL4: 0.0085 - midlegR4: 0.0085 - hindlegL4: 0.0083 - hindlegR4: 0.0075 - eyeL: 0.0057 - eyeR: 0.0056 - val_loss: 0.0058 - val_head: 5.3449e-04 - val_thorax: 4.2500e-04 - val_abdomen: 0.0015 - val_wingL: 0.0039 - val_wingR: 0.0036 - val_forelegL4: 0.0075 - val_forelegR4: 0.0080 - val_midlegL4: 0.0091 - val_midlegR4: 0.0100 - val_hindlegL4: 0.0090 - val_hindlegR4: 0.0099 - val_eyeL: 0.0057 - val_eyeR: 0.0062 - lr: 1.0000e-04 - 12s/epoch - 61ms/step Epoch 20/200 Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_070843.centered_instance.n=149/viz/validation.*.png 2023-10-16 07:13:00.626163: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: 1 } dim { size: 1024 } dim { size: 1024 } dim { size: 3 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 10 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 144 } dim { size: 144 } dim { size: 3 } } } 200/200 - 12s - loss: 0.0051 - head: 4.1717e-04 - thorax: 4.6850e-04 - abdomen: 0.0013 - wingL: 0.0025 - wingR: 0.0034 - forelegL4: 0.0072 - forelegR4: 0.0071 - midlegL4: 0.0083 - midlegR4: 0.0085 - hindlegL4: 0.0081 - hindlegR4: 0.0076 - eyeL: 0.0056 - eyeR: 0.0057 - val_loss: 0.0055 - val_head: 6.7695e-04 - val_thorax: 5.0953e-04 - val_abdomen: 0.0014 - val_wingL: 0.0038 - val_wingR: 0.0034 - val_forelegL4: 0.0077 - val_forelegR4: 0.0078 - val_midlegL4: 0.0081 - val_midlegR4: 0.0093 - val_hindlegL4: 0.0091 - val_hindlegR4: 0.0081 - val_eyeL: 0.0056 - val_eyeR: 0.0059 - lr: 1.0000e-04 - 12s/epoch - 62ms/step Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_070843.centered_instance.n=149/viz/validation.*.png Epoch 21/200 2023-10-16 07:13:12.832365: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: 1 } dim { size: 1024 } dim { size: 1024 } dim { size: 3 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 10 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 144 } dim { size: 144 } dim { size: 3 } } } 200/200 - 12s - loss: 0.0050 - head: 4.2067e-04 - thorax: 4.5395e-04 - abdomen: 0.0012 - wingL: 0.0023 - wingR: 0.0034 - forelegL4: 0.0072 - forelegR4: 0.0071 - midlegL4: 0.0081 - midlegR4: 0.0084 - hindlegL4: 0.0079 - hindlegR4: 0.0073 - eyeL: 0.0056 - eyeR: 0.0056 - val_loss: 0.0056 - val_head: 5.4406e-04 - val_thorax: 6.7325e-04 - val_abdomen: 0.0018 - val_wingL: 0.0036 - val_wingR: 0.0031 - val_forelegL4: 0.0078 - val_forelegR4: 0.0075 - val_midlegL4: 0.0090 - val_midlegR4: 0.0091 - val_hindlegL4: 0.0090 - val_hindlegR4: 0.0089 - val_eyeL: 0.0056 - val_eyeR: 0.0058 - lr: 1.0000e-04 - 12s/epoch - 61ms/step Epoch 22/200 Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_070843.centered_instance.n=149/viz/validation.*.png 2023-10-16 07:13:24.987091: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: 1 } dim { size: 1024 } dim { size: 1024 } dim { size: 3 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 10 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 144 } dim { size: 144 } dim { size: 3 } } } 200/200 - 12s - loss: 0.0050 - head: 4.2020e-04 - thorax: 4.7335e-04 - abdomen: 0.0013 - wingL: 0.0022 - wingR: 0.0032 - forelegL4: 0.0072 - forelegR4: 0.0070 - midlegL4: 0.0082 - midlegR4: 0.0083 - hindlegL4: 0.0077 - hindlegR4: 0.0072 - eyeL: 0.0057 - eyeR: 0.0056 - val_loss: 0.0058 - val_head: 5.7998e-04 - val_thorax: 4.9280e-04 - val_abdomen: 0.0016 - val_wingL: 0.0036 - val_wingR: 0.0030 - val_forelegL4: 0.0080 - val_forelegR4: 0.0082 - val_midlegL4: 0.0085 - val_midlegR4: 0.0097 - val_hindlegL4: 0.0100 - val_hindlegR4: 0.0097 - val_eyeL: 0.0058 - val_eyeR: 0.0061 - lr: 1.0000e-04 - 12s/epoch - 61ms/step Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_070843.centered_instance.n=149/viz/validation.*.png Epoch 23/200 2023-10-16 07:13:37.162642: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: 1 } dim { size: 1024 } dim { size: 1024 } dim { size: 3 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 10 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 144 } dim { size: 144 } dim { size: 3 } } } 200/200 - 12s - loss: 0.0049 - head: 4.1866e-04 - thorax: 4.2666e-04 - abdomen: 0.0013 - wingL: 0.0022 - wingR: 0.0032 - forelegL4: 0.0071 - forelegR4: 0.0069 - midlegL4: 0.0080 - midlegR4: 0.0081 - hindlegL4: 0.0077 - hindlegR4: 0.0072 - eyeL: 0.0055 - eyeR: 0.0056 - val_loss: 0.0056 - val_head: 4.9101e-04 - val_thorax: 4.3201e-04 - val_abdomen: 0.0014 - val_wingL: 0.0037 - val_wingR: 0.0032 - val_forelegL4: 0.0077 - val_forelegR4: 0.0074 - val_midlegL4: 0.0090 - val_midlegR4: 0.0090 - val_hindlegL4: 0.0090 - val_hindlegR4: 0.0088 - val_eyeL: 0.0060 - val_eyeR: 0.0061 - lr: 1.0000e-04 - 12s/epoch - 61ms/step Epoch 24/200 Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_070843.centered_instance.n=149/viz/validation.*.png 2023-10-16 07:13:49.387795: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: 1 } dim { size: 1024 } dim { size: 1024 } dim { size: 3 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 10 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 144 } dim { size: 144 } dim { size: 3 } } } 200/200 - 12s - loss: 0.0048 - head: 4.1083e-04 - thorax: 4.0826e-04 - abdomen: 0.0012 - wingL: 0.0019 - wingR: 0.0030 - forelegL4: 0.0071 - forelegR4: 0.0070 - midlegL4: 0.0080 - midlegR4: 0.0080 - hindlegL4: 0.0073 - hindlegR4: 0.0069 - eyeL: 0.0056 - eyeR: 0.0056 - val_loss: 0.0053 - val_head: 5.3566e-04 - val_thorax: 4.5736e-04 - val_abdomen: 0.0013 - val_wingL: 0.0035 - val_wingR: 0.0033 - val_forelegL4: 0.0073 - val_forelegR4: 0.0079 - val_midlegL4: 0.0083 - val_midlegR4: 0.0087 - val_hindlegL4: 0.0086 - val_hindlegR4: 0.0081 - val_eyeL: 0.0053 - val_eyeR: 0.0060 - lr: 1.0000e-04 - 12s/epoch - 62ms/step Epoch 25/200 Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_070843.centered_instance.n=149/viz/validation.*.png 2023-10-16 07:14:01.756489: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: 1 } dim { size: 1024 } dim { size: 1024 } dim { size: 3 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 10 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 144 } dim { size: 144 } dim { size: 3 } } } 200/200 - 12s - loss: 0.0047 - head: 4.3522e-04 - thorax: 4.3224e-04 - abdomen: 0.0012 - wingL: 0.0021 - wingR: 0.0030 - forelegL4: 0.0070 - forelegR4: 0.0069 - midlegL4: 0.0077 - midlegR4: 0.0078 - hindlegL4: 0.0072 - hindlegR4: 0.0068 - eyeL: 0.0055 - eyeR: 0.0055 - val_loss: 0.0052 - val_head: 5.3260e-04 - val_thorax: 4.2599e-04 - val_abdomen: 0.0011 - val_wingL: 0.0033 - val_wingR: 0.0030 - val_forelegL4: 0.0078 - val_forelegR4: 0.0075 - val_midlegL4: 0.0088 - val_midlegR4: 0.0091 - val_hindlegL4: 0.0073 - val_hindlegR4: 0.0075 - val_eyeL: 0.0055 - val_eyeR: 0.0061 - lr: 1.0000e-04 - 12s/epoch - 62ms/step Epoch 26/200 Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_070843.centered_instance.n=149/viz/validation.*.png 2023-10-16 07:14:14.094093: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: 1 } dim { size: 1024 } dim { size: 1024 } dim { size: 3 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 10 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 144 } dim { size: 144 } dim { size: 3 } } } 200/200 - 12s - loss: 0.0046 - head: 4.5979e-04 - thorax: 4.6782e-04 - abdomen: 0.0012 - wingL: 0.0019 - wingR: 0.0028 - forelegL4: 0.0070 - forelegR4: 0.0067 - midlegL4: 0.0074 - midlegR4: 0.0077 - hindlegL4: 0.0069 - hindlegR4: 0.0064 - eyeL: 0.0054 - eyeR: 0.0055 - val_loss: 0.0053 - val_head: 6.2012e-04 - val_thorax: 5.0883e-04 - val_abdomen: 0.0013 - val_wingL: 0.0038 - val_wingR: 0.0033 - val_forelegL4: 0.0074 - val_forelegR4: 0.0073 - val_midlegL4: 0.0088 - val_midlegR4: 0.0094 - val_hindlegL4: 0.0081 - val_hindlegR4: 0.0073 - val_eyeL: 0.0057 - val_eyeR: 0.0057 - lr: 1.0000e-04 - 12s/epoch - 61ms/step Epoch 27/200 Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_070843.centered_instance.n=149/viz/validation.*.png 2023-10-16 07:14:26.352288: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: 1 } dim { size: 1024 } dim { size: 1024 } dim { size: 3 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 10 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 144 } dim { size: 144 } dim { size: 3 } } } 200/200 - 12s - loss: 0.0046 - head: 4.1399e-04 - thorax: 4.1495e-04 - abdomen: 0.0012 - wingL: 0.0020 - wingR: 0.0028 - forelegL4: 0.0069 - forelegR4: 0.0067 - midlegL4: 0.0074 - midlegR4: 0.0075 - hindlegL4: 0.0068 - hindlegR4: 0.0064 - eyeL: 0.0053 - eyeR: 0.0054 - val_loss: 0.0056 - val_head: 4.5388e-04 - val_thorax: 4.2171e-04 - val_abdomen: 0.0015 - val_wingL: 0.0032 - val_wingR: 0.0035 - val_forelegL4: 0.0079 - val_forelegR4: 0.0072 - val_midlegL4: 0.0095 - val_midlegR4: 0.0100 - val_hindlegL4: 0.0094 - val_hindlegR4: 0.0082 - val_eyeL: 0.0058 - val_eyeR: 0.0062 - lr: 1.0000e-04 - 12s/epoch - 61ms/step Epoch 28/200 Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_070843.centered_instance.n=149/viz/validation.*.png 2023-10-16 07:14:38.578576: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: 1 } dim { size: 1024 } dim { size: 1024 } dim { size: 3 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 10 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 144 } dim { size: 144 } dim { size: 3 } } } 200/200 - 12s - loss: 0.0046 - head: 4.3010e-04 - thorax: 4.7511e-04 - abdomen: 0.0012 - wingL: 0.0019 - wingR: 0.0027 - forelegL4: 0.0069 - forelegR4: 0.0068 - midlegL4: 0.0074 - midlegR4: 0.0074 - hindlegL4: 0.0069 - hindlegR4: 0.0065 - eyeL: 0.0053 - eyeR: 0.0054 - val_loss: 0.0051 - val_head: 5.2094e-04 - val_thorax: 3.8217e-04 - val_abdomen: 8.2952e-04 - val_wingL: 0.0036 - val_wingR: 0.0027 - val_forelegL4: 0.0078 - val_forelegR4: 0.0071 - val_midlegL4: 0.0081 - val_midlegR4: 0.0080 - val_hindlegL4: 0.0084 - val_hindlegR4: 0.0083 - val_eyeL: 0.0053 - val_eyeR: 0.0056 - lr: 1.0000e-04 - 12s/epoch - 61ms/step Epoch 29/200 Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_070843.centered_instance.n=149/viz/validation.*.png 2023-10-16 07:14:50.993716: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: 1 } dim { size: 1024 } dim { size: 1024 } dim { size: 3 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 10 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 144 } dim { size: 144 } dim { size: 3 } } } 200/200 - 12s - loss: 0.0044 - head: 4.1449e-04 - thorax: 4.1248e-04 - abdomen: 0.0011 - wingL: 0.0018 - wingR: 0.0025 - forelegL4: 0.0069 - forelegR4: 0.0067 - midlegL4: 0.0069 - midlegR4: 0.0070 - hindlegL4: 0.0065 - hindlegR4: 0.0061 - eyeL: 0.0052 - eyeR: 0.0052 - val_loss: 0.0053 - val_head: 5.5684e-04 - val_thorax: 4.1731e-04 - val_abdomen: 0.0016 - val_wingL: 0.0038 - val_wingR: 0.0029 - val_forelegL4: 0.0068 - val_forelegR4: 0.0078 - val_midlegL4: 0.0085 - val_midlegR4: 0.0088 - val_hindlegL4: 0.0084 - val_hindlegR4: 0.0087 - val_eyeL: 0.0052 - val_eyeR: 0.0055 - lr: 1.0000e-04 - 12s/epoch - 62ms/step Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_070843.centered_instance.n=149/viz/validation.*.png Epoch 30/200 2023-10-16 07:15:03.164209: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: 1 } dim { size: 1024 } dim { size: 1024 } dim { size: 3 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 10 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 144 } dim { size: 144 } dim { size: 3 } } } 200/200 - 12s - loss: 0.0043 - head: 3.9878e-04 - thorax: 3.9097e-04 - abdomen: 0.0011 - wingL: 0.0019 - wingR: 0.0025 - forelegL4: 0.0068 - forelegR4: 0.0066 - midlegL4: 0.0067 - midlegR4: 0.0068 - hindlegL4: 0.0063 - hindlegR4: 0.0060 - eyeL: 0.0051 - eyeR: 0.0051 - val_loss: 0.0053 - val_head: 4.8584e-04 - val_thorax: 4.9301e-04 - val_abdomen: 0.0013 - val_wingL: 0.0040 - val_wingR: 0.0026 - val_forelegL4: 0.0072 - val_forelegR4: 0.0081 - val_midlegL4: 0.0090 - val_midlegR4: 0.0090 - val_hindlegL4: 0.0078 - val_hindlegR4: 0.0072 - val_eyeL: 0.0054 - val_eyeR: 0.0060 - lr: 1.0000e-04 - 12s/epoch - 61ms/step Epoch 31/200 Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_070843.centered_instance.n=149/viz/validation.*.png 2023-10-16 07:15:15.299034: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: 1 } dim { size: 1024 } dim { size: 1024 } dim { size: 3 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 10 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 144 } dim { size: 144 } dim { size: 3 } } } 200/200 - 12s - loss: 0.0043 - head: 4.0585e-04 - thorax: 3.9195e-04 - abdomen: 0.0011 - wingL: 0.0018 - wingR: 0.0026 - forelegL4: 0.0066 - forelegR4: 0.0065 - midlegL4: 0.0067 - midlegR4: 0.0068 - hindlegL4: 0.0063 - hindlegR4: 0.0061 - eyeL: 0.0050 - eyeR: 0.0050 - val_loss: 0.0051 - val_head: 4.4100e-04 - val_thorax: 4.1315e-04 - val_abdomen: 0.0011 - val_wingL: 0.0034 - val_wingR: 0.0031 - val_forelegL4: 0.0079 - val_forelegR4: 0.0075 - val_midlegL4: 0.0077 - val_midlegR4: 0.0084 - val_hindlegL4: 0.0071 - val_hindlegR4: 0.0070 - val_eyeL: 0.0058 - val_eyeR: 0.0058 - lr: 1.0000e-04 - 12s/epoch - 61ms/step Epoch 32/200 Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_070843.centered_instance.n=149/viz/validation.*.png 2023-10-16 07:15:27.416545: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: 1 } dim { size: 1024 } dim { size: 1024 } dim { size: 3 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 10 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 144 } dim { size: 144 } dim { size: 3 } } } 200/200 - 12s - loss: 0.0042 - head: 4.0799e-04 - thorax: 3.5961e-04 - abdomen: 0.0011 - wingL: 0.0017 - wingR: 0.0024 - forelegL4: 0.0067 - forelegR4: 0.0065 - midlegL4: 0.0065 - midlegR4: 0.0067 - hindlegL4: 0.0062 - hindlegR4: 0.0060 - eyeL: 0.0050 - eyeR: 0.0050 - val_loss: 0.0052 - val_head: 3.9909e-04 - val_thorax: 6.4130e-04 - val_abdomen: 0.0014 - val_wingL: 0.0039 - val_wingR: 0.0023 - val_forelegL4: 0.0075 - val_forelegR4: 0.0076 - val_midlegL4: 0.0093 - val_midlegR4: 0.0085 - val_hindlegL4: 0.0079 - val_hindlegR4: 0.0082 - val_eyeL: 0.0051 - val_eyeR: 0.0050 - lr: 1.0000e-04 - 12s/epoch - 60ms/step Epoch 33/200 Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_070843.centered_instance.n=149/viz/validation.*.png 2023-10-16 07:15:39.671708: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: 1 } dim { size: 1024 } dim { size: 1024 } dim { size: 3 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 10 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 144 } dim { size: 144 } dim { size: 3 } } } 200/200 - 12s - loss: 0.0040 - head: 3.8553e-04 - thorax: 3.5829e-04 - abdomen: 0.0010 - wingL: 0.0017 - wingR: 0.0023 - forelegL4: 0.0065 - forelegR4: 0.0064 - midlegL4: 0.0062 - midlegR4: 0.0063 - hindlegL4: 0.0058 - hindlegR4: 0.0056 - eyeL: 0.0047 - eyeR: 0.0048 - val_loss: 0.0051 - val_head: 4.5146e-04 - val_thorax: 4.6433e-04 - val_abdomen: 0.0013 - val_wingL: 0.0032 - val_wingR: 0.0027 - val_forelegL4: 0.0077 - val_forelegR4: 0.0075 - val_midlegL4: 0.0078 - val_midlegR4: 0.0081 - val_hindlegL4: 0.0079 - val_hindlegR4: 0.0070 - val_eyeL: 0.0058 - val_eyeR: 0.0057 - lr: 1.0000e-04 - 12s/epoch - 62ms/step Epoch 34/200 Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_070843.centered_instance.n=149/viz/validation.*.png 2023-10-16 07:15:51.898574: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: 1 } dim { size: 1024 } dim { size: 1024 } dim { size: 3 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 10 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 144 } dim { size: 144 } dim { size: 3 } } } 200/200 - 12s - loss: 0.0040 - head: 3.9407e-04 - thorax: 3.7663e-04 - abdomen: 0.0010 - wingL: 0.0016 - wingR: 0.0022 - forelegL4: 0.0064 - forelegR4: 0.0063 - midlegL4: 0.0061 - midlegR4: 0.0062 - hindlegL4: 0.0059 - hindlegR4: 0.0057 - eyeL: 0.0046 - eyeR: 0.0046 - val_loss: 0.0053 - val_head: 4.3299e-04 - val_thorax: 4.6334e-04 - val_abdomen: 0.0011 - val_wingL: 0.0031 - val_wingR: 0.0026 - val_forelegL4: 0.0064 - val_forelegR4: 0.0075 - val_midlegL4: 0.0095 - val_midlegR4: 0.0089 - val_hindlegL4: 0.0092 - val_hindlegR4: 0.0081 - val_eyeL: 0.0057 - val_eyeR: 0.0056 - lr: 1.0000e-04 - 12s/epoch - 61ms/step Epoch 35/200 Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_070843.centered_instance.n=149/viz/validation.*.png 2023-10-16 07:16:04.141239: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: 1 } dim { size: 1024 } dim { size: 1024 } dim { size: 3 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 10 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 144 } dim { size: 144 } dim { size: 3 } } } 200/200 - 12s - loss: 0.0038 - head: 3.8693e-04 - thorax: 3.5724e-04 - abdomen: 0.0010 - wingL: 0.0016 - wingR: 0.0022 - forelegL4: 0.0063 - forelegR4: 0.0061 - midlegL4: 0.0059 - midlegR4: 0.0059 - hindlegL4: 0.0056 - hindlegR4: 0.0054 - eyeL: 0.0043 - eyeR: 0.0044 - val_loss: 0.0053 - val_head: 4.1844e-04 - val_thorax: 3.9218e-04 - val_abdomen: 0.0010 - val_wingL: 0.0029 - val_wingR: 0.0027 - val_forelegL4: 0.0078 - val_forelegR4: 0.0069 - val_midlegL4: 0.0086 - val_midlegR4: 0.0088 - val_hindlegL4: 0.0079 - val_hindlegR4: 0.0071 - val_eyeL: 0.0067 - val_eyeR: 0.0070 - lr: 1.0000e-04 - 12s/epoch - 61ms/step Epoch 36/200 Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_070843.centered_instance.n=149/viz/validation.*.png 2023-10-16 07:16:16.566185: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: 1 } dim { size: 1024 } dim { size: 1024 } dim { size: 3 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" model: "0" num_cores: 10 environment { key: "cpu_instruction_set" value: "ARM NEON" } environment { key: "eigen" value: "3.4.90" } l1_cache_size: 16384 l2_cache_size: 524288 l3_cache_size: 524288 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 144 } dim { size: 144 } dim { size: 3 } } } 200/200 - 12s - loss: 0.0037 - head: 3.9279e-04 - thorax: 3.7463e-04 - abdomen: 0.0010 - wingL: 0.0015 - wingR: 0.0020 - forelegL4: 0.0061 - forelegR4: 0.0060 - midlegL4: 0.0055 - midlegR4: 0.0057 - hindlegL4: 0.0055 - hindlegR4: 0.0054 - eyeL: 0.0044 - eyeR: 0.0044 - val_loss: 0.0048 - val_head: 3.7802e-04 - val_thorax: 3.4178e-04 - val_abdomen: 0.0010 - val_wingL: 0.0037 - val_wingR: 0.0025 - val_forelegL4: 0.0068 - val_forelegR4: 0.0080 - val_midlegL4: 0.0080 - val_midlegR4: 0.0078 - val_hindlegL4: 0.0065 - val_hindlegR4: 0.0068 - val_eyeL: 0.0052 - val_eyeR: 0.0058 - lr: 1.0000e-04 - 12s/epoch - 62ms/step Epoch 37/200 Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_063130.centroid.n=149/viz/validation.*.png Polling: /Users/liezlmaree/Projects/sleap-datasets/drosophila-melanogaster-courtship/models/231016_070843.centered_instance.n=149/viz/validation.*.pngHow to reproduce