agosztolai
commented
5 months ago Open agosztolai opened 5 months ago
agosztolai
commented
5 months ago agosztolai
commented
4 months ago Ok, I think I have figured this out. The issue seems to be that I have defined a skeleton with two connected components. I noticed this because I tried to use the same pipeline as in the tutorial. The tutorial uses a multi-animal model in which the "Run" button is disabled when there are multiple connected components. However, the single-animal model trains well with skeletons having multiple connected components. Except that at inference time, no predicted instances are returned.
Could this be the issue that the model only accepts a single connected component? If yes, it would be nice to include a checkpoint for this in the code.
Lateef-Saheed
commented
4 months ago Hello! I am having a similar issue as you. Just curious if in your training settings, you have the "rotate" setting in the Augmentation section of the Single Instance Model configuration tab checked?
agosztolai
commented
4 months ago Yes, for me it works with ‘rotate’ on. Perhaps try rotate = off. If that works, maybe report the bug?
On 25 Jun 2024, at 20:14, Lateef-Saheed @.***> wrote:
Hello! I am having a similar issue as you. Just curious if in your training settings, you have the "rotate" setting in the Augmentation section of the Single Instance Model configuration tab checked?
— Reply to this email directly, view it on GitHub https://github.com/talmolab/sleap/issues/1821#issuecomment-2189659113, or unsubscribe https://github.com/notifications/unsubscribe-auth/AK6WK5D3VZLGACSWFN2PP2LZJGXP7AVCNFSM6AAAAABJVZQP2OVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCOBZGY2TSMJRGM. You are receiving this because you authored the thread.
Lateef-Saheed
commented
4 months ago Sounds good, I tried turning off rotation and that got me inferences, albeit ones that weren't accurate. Thanks for the confirmation!
talmo
commented
4 months ago Hi folks,
Apologies for the delay! We're a bit behind on support responses at the moment.
Neither rotation nor the skeleton edge configuration should make any difference to the single animal model.
If you're not getting predictions, the most likely cause is that the model is underperforming. The easiest thing to try is to just add more labels, but it might be that further model parameter tuning would help.
Turning off rotation will cause the model to seriously overfit, which means it'll work on images very similar to those in your training data, but fail to generalize to new ones.
It sounds like @agosztolai has a working solution, but @Lateef-Saheed if you don't mind creating a new Discussion with some information about your project, we'd be happy to help!
Lateef-Saheed
commented
3 months ago Noted! We ended up labeling more frames and using the "Latest" model instead of "Best" and it ended up giving us predictions. I am curious about your comment about rotation. What exactly does the rotation augmentation feature mean and how would not selecting it result in failing to generalize to new videos? Thanks for the help!
On Fri, Jul 19, 2024 at 7:02 PM Talmo Pereira @.***> wrote:
Hi folks,
Apologies for the delay! We're a bit behind on support responses at the moment.
Neither rotation nor the skeleton edge configuration should make any difference to the single animal model.
If you're not getting predictions, the most likely cause is that the model is underperforming. The easiest thing to try is to just add more labels, but it might be that further model parameter tuning would help.
Turning off rotation will cause the model to seriously overfit, which means it'll work on images very similar to those in your training data, but fail to generalize to new ones.
It sounds like @agosztolai https://github.com/agosztolai has a working solution, but @Lateef-Saheed https://github.com/Lateef-Saheed if you don't mind creating a new Discussion https://github.com/talmolab/sleap/discussions with some information about your project, we'd be happy to help!
— Reply to this email directly, view it on GitHub https://github.com/talmolab/sleap/issues/1821#issuecomment-2240781386, or unsubscribe https://github.com/notifications/unsubscribe-auth/BG7CTUA3QCZDTGFHPUO23JTZNGSKFAVCNFSM6AAAAABJVZQP2OVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENBQG44DCMZYGY . You are receiving this because you were mentioned.Message ID: @.***>
Bug description
After training, the inference step outputs no labels.
Expected behaviour
I expect labels (yellow markers) to emerge in the GUI after inference.
Actual behaviour
Nothing happens.
Your personal set up
Environment packages
``` # paste output of `pip freeze` or `conda list` here ``` # Name Version Build Channel abseil-cpp 20211102.0 he4e09e4_3 conda-forge absl-py 1.4.0 pypi_0 pypi aiohttp 3.9.5 py39h17cfd9d_0 conda-forge aiosignal 1.3.1 pyhd8ed1ab_0 conda-forge aom 3.5.0 h7ea286d_0 conda-forge astunparse 1.6.3 pyhd8ed1ab_0 conda-forge async-timeout 4.0.3 pyhd8ed1ab_0 conda-forge attrs 23.2.0 pyh71513ae_0 conda-forge blinker 1.8.2 pyhd8ed1ab_0 conda-forge blosc 1.21.5 hc338f07_0 conda-forge brotli 1.0.9 h1a8c8d9_9 conda-forge brotli-bin 1.0.9 h1a8c8d9_9 conda-forge brotli-python 1.0.9 py39h23fbdae_9 conda-forge brunsli 0.1 h9f76cd9_0 conda-forge bzip2 1.0.8 h93a5062_5 conda-forge c-ares 1.28.1 h93a5062_0 conda-forge c-blosc2 2.12.0 ha57e6be_0 conda-forge ca-certificates 2024.6.2 hf0a4a13_0 conda-forge cached-property 1.5.2 hd8ed1ab_1 conda-forge cached_property 1.5.2 pyha770c72_1 conda-forge cachetools 5.3.1 pypi_0 pypi cairo 1.16.0 had492bb_1012 conda-forge cattrs 1.1.1 pyhd8ed1ab_0 conda-forge certifi 2024.6.2 pyhd8ed1ab_0 conda-forge cffi 1.16.0 py39he153c15_0 conda-forge cfitsio 4.2.0 h2f961c4_0 conda-forge charls 2.3.4 hbdafb3b_0 conda-forge charset-normalizer 3.2.0 pypi_0 pypi click 8.1.7 unix_pyh707e725_0 conda-forge contourpy 1.2.1 py39h48c5dd5_0 conda-forge cryptography 39.0.0 py39haa0b8cc_0 conda-forge cycler 0.12.1 pyhd8ed1ab_0 conda-forge dav1d 1.2.1 hb547adb_0 conda-forge efficientnet 1.0.0 pypi_0 pypi expat 2.6.2 hebf3989_0 conda-forge ffmpeg 4.4.2 gpl_hf318d42_112 conda-forge font-ttf-dejavu-sans-mono 2.37 hab24e00_0 conda-forge font-ttf-inconsolata 3.000 h77eed37_0 conda-forge font-ttf-source-code-pro 2.038 h77eed37_0 conda-forge font-ttf-ubuntu 0.83 h77eed37_2 conda-forge fontconfig 2.14.2 h82840c6_0 conda-forge fonts-conda-ecosystem 1 0 conda-forge fonts-conda-forge 1 0 conda-forge fonttools 4.53.0 py39hfea33bf_0 conda-forge freetype 2.12.1 hadb7bae_2 conda-forge frozenlist 1.4.1 py39h17cfd9d_0 conda-forge gast 0.4.0 pyh9f0ad1d_0 conda-forge geos 3.12.1 h965bd2d_0 conda-forge gettext 0.22.5 h8fbad5d_2 conda-forge gettext-tools 0.22.5 h8fbad5d_2 conda-forge giflib 5.2.2 h93a5062_0 conda-forge glib 2.80.2 h59d46d9_1 conda-forge glib-tools 2.80.2 h8ba3eef_1 conda-forge gmp 6.3.0 hebf3989_1 conda-forge gnutls 3.7.9 hd26332c_0 conda-forge google-auth 2.23.0 pypi_0 pypi google-auth-oauthlib 0.4.6 pyhd8ed1ab_0 conda-forge google-pasta 0.2.0 pyh8c360ce_0 conda-forge graphite2 1.3.13 hebf3989_1003 conda-forge grpc-cpp 1.46.3 hacd037c_3 conda-forge grpcio 1.58.0 pypi_0 pypi gst-plugins-base 1.22.9 h09b4b5e_1 conda-forge gstreamer 1.22.9 h551c6ff_1 conda-forge h5py 3.8.0 nompi_py39hc9149d8_100 conda-forge harfbuzz 5.3.0 hddbc195_0 conda-forge hdf5 1.12.2 nompi_h55deafc_101 conda-forge hdmf 3.9.0 pypi_0 pypi icu 70.1 h6b3803e_0 conda-forge idna 3.4 pypi_0 pypi image-classifiers 1.0.0 pypi_0 pypi imagecodecs 2022.9.26 py39hd7f743f_4 conda-forge imageio 2.34.1 pyh4b66e23_0 conda-forge imgaug 0.4.0 pyhd8ed1ab_1 conda-forge imgstore 0.2.9 pypi_0 pypi importlib-metadata 7.1.0 pyha770c72_0 conda-forge importlib-resources 6.4.0 pyhd8ed1ab_0 conda-forge importlib_resources 6.4.0 pyhd8ed1ab_0 conda-forge jasper 2.0.33 hc3cd1e9_1 conda-forge joblib 1.4.2 pyhd8ed1ab_0 conda-forge jpeg 9e h1a8c8d9_3 conda-forge jsmin 3.0.1 pyhd8ed1ab_0 conda-forge jsonpickle 1.2 py_0 conda-forge jsonschema 4.19.0 pypi_0 pypi jsonschema-specifications 2023.7.1 pypi_0 pypi jxrlib 1.1 h93a5062_3 conda-forge keras 2.9.0 pyhd8ed1ab_0 conda-forge keras-applications 1.0.8 pypi_0 pypi keras-preprocessing 1.1.2 pyhd8ed1ab_0 conda-forge kiwisolver 1.4.5 py39hbd775c9_1 conda-forge krb5 1.20.1 h127bd45_0 conda-forge lame 3.100 h1a8c8d9_1003 conda-forge lazy_loader 0.4 pyhd8ed1ab_0 conda-forge lcms2 2.14 h8193b64_0 conda-forge lerc 4.0.0 h9a09cb3_0 conda-forge libabseil 20211102.0 cxx17_h28b99d4_3 conda-forge libaec 1.1.3 hebf3989_0 conda-forge libasprintf 0.22.5 h8fbad5d_2 conda-forge libasprintf-devel 0.22.5 h8fbad5d_2 conda-forge libavif 0.11.1 h9f83d30_2 conda-forge libblas 3.9.0 20_osxarm64_openblas conda-forge libbrotlicommon 1.0.9 h1a8c8d9_9 conda-forge libbrotlidec 1.0.9 h1a8c8d9_9 conda-forge libbrotlienc 1.0.9 h1a8c8d9_9 conda-forge libcblas 3.9.0 20_osxarm64_openblas conda-forge libclang 16.0.6 pypi_0 pypi libclang13 14.0.6 default_hc7183e1_1 conda-forge libcurl 7.87.0 hbe9bab4_0 conda-forge libcxx 17.0.6 h5f092b4_0 conda-forge libdeflate 1.14 h1a8c8d9_0 conda-forge libedit 3.1.20191231 hc8eb9b7_2 conda-forge libev 4.33 h93a5062_2 conda-forge libexpat 2.6.2 hebf3989_0 conda-forge libffi 3.4.2 h3422bc3_5 conda-forge libgettextpo 0.22.5 h8fbad5d_2 conda-forge libgettextpo-devel 0.22.5 h8fbad5d_2 conda-forge libgfortran 5.0.0 13_2_0_hd922786_3 conda-forge libgfortran5 13.2.0 hf226fd6_3 conda-forge libglib 2.80.2 h59d46d9_1 conda-forge libiconv 1.17 h0d3ecfb_2 conda-forge libidn2 2.3.7 h93a5062_0 conda-forge libintl 0.22.5 h8fbad5d_2 conda-forge libintl-devel 0.22.5 h8fbad5d_2 conda-forge liblapack 3.9.0 20_osxarm64_openblas conda-forge liblapacke 3.9.0 20_osxarm64_openblas conda-forge libllvm14 14.0.6 hd1a9a77_4 conda-forge libnghttp2 1.51.0 hd184df1_0 conda-forge libogg 1.3.4 h27ca646_1 conda-forge libopenblas 0.3.25 openmp_h6c19121_0 conda-forge libopencv 4.6.0 py39he1c1adf_3 conda-forge libopus 1.3.1 h27ca646_1 conda-forge libpng 1.6.43 h091b4b1_0 conda-forge libpq 15.1 hbce9e56_3 conda-forge libprotobuf 3.20.3 hb5ab8b9_0 conda-forge libsodium 1.0.18 h27ca646_1 conda-forge libsqlite 3.46.0 hfb93653_0 conda-forge libssh2 1.10.0 hb80f160_3 conda-forge libtasn1 4.19.0 h1a8c8d9_0 conda-forge libtiff 4.4.0 heb92581_5 conda-forge libunistring 0.9.10 h3422bc3_0 conda-forge libvorbis 1.3.7 h9f76cd9_0 conda-forge libvpx 1.11.0 hc470f4d_3 conda-forge libwebp-base 1.4.0 h93a5062_0 conda-forge libxcb 1.13 h9b22ae9_1004 conda-forge libxml2 2.10.3 h67585b2_4 conda-forge libxslt 1.1.37 h1bd8bc4_0 conda-forge libzlib 1.3.1 hfb2fe0b_1 conda-forge libzopfli 1.0.3 h9f76cd9_0 conda-forge llvm-openmp 18.1.7 hde57baf_0 conda-forge lz4-c 1.9.4 hb7217d7_0 conda-forge markdown 3.4.4 pypi_0 pypi markdown-it-py 3.0.0 pyhd8ed1ab_0 conda-forge markupsafe 2.1.3 pypi_0 pypi matplotlib-base 3.8.4 py39h15359f4_2 conda-forge mdurl 0.1.2 pyhd8ed1ab_0 conda-forge multidict 6.0.5 py39h02fc5c5_0 conda-forge munkres 1.1.4 pyh9f0ad1d_0 conda-forge mysql-common 8.0.32 hab468bb_0 conda-forge mysql-libs 8.0.32 hea58576_0 conda-forge ncurses 6.5 hb89a1cb_0 conda-forge ndx-pose 0.1.1 pypi_0 pypi nettle 3.9.1 h40ed0f5_0 conda-forge networkx 3.2.1 pyhd8ed1ab_0 conda-forge nixio 1.5.3 pypi_0 pypi nspr 4.35 hb7217d7_0 conda-forge nss 3.101 hc42bcbf_0 conda-forge numpy 1.22.4 py39h7df2422_0 conda-forge oauthlib 3.2.2 pyhd8ed1ab_0 conda-forge opencv 4.6.0 py39hdf13c20_3 conda-forge openh264 2.3.1 hb7217d7_2 conda-forge openjpeg 2.5.0 h5d4e404_1 conda-forge openssl 1.1.1w h53f4e23_0 conda-forge opt_einsum 3.3.0 pyhc1e730c_2 conda-forge p11-kit 0.24.1 h29577a5_0 conda-forge packaging 24.1 pyhd8ed1ab_0 conda-forge pandas 2.2.2 py39h998126f_1 conda-forge patsy 0.5.6 pyhd8ed1ab_0 conda-forge pcre2 10.44 h297a79d_0 conda-forge pillow 9.2.0 py39h139752e_3 conda-forge pip 24.0 pyhd8ed1ab_0 conda-forge pixman 0.43.4 hebf3989_0 conda-forge protobuf 3.19.6 pypi_0 pypi psutil 5.9.8 py39h17cfd9d_0 conda-forge pthread-stubs 0.4 h27ca646_1001 conda-forge py-opencv 4.6.0 py39hfa6204d_3 conda-forge pyasn1 0.5.0 pypi_0 pypi pyasn1-modules 0.3.0 pypi_0 pypi pycparser 2.22 pyhd8ed1ab_0 conda-forge pygments 2.18.0 pyhd8ed1ab_0 conda-forge pyjwt 2.8.0 pyhd8ed1ab_1 conda-forge pykalman 0.9.7 pyhd8ed1ab_0 conda-forge pynwb 2.5.0 pypi_0 pypi pyopenssl 23.2.0 pyhd8ed1ab_1 conda-forge pyparsing 3.1.2 pyhd8ed1ab_0 conda-forge pyside2 5.15.8 py39h0adaba8_2 conda-forge pysocks 1.7.1 pyha2e5f31_6 conda-forge python 3.9.15 h2d96c93_0_cpython conda-forge python-dateutil 2.9.0 pyhd8ed1ab_0 conda-forge python-flatbuffers 1.12 pyhd8ed1ab_1 conda-forge python-rapidjson 1.17 py39hbf7db11_0 conda-forge python-tzdata 2024.1 pyhd8ed1ab_0 conda-forge python_abi 3.9 4_cp39 conda-forge pytz 2024.1 pyhd8ed1ab_0 conda-forge pyu2f 0.1.5 pyhd8ed1ab_0 conda-forge pywavelets 1.4.1 py39hf4a74a7_1 conda-forge pyyaml 6.0.1 py39h0f82c59_1 conda-forge pyzmq 26.0.3 py39he7f0319_0 conda-forge qimage2ndarray 1.10.0 pypi_0 pypi qt-main 5.15.8 hfe8d25c_6 conda-forge qtpy 2.4.1 pyhd8ed1ab_0 conda-forge re2 2022.06.01 h9a09cb3_1 conda-forge readline 8.2 h92ec313_1 conda-forge referencing 0.30.2 pypi_0 pypi requests 2.31.0 pypi_0 pypi requests-oauthlib 1.3.1 pypi_0 pypi rich 13.7.1 pyhd8ed1ab_0 conda-forge rpds-py 0.10.3 pypi_0 pypi rsa 4.9 pyhd8ed1ab_0 conda-forge ruamel-yaml 0.17.32 pypi_0 pypi ruamel-yaml-clib 0.2.7 pypi_0 pypi scikit-image 0.22.0 py39hf8cecc8_2 conda-forge scikit-learn 1.0 py39h12ba089_1 conda-forge scikit-video 1.1.11 pyh24bf2e0_0 conda-forge scipy 1.9.0 py39h14896cb_0 conda-forge seaborn 0.13.2 hd8ed1ab_2 conda-forge seaborn-base 0.13.2 pyhd8ed1ab_2 conda-forge segmentation-models 1.0.1 pypi_0 pypi setuptools 70.0.0 pyhd8ed1ab_0 conda-forge shapely 2.0.4 py39h8b557c8_1 conda-forge six 1.15.0 pypi_0 pypi sleap 1.3.3 pypi_0 pypi snappy 1.1.10 hd04f947_1 conda-forge sqlite 3.46.0 h5838104_0 conda-forge statsmodels 0.14.2 py39h161d348_0 conda-forge svt-av1 1.4.1 h7ea286d_0 conda-forge tensorboard 2.9.1 pypi_0 pypi tensorboard-data-server 0.6.1 py39haa0b8cc_4 conda-forge tensorboard-plugin-wit 1.8.1 pyhd8ed1ab_0 conda-forge tensorflow 2.9.1 cpu_py39h2839aeb_0 conda-forge tensorflow-base 2.9.1 cpu_py39ha1ad4ae_0 conda-forge tensorflow-estimator 2.9.1 cpu_py39h7b621ec_0 conda-forge tensorflow-hub 0.12.0 pyhca92ed8_0 conda-forge tensorflow-macos 2.9.2 pypi_0 pypi tensorflow-metal 0.5.0 pypi_0 pypi termcolor 2.3.0 pypi_0 pypi threadpoolctl 3.5.0 pyhc1e730c_0 conda-forge tifffile 2022.10.10 pyhd8ed1ab_0 conda-forge tk 8.6.13 h5083fa2_1 conda-forge typing-extensions 4.12.2 hd8ed1ab_0 conda-forge typing_extensions 4.12.2 pyha770c72_0 conda-forge tzdata 2024a h0c530f3_0 conda-forge tzlocal 5.0.1 pypi_0 pypi unicodedata2 15.1.0 py39h0f82c59_0 conda-forge urllib3 1.26.16 pypi_0 pypi werkzeug 2.3.7 pypi_0 pypi wheel 0.43.0 pyhd8ed1ab_1 conda-forge wrapt 1.15.0 pypi_0 pypi x264 1!164.3095 h57fd34a_2 conda-forge x265 3.5 hbc6ce65_3 conda-forge xorg-libxau 1.0.11 hb547adb_0 conda-forge xorg-libxdmcp 1.1.3 h27ca646_0 conda-forge xz 5.2.6 h57fd34a_0 conda-forge yaml 0.2.5 h3422bc3_2 conda-forge yarl 1.9.4 py39h17cfd9d_0 conda-forge zeromq 4.3.5 hebf3989_1 conda-forge zfp 1.0.1 ha8f4885_0 conda-forge zipp 3.19.2 pyhd8ed1ab_0 conda-forge zlib-ng 2.0.7 h1a8c8d9_0 conda-forge zstd 1.5.6 hb46c0d2_0 conda-forgeLogs
(sleap) adamgosztolai@Adams-MacBook-Pro-2 data % sleap-label Saving config: /Users/adamgosztolai/.sleap/1.3.3/preferences.yaml Restoring GUI state... Software versions: SLEAP: 1.3.3 TensorFlow: 2.9.2 Numpy: 1.22.4 Python: 3.9.15 OS: macOS-14.5-arm64-arm-64bit Happy SLEAPing! :) qt.qpa.fonts: Populating font family aliases took 155 ms. Replace uses of missing font family ".AppleSystemUIFont" with one that exists to avoid this cost. Resetting monitor window. Polling: /Users/adamgosztolai/Documents/GitHub/large_kinematic_model/preprocessing/sleap/models/calibration240621_144245.single_instance.n=1/viz/validation.*.png Start training single_instance... ['sleap-train', '/var/folders/3n/71gnzd013y5f29s9t0tyyrwh0000gn/T/tmpnecxtno4/240621_144245_training_job.json', '/Users/adamgosztolai/Documents/GitHub/large_kinematic_model/preprocessing/sleap/labels.v001.slp', '--zmq', '--save_viz'] INFO:sleap.nn.training:Versions: SLEAP: 1.3.3 TensorFlow: 2.9.2 Numpy: 1.22.4 Python: 3.9.15 OS: macOS-14.5-arm64-arm-64bit INFO:sleap.nn.training:Training labels file: /Users/adamgosztolai/Documents/GitHub/large_kinematic_model/preprocessing/sleap/labels.v001.slp INFO:sleap.nn.training:Training profile: /var/folders/3n/71gnzd013y5f29s9t0tyyrwh0000gn/T/tmpnecxtno4/240621_144245_training_job.json INFO:sleap.nn.training: INFO:sleap.nn.training:Arguments: INFO:sleap.nn.training:{ "training_job_path": "/var/folders/3n/71gnzd013y5f29s9t0tyyrwh0000gn/T/tmpnecxtno4/240621_144245_training_job.json", "labels_path": "/Users/adamgosztolai/Documents/GitHub/large_kinematic_model/preprocessing/sleap/labels.v001.slp", "video_paths": [ "" ], "val_labels": null, "test_labels": null, "base_checkpoint": null, "tensorboard": false, "save_viz": true, "zmq": true, "run_name": "", "prefix": "", "suffix": "", "cpu": false, "first_gpu": false, "last_gpu": false, "gpu": "auto" } INFO:sleap.nn.training: INFO:sleap.nn.training:Training job: INFO:sleap.nn.training:{ "data": { "labels": { "training_labels": null, "validation_labels": null, "validation_fraction": 0.1, "test_labels": null, "split_by_inds": false, "training_inds": null, "validation_inds": null, "test_inds": null, "search_path_hints": [], "skeletons": [] }, "preprocessing": { "ensure_rgb": false, "ensure_grayscale": false, "imagenet_mode": null, "input_scaling": 1.0, "pad_to_stride": null, "resize_and_pad_to_target": true, "target_height": null, "target_width": null }, "instance_cropping": { "center_on_part": null, "crop_size": null, "crop_size_detection_padding": 16 } }, "model": { "backbone": { "leap": null, "unet": { "stem_stride": null, "max_stride": 16, "output_stride": 2, "filters": 16, "filters_rate": 2.0, "middle_block": true, "up_interpolate": true, "stacks": 1 }, "hourglass": null, "resnet": null, "pretrained_encoder": null }, "heads": { "single_instance": { "part_names": null, "sigma": 2.5, "output_stride": 2, "loss_weight": 1.0, "offset_refinement": false }, "centroid": null, "centered_instance": null, "multi_instance": null, "multi_class_bottomup": null, "multi_class_topdown": null }, "base_checkpoint": null }, "optimization": { "preload_data": true, "augmentation_config": { "rotate": true, "rotation_min_angle": -15.0, "rotation_max_angle": 15.0, "translate": false, "translate_min": -5, "translate_max": 5, "scale": false, "scale_min": 0.9, "scale_max": 1.1, "uniform_noise": false, "uniform_noise_min_val": 0.0, "uniform_noise_max_val": 10.0, "gaussian_noise": false, "gaussian_noise_mean": 5.0, "gaussian_noise_stddev": 1.0, "contrast": false, "contrast_min_gamma": 0.5, "contrast_max_gamma": 2.0, "brightness": false, "brightness_min_val": 0.0, "brightness_max_val": 10.0, "random_crop": false, "random_crop_height": 256, "random_crop_width": 256, "random_flip": true, "flip_horizontal": false }, "online_shuffling": true, "shuffle_buffer_size": 128, "prefetch": true, "batch_size": 1, "batches_per_epoch": null, "min_batches_per_epoch": 200, "val_batches_per_epoch": null, "min_val_batches_per_epoch": 10, "epochs": 2, "optimizer": "adam", "initial_learning_rate": 0.0001, "learning_rate_schedule": { "reduce_on_plateau": true, "reduction_factor": 0.5, "plateau_min_delta": 1e-06, "plateau_patience": 5, "plateau_cooldown": 3, "min_learning_rate": 1e-08 }, "hard_keypoint_mining": { "online_mining": false, "hard_to_easy_ratio": 2.0, "min_hard_keypoints": 2, "max_hard_keypoints": null, "loss_scale": 5.0 }, "early_stopping": { "stop_training_on_plateau": true, "plateau_min_delta": 1e-08, "plateau_patience": 10 } }, "outputs": { "save_outputs": true, "run_name": "240621_144245.single_instance.n=1", "run_name_prefix": "calibration", "run_name_suffix": "", "runs_folder": "/Users/adamgosztolai/Documents/GitHub/large_kinematic_model/preprocessing/sleap/models", "tags": [ "" ], "save_visualizations": true, "delete_viz_images": true, "zip_outputs": false, "log_to_csv": true, "checkpointing": { "initial_model": false, "best_model": true, "every_epoch": false, "latest_model": false, "final_model": false }, "tensorboard": { "write_logs": false, "loss_frequency": "epoch", "architecture_graph": false, "profile_graph": false, "visualizations": true }, "zmq": { "subscribe_to_controller": true, "controller_address": "tcp://127.0.0.1:9000", "controller_polling_timeout": 10, "publish_updates": true, "publish_address": "tcp://127.0.0.1:9001" } }, "name": "", "description": "", "sleap_version": "1.3.3", "filename": "/var/folders/3n/71gnzd013y5f29s9t0tyyrwh0000gn/T/tmpnecxtno4/240621_144245_training_job.json" } INFO:sleap.nn.training: INFO:sleap.nn.training:Failed to query GPU memory from nvidia-smi. Defaulting to first GPU. INFO:sleap.nn.training:Using GPU 0 for acceleration. INFO:sleap.nn.training:Disabled GPU memory pre-allocation. INFO:sleap.nn.training:System: GPUs: 1/1 available Device: /physical_device:GPU:0 Available: True Initalized: False Memory growth: True INFO:sleap.nn.training: INFO:sleap.nn.training:Initializing trainer... INFO:sleap.nn.training:Loading training labels from: /Users/adamgosztolai/Documents/GitHub/large_kinematic_model/preprocessing/sleap/labels.v001.slp INFO:sleap.nn.training:Creating training and validation splits from validation fraction: 0.1 INFO:sleap.nn.training: Splits: Training = 1 / Validation = 1. INFO:sleap.nn.training:Setting up for training... INFO:sleap.nn.training:Setting up pipeline builders... INFO:sleap.nn.training:Setting up model... INFO:sleap.nn.training:Building test pipeline... Metal device set to: Apple M3 Pro systemMemory: 36.00 GB maxCacheSize: 13.50 GB 2024-06-21 14:42:49.053848: I tensorflow/core/common_runtime/pluggable_device/pluggable_device_factory.cc:305] Could not identify NUMA node of platform GPU ID 0, defaulting to 0. Your kernel may not have been built with NUMA support. 2024-06-21 14:42:49.053983: I tensorflow/core/common_runtime/pluggable_device/pluggable_device_factory.cc:271] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 0 MB memory) -> physical PluggableDevice (device: 0, name: METAL, pci bus id: