search

talmolab / sleap

A deep learning framework for multi-animal pose tracking.
https://sleap.ai
Other
427 stars 97 forks source link

Albumentations is breaking with some edge cases #1722

Open talmo opened 6 months ago

talmo commented 6 months ago

Bug description

In #1623, we switched from imaug to albumentations for our augmentation backend (see also #648).

This is mostly working, but using it in a real application resulted in some issues related to when points are close to the edge of the image or nan during training. See the logs below.

We should find out if this is a known issue with albumentation and if we should do some preprocessing to handle these cases. We should also add tests for augmentation with nans, zeros and image height/width.

This is on albumentations 1.3.1, installed via conda-forge.

Relevant error

  File "d:\sleap_develop\sleap\nn\data\augmentation.py", line 239, in py_augment
    augmented = self.augmenter(image=img, keypoints=kps)
...
ValueError: Expected x for keypoint (840.0, 93.648865, 0.0, 0.0) to be in the range [0.0, 840], got 840.0.

Also:

ValueError: Expected x for keypoint (nan, nan, 0.0, 0.0) to be in the range [0.0, 840], got nan.

Your personal set up

Environment packages ``` # packages in environment at C:\Miniconda3\envs\sleap_develop: # # Name Version Build Channel absl-py 2.1.0 pypi_0 pypi accessible-pygments 0.0.4 pypi_0 pypi alabaster 0.7.13 pypi_0 pypi albumentations 1.3.1 pyhd8ed1ab_0 conda-forge anyio 3.7.1 pypi_0 pypi aom 3.6.1 h63175ca_0 conda-forge appdirs 1.4.4 pypi_0 pypi argon2-cffi 23.1.0 pypi_0 pypi argon2-cffi-bindings 21.2.0 pypi_0 pypi astunparse 1.6.3 pypi_0 pypi attrs 21.4.0 pypi_0 pypi babel 2.14.0 pypi_0 pypi backcall 0.2.0 pypi_0 pypi backports-zoneinfo 0.2.1 pypi_0 pypi beautifulsoup4 4.12.3 pypi_0 pypi black 21.6b0 pypi_0 pypi bleach 6.0.0 pypi_0 pypi blosc 1.21.5 hdccc3a2_0 conda-forge brotli 1.0.9 hcfcfb64_9 conda-forge brotli-bin 1.0.9 hcfcfb64_9 conda-forge bzip2 1.0.8 hcfcfb64_5 conda-forge c-blosc2 2.13.2 h183a6f4_0 conda-forge ca-certificates 2024.2.2 h56e8100_0 conda-forge cached-property 1.5.2 pypi_0 pypi cachetools 5.3.3 pypi_0 pypi cattrs 1.1.1 pyhd8ed1ab_0 conda-forge certifi 2024.2.2 pyhd8ed1ab_0 conda-forge cffi 1.15.1 pypi_0 pypi cfgv 3.3.1 pypi_0 pypi cfitsio 4.0.0 hd67004f_0 conda-forge charls 2.3.4 h39d44d4_0 conda-forge charset-normalizer 3.3.2 pypi_0 pypi click 8.0.4 pypi_0 pypi cloudpickle 2.2.1 pyhd8ed1ab_0 conda-forge colorama 0.4.6 pypi_0 pypi comm 0.1.4 pypi_0 pypi coverage 7.2.7 pypi_0 pypi cryptography 42.0.5 pypi_0 pypi cuda-nvcc 11.3.58 hb8d16a4_0 nvidia cudatoolkit 11.3.1 hf2f0253_13 conda-forge cudnn 8.2.1.32 h754d62a_0 conda-forge cycler 0.11.0 pyhd8ed1ab_0 conda-forge cytoolz 0.12.0 py37hcc03f2d_0 conda-forge dask-core 2022.2.0 pyhd8ed1ab_0 conda-forge dav1d 1.2.1 hcfcfb64_0 conda-forge debugpy 1.7.0 pypi_0 pypi decorator 5.1.1 pypi_0 pypi defusedxml 0.7.1 pypi_0 pypi deprecated 1.2.14 pypi_0 pypi distlib 0.3.8 pypi_0 pypi docutils 0.17.1 pypi_0 pypi efficientnet 1.0.0 pypi_0 pypi entrypoints 0.4 pypi_0 pypi exceptiongroup 1.2.0 pypi_0 pypi expat 2.6.2 h63175ca_0 conda-forge ffmpeg 5.1.2 gpl_h8bb4bc8_112 conda-forge filelock 3.12.2 pypi_0 pypi flatbuffers 2.0.7 pypi_0 pypi font-ttf-dejavu-sans-mono 2.37 hab24e00_0 conda-forge font-ttf-inconsolata 3.000 h77eed37_0 conda-forge font-ttf-source-code-pro 2.038 h77eed37_0 conda-forge font-ttf-ubuntu 0.83 h77eed37_1 conda-forge fontconfig 2.14.2 hbde0cde_0 conda-forge fonts-conda-ecosystem 1 0 conda-forge fonts-conda-forge 1 0 conda-forge fonttools 4.38.0 py37h51bd9d9_0 conda-forge freeglut 3.2.2 h63175ca_2 conda-forge freetype 2.12.1 hdaf720e_2 conda-forge fsspec 2023.1.0 pyhd8ed1ab_0 conda-forge gast 0.4.0 pypi_0 pypi gettext 0.21.1 h5728263_0 conda-forge giflib 5.2.1 h64bf75a_3 conda-forge gitdb 4.0.11 pypi_0 pypi gitpython 3.1.42 pypi_0 pypi glib 2.80.0 h39d0aa6_1 conda-forge glib-tools 2.80.0 h0a98069_1 conda-forge google-auth 2.28.2 pypi_0 pypi google-auth-oauthlib 0.4.6 pypi_0 pypi google-pasta 0.2.0 pypi_0 pypi greenlet 3.0.3 pypi_0 pypi grpcio 1.62.1 pypi_0 pypi gst-plugins-base 1.22.9 h001b923_0 conda-forge gstreamer 1.22.9 hb4038d2_0 conda-forge h5py 3.1.0 pypi_0 pypi hdmf 3.5.5 pypi_0 pypi icu 70.1 h0e60522_0 conda-forge identify 2.5.24 pypi_0 pypi idna 3.6 pypi_0 pypi image-classifiers 1.0.0 pypi_0 pypi imagecodecs 2021.11.20 py37hfea8715_2 conda-forge imageio 2.34.0 pyh4b66e23_0 conda-forge imagesize 1.4.1 pypi_0 pypi imgstore 0.2.9 pypi_0 pypi importlib-metadata 6.7.0 pypi_0 pypi importlib-resources 5.12.0 pypi_0 pypi iniconfig 2.0.0 pypi_0 pypi intel-openmp 2024.0.0 h57928b3_49841 conda-forge ipykernel 6.16.2 pypi_0 pypi ipython 7.34.0 pypi_0 pypi ipython-genutils 0.2.0 pypi_0 pypi ipywidgets 7.8.1 pypi_0 pypi jaraco-classes 3.2.3 pypi_0 pypi jasper 2.0.33 hc2e4405_1 conda-forge jedi 0.17.2 pypi_0 pypi jinja2 3.1.3 pypi_0 pypi joblib 1.3.2 pyhd8ed1ab_0 conda-forge jpeg 9e hcfcfb64_3 conda-forge jsmin 3.0.1 pyhd8ed1ab_0 conda-forge json5 0.9.16 pypi_0 pypi jsonpickle 1.2 py_0 conda-forge jsonschema 4.17.3 pypi_0 pypi jupyter-cache 0.4.3 pypi_0 pypi jupyter-client 7.4.9 pypi_0 pypi jupyter-core 4.12.0 pypi_0 pypi jupyter-server 1.13.5 pypi_0 pypi jupyter-server-mathjax 0.2.6 pypi_0 pypi jupyter-sphinx 0.3.2 pypi_0 pypi jupyterlab 3.3.4 pypi_0 pypi jupyterlab-pygments 0.2.2 pypi_0 pypi jupyterlab-server 2.16.3 pypi_0 pypi jupyterlab-widgets 1.1.7 pypi_0 pypi jxrlib 1.1 hcfcfb64_3 conda-forge keras 2.7.0 pypi_0 pypi keras-applications 1.0.8 pypi_0 pypi keras-preprocessing 1.1.2 pypi_0 pypi keyring 24.1.1 pypi_0 pypi kiwisolver 1.4.4 py37h8c56517_0 conda-forge krb5 1.20.1 heb0366b_0 conda-forge lcms2 2.14 h90d422f_0 conda-forge lerc 3.0 h0e60522_0 conda-forge libaec 1.1.3 h63175ca_0 conda-forge libblas 3.9.0 21_win64_mkl conda-forge libbrotlicommon 1.0.9 hcfcfb64_9 conda-forge libbrotlidec 1.0.9 hcfcfb64_9 conda-forge libbrotlienc 1.0.9 hcfcfb64_9 conda-forge libcblas 3.9.0 21_win64_mkl conda-forge libclang 18.1.1 pypi_0 pypi libclang13 15.0.7 default_h85b4d89_4 conda-forge libcurl 8.1.2 h68f0423_0 conda-forge libdeflate 1.10 h8ffe710_0 conda-forge libexpat 2.6.2 h63175ca_0 conda-forge libffi 3.4.2 h8ffe710_5 conda-forge libglib 2.80.0 h39d0aa6_1 conda-forge libhwloc 2.9.3 default_haede6df_1009 conda-forge libiconv 1.17 hcfcfb64_2 conda-forge liblapack 3.9.0 21_win64_mkl conda-forge liblapacke 3.9.0 21_win64_mkl conda-forge libogg 1.3.4 h8ffe710_1 conda-forge libopencv 4.6.0 py37hefdbd86_5 conda-forge libopus 1.3.1 h8ffe710_1 conda-forge libpng 1.6.43 h19919ed_0 conda-forge libprotobuf 3.21.8 h12be248_0 conda-forge libsodium 1.0.18 h8d14728_1 conda-forge libsqlite 3.45.2 hcfcfb64_0 conda-forge libssh2 1.11.0 h7dfc565_0 conda-forge libtiff 4.4.0 hc4061b1_0 conda-forge libvorbis 1.3.7 h0e60522_0 conda-forge libwebp 1.3.2 hcfcfb64_1 conda-forge libwebp-base 1.3.2 hcfcfb64_0 conda-forge libxcb 1.13 hcd874cb_1004 conda-forge libxml2 2.12.6 hc3477c8_0 conda-forge libxslt 1.1.39 h3df6e99_0 conda-forge libzlib 1.2.13 hcfcfb64_5 conda-forge libzopfli 1.0.3 h0e60522_0 conda-forge linkify-it-py 2.0.3 pypi_0 pypi livereload 2.6.3 pypi_0 pypi locket 1.0.0 pyhd8ed1ab_0 conda-forge lxml 5.1.0 pypi_0 pypi lz4-c 1.9.3 h8ffe710_1 conda-forge m2w64-gcc-libgfortran 5.3.0 6 conda-forge m2w64-gcc-libs 5.3.0 7 conda-forge m2w64-gcc-libs-core 5.3.0 7 conda-forge m2w64-gmp 6.1.0 2 conda-forge m2w64-libwinpthread-git 5.0.0.4634.697f757 2 conda-forge markdown 3.4.4 pypi_0 pypi markdown-it-py 1.1.0 pypi_0 pypi markupsafe 2.1.5 pypi_0 pypi matplotlib-base 3.5.3 py37hbaab90a_2 conda-forge matplotlib-inline 0.1.6 pypi_0 pypi mdit-py-plugins 0.2.8 pypi_0 pypi mdurl 0.1.2 pyhd8ed1ab_0 conda-forge mistune 0.8.4 pypi_0 pypi mkl 2024.0.0 h66d3029_49657 conda-forge more-itertools 9.1.0 pypi_0 pypi msys2-conda-epoch 20160418 1 conda-forge munkres 1.1.4 pyh9f0ad1d_0 conda-forge mypy-extensions 1.0.0 pypi_0 pypi myst-nb 0.13.2 pypi_0 pypi myst-parser 0.15.2 pypi_0 pypi nbclassic 0.5.6 pypi_0 pypi nbclient 0.5.13 pypi_0 pypi nbconvert 6.5.4 pypi_0 pypi nbdime 4.0.1 pypi_0 pypi nbformat 5.1.3 pypi_0 pypi ndx-pose 0.1.1 pypi_0 pypi nest-asyncio 1.6.0 pypi_0 pypi networkx 2.7 pyhd8ed1ab_0 conda-forge nixio 1.5.3 pypi_0 pypi nodeenv 1.8.0 pypi_0 pypi notebook 6.5.6 pypi_0 pypi notebook-shim 0.2.4 pypi_0 pypi numpy 1.21.6 py37h2830a78_0 conda-forge oauthlib 3.2.2 pypi_0 pypi opencv 4.6.0 py37h03978a9_5 conda-forge opencv-python-headless 4.9.0.80 pypi_0 pypi openh264 2.3.1 h63175ca_2 conda-forge openjpeg 2.5.0 hc9384bd_1 conda-forge openssl 3.2.1 hcfcfb64_1 conda-forge opt-einsum 3.3.0 pypi_0 pypi packaging 24.0 pypi_0 pypi pandas 1.3.5 py37h9386db6_0 conda-forge pandocfilters 1.5.1 pypi_0 pypi parso 0.7.1 pypi_0 pypi partd 1.4.1 pyhd8ed1ab_0 conda-forge pathspec 0.11.2 pypi_0 pypi patsy 0.5.6 pyhd8ed1ab_0 conda-forge pcre2 10.43 h17e33f8_0 conda-forge pickleshare 0.7.5 pypi_0 pypi pillow 9.2.0 py37h42a8222_2 conda-forge pip 24.0 pyhd8ed1ab_0 conda-forge pkginfo 1.10.0 pypi_0 pypi pkgutil-resolve-name 1.3.10 pypi_0 pypi platformdirs 4.0.0 pypi_0 pypi pluggy 1.2.0 pypi_0 pypi pre-commit 2.21.0 pypi_0 pypi prometheus-client 0.17.1 pypi_0 pypi prompt-toolkit 3.0.43 pypi_0 pypi protobuf 3.19.6 pypi_0 pypi psutil 5.9.3 py37h51bd9d9_0 conda-forge pthread-stubs 0.4 hcd874cb_1001 conda-forge pthreads-win32 2.9.1 hfa6e2cd_3 conda-forge py-opencv 4.6.0 py37h8013278_5 conda-forge pyasn1 0.5.1 pypi_0 pypi pyasn1-modules 0.3.0 pypi_0 pypi pycparser 2.21 pypi_0 pypi pydata-sphinx-theme 0.13.3 pypi_0 pypi pygithub 2.2.0 pypi_0 pypi pygments 2.17.2 pyhd8ed1ab_0 conda-forge pyjwt 2.8.0 pypi_0 pypi pykalman 0.9.5 py_1 conda-forge pynacl 1.5.0 pypi_0 pypi pynwb 2.3.3 pypi_0 pypi pyparsing 3.1.2 pyhd8ed1ab_0 conda-forge pyrsistent 0.19.3 pypi_0 pypi pyside2 5.15.6 py37h7185d37_2 conda-forge pytest 7.4.4 pypi_0 pypi pytest-cov 3.0.0 pypi_0 pypi pytest-qt 4.2.0 pypi_0 pypi pytest-xvfb 3.0.0 pypi_0 pypi python 3.7.12 h900ac77_100_cpython conda-forge python-dateutil 2.9.0 pyhd8ed1ab_0 conda-forge python-rapidjson 1.9 py37h7f67f24_0 conda-forge python_abi 3.7 4_cp37m conda-forge pytz 2024.1 pyhd8ed1ab_0 conda-forge pyvirtualdisplay 3.0 pypi_0 pypi pywavelets 1.3.0 py37h3a130e4_1 conda-forge pywin32 306 pypi_0 pypi pywin32-ctypes 0.2.2 pypi_0 pypi pywinpty 1.1.6 pypi_0 pypi pyyaml 6.0 py37hcc03f2d_4 conda-forge pyzmq 24.0.1 py37h7347f05_0 conda-forge qimage2ndarray 1.10.0 pypi_0 pypi qt-main 5.15.8 h720456b_6 conda-forge qt-webengine 5.15.8 h5b1ea0b_0 conda-forge qtpy 2.4.1 pyhd8ed1ab_0 conda-forge qudida 0.0.4 pyhd8ed1ab_0 conda-forge readme-renderer 37.3 pypi_0 pypi regex 2023.12.25 pypi_0 pypi requests 2.31.0 pypi_0 pypi requests-oauthlib 1.4.0 pypi_0 pypi requests-toolbelt 1.0.0 pypi_0 pypi rfc3986 2.0.0 pypi_0 pypi rich 13.7.1 pyhd8ed1ab_0 conda-forge ruamel-yaml 0.18.6 pypi_0 pypi ruamel-yaml-clib 0.2.8 pypi_0 pypi scikit-image 0.19.2 py37h9386db6_0 conda-forge scikit-learn 1.0 py37ha78be43_1 conda-forge scikit-video 1.1.11 pyh24bf2e0_0 conda-forge scipy 1.7.3 py37hb6553fb_0 conda-forge seaborn 0.12.2 hd8ed1ab_0 conda-forge seaborn-base 0.12.2 pyhd8ed1ab_0 conda-forge segmentation-models 1.0.1 pypi_0 pypi send2trash 1.8.2 pypi_0 pypi setuptools 59.8.0 py37h03978a9_1 conda-forge six 1.16.0 pyh6c4a22f_0 conda-forge sleap 1.3.3 dev_0 smmap 5.0.1 pypi_0 pypi snappy 1.1.10 hfb803bf_0 conda-forge sniffio 1.3.1 pypi_0 pypi snowballstemmer 2.2.0 pypi_0 pypi soupsieve 2.4.1 pypi_0 pypi sphinx 4.5.0 pypi_0 pypi sphinx-autobuild 2021.3.14 pypi_0 pypi sphinx-book-theme 1.0.1 pypi_0 pypi sphinx-copybutton 0.5.2 pypi_0 pypi sphinx-togglebutton 0.3.2 pypi_0 pypi sphinxcontrib-applehelp 1.0.2 pypi_0 pypi sphinxcontrib-devhelp 1.0.2 pypi_0 pypi sphinxcontrib-htmlhelp 2.0.0 pypi_0 pypi sphinxcontrib-jsmath 1.0.1 pypi_0 pypi sphinxcontrib-qthelp 1.0.3 pypi_0 pypi sphinxcontrib-serializinghtml 1.1.5 pypi_0 pypi sqlalchemy 1.4.52 pypi_0 pypi sqlite 3.45.2 hcfcfb64_0 conda-forge statsmodels 0.13.2 py37h3a130e4_0 conda-forge svt-av1 1.4.1 h63175ca_0 conda-forge tbb 2021.11.0 h91493d7_1 conda-forge tensorboard 2.10.1 pypi_0 pypi tensorboard-data-server 0.6.1 pypi_0 pypi tensorboard-plugin-wit 1.8.1 pypi_0 pypi tensorflow 2.7.0 pypi_0 pypi tensorflow-estimator 2.7.0 pypi_0 pypi tensorflow-hub 0.12.0 pyhca92ed8_0 conda-forge tensorflow-io-gcs-filesystem 0.31.0 pypi_0 pypi termcolor 2.3.0 pypi_0 pypi terminado 0.17.1 pypi_0 pypi threadpoolctl 3.1.0 pyh8a188c0_0 conda-forge tifffile 2021.11.2 pyhd8ed1ab_0 conda-forge tinycss2 1.2.1 pypi_0 pypi tk 8.6.13 h5226925_1 conda-forge toml 0.10.2 pypi_0 pypi tomli 2.0.1 pypi_0 pypi toolz 0.12.1 pyhd8ed1ab_0 conda-forge tornado 6.2 pypi_0 pypi tqdm 4.66.2 pypi_0 pypi traitlets 5.9.0 pypi_0 pypi twine 3.3.0 pypi_0 pypi typed-ast 1.5.5 pypi_0 pypi typing-extensions 4.7.1 hd8ed1ab_0 conda-forge typing_extensions 4.7.1 pyha770c72_0 conda-forge tzdata 2024.1 pypi_0 pypi tzlocal 5.1 pypi_0 pypi uc-micro-py 1.0.3 pypi_0 pypi ucrt 10.0.22621.0 h57928b3_0 conda-forge unicodedata2 14.0.0 py37hcc03f2d_1 conda-forge urllib3 2.0.7 pypi_0 pypi vc 14.3 hcf57466_18 conda-forge vc14_runtime 14.38.33130 h82b7239_18 conda-forge virtualenv 20.25.1 pypi_0 pypi vs2015_runtime 14.38.33130 hcb4865c_18 conda-forge wcwidth 0.2.13 pypi_0 pypi webencodings 0.5.1 pypi_0 pypi websocket-client 1.6.1 pypi_0 pypi werkzeug 2.2.3 pypi_0 pypi wheel 0.43.0 pyhd8ed1ab_0 conda-forge widgetsnbextension 3.6.6 pypi_0 pypi wrapt 1.16.0 pypi_0 pypi x264 1!164.3095 h8ffe710_2 conda-forge x265 3.5 h2d74725_3 conda-forge xorg-libxau 1.0.11 hcd874cb_0 conda-forge xorg-libxdmcp 1.1.3 hcd874cb_0 conda-forge xz 5.2.6 h8d14728_0 conda-forge yaml 0.2.5 h8ffe710_2 conda-forge zeromq 4.3.4 h0e60522_1 conda-forge zfp 0.5.5 h0e60522_8 conda-forge zipp 3.15.0 pypi_0 pypi zlib-ng 2.0.7 hcfcfb64_0 conda-forge zstd 1.5.5 h12be248_0 conda-forge ```
Logs ``` 2024-03-22 17:20:12.626901: W tensorflow/core/framework/op_kernel.cc:1733] INVALID_ARGUMENT: ValueError: Expected x for keypoint (840.0, 93.648865, 0.0, 0.0) to be in the range [0.0, 840], got 840.0. Traceback (most recent call last): File "C:\Miniconda3\envs\sleap_develop\lib\site-packages\tensorflow\python\ops\script_ops.py", line 273, in __call__ return func(device, token, args) File "C:\Miniconda3\envs\sleap_develop\lib\site-packages\tensorflow\python\ops\script_ops.py", line 151, in __call__ outputs = self._call(device, args) File "C:\Miniconda3\envs\sleap_develop\lib\site-packages\tensorflow\python\ops\script_ops.py", line 158, in _call ret = self._func(*args) File "C:\Miniconda3\envs\sleap_develop\lib\site-packages\tensorflow\python\autograph\impl\api.py", line 649, in wrapper return func(*args, **kwargs) File "d:\sleap_develop\sleap\nn\data\augmentation.py", line 239, in py_augment augmented = self.augmenter(image=img, keypoints=kps) File "C:\Miniconda3\envs\sleap_develop\lib\site-packages\albumentations\core\composition.py", line 207, in __call__ p.preprocess(data) File "C:\Miniconda3\envs\sleap_develop\lib\site-packages\albumentations\core\utils.py", line 83, in preprocess data[data_name] = self.check_and_convert(data[data_name], rows, cols, direction="to") File "C:\Miniconda3\envs\sleap_develop\lib\site-packages\albumentations\core\utils.py", line 91, in check_and_convert return self.convert_to_albumentations(data, rows, cols) File "C:\Miniconda3\envs\sleap_develop\lib\site-packages\albumentations\core\keypoints_utils.py", line 146, in convert_to_albumentations angle_in_degrees=params.angle_in_degrees, File "C:\Miniconda3\envs\sleap_develop\lib\site-packages\albumentations\core\keypoints_utils.py", line 271, in convert_keypoints_to_albumentations for kp in keypoints File "C:\Miniconda3\envs\sleap_develop\lib\site-packages\albumentations\core\keypoints_utils.py", line 271, in for kp in keypoints File "C:\Miniconda3\envs\sleap_develop\lib\site-packages\albumentations\core\keypoints_utils.py", line 220, in convert_keypoint_to_albumentations check_keypoint(keypoint, rows, cols) File "C:\Miniconda3\envs\sleap_develop\lib\site-packages\albumentations\core\keypoints_utils.py", line 156, in check_keypoint "to be in the range [0.0, {size}], got {value}.".format(kp=kp, name=name, value=value, size=size) ValueError: Expected x for keypoint (840.0, 93.648865, 0.0, 0.0) to be in the range [0.0, 840], got 840.0. ... Traceback (most recent call last): File "C:\Miniconda3\envs\sleap_develop\lib\site-packages\tensorflow\python\ops\script_ops.py", line 273, in __call__ return func(device, token, args) File "C:\Miniconda3\envs\sleap_develop\lib\site-packages\tensorflow\python\ops\script_ops.py", line 151, in __call__ outputs = self._call(device, args) File "C:\Miniconda3\envs\sleap_develop\lib\site-packages\tensorflow\python\ops\script_ops.py", line 158, in _call ret = self._func(*args) File "C:\Miniconda3\envs\sleap_develop\lib\site-packages\tensorflow\python\autograph\impl\api.py", line 649, in wrapper return func(*args, **kwargs) File "d:\sleap_develop\sleap\nn\data\augmentation.py", line 239, in py_augment augmented = self.augmenter(image=img, keypoints=kps) File "C:\Miniconda3\envs\sleap_develop\lib\site-packages\albumentations\core\composition.py", line 207, in __call__ p.preprocess(data) File "C:\Miniconda3\envs\sleap_develop\lib\site-packages\albumentations\core\utils.py", line 83, in preprocess data[data_name] = self.check_and_convert(data[data_name], rows, cols, direction="to") File "C:\Miniconda3\envs\sleap_develop\lib\site-packages\albumentations\core\utils.py", line 91, in check_and_convert return self.convert_to_albumentations(data, rows, cols) File "C:\Miniconda3\envs\sleap_develop\lib\site-packages\albumentations\core\keypoints_utils.py", line 146, in convert_to_albumentations angle_in_degrees=params.angle_in_degrees, File "C:\Miniconda3\envs\sleap_develop\lib\site-packages\albumentations\core\keypoints_utils.py", line 271, in convert_keypoints_to_albumentations for kp in keypoints File "C:\Miniconda3\envs\sleap_develop\lib\site-packages\albumentations\core\keypoints_utils.py", line 271, in for kp in keypoints File "C:\Miniconda3\envs\sleap_develop\lib\site-packages\albumentations\core\keypoints_utils.py", line 220, in convert_keypoint_to_albumentations check_keypoint(keypoint, rows, cols) File "C:\Miniconda3\envs\sleap_develop\lib\site-packages\albumentations\core\keypoints_utils.py", line 156, in check_keypoint "to be in the range [0.0, {size}], got {value}.".format(kp=kp, name=name, value=value, size=size) ValueError: Expected x for keypoint (nan, nan, 0.0, 0.0) to be in the range [0.0, 840], got nan. [[{{node EagerPyFunc}}]] [Op:IteratorGetNext] ```
talmo commented 6 months ago

Update: The fix should actually be simple -- on this line:

https://github.com/talmolab/sleap/blob/eb147646a79d057b508d7cbfa8f4c5e158601104/sleap/nn/data/augmentation.py#L197-L199

We can just add an optional arg to not discard out-of-bounds keypoints:

            augmenter=A.Compose(
                aug_stack, keypoint_params=A.KeypointParams(
                    format="xy",
                    remove_invisible=False
                )
            ),

In quick testing this seems to bypass the error, but I suppose we should double check for correctness by placing a keypoint out of bounds, rotating, then making sure it's still there.