search

talmolab / sleap

A deep learning framework for multi-animal pose tracking.
https://sleap.ai
Other
435 stars 96 forks source link

New label placed outside the frame #1551

Open roaldarbol opened 1 year ago

roaldarbol commented 1 year ago

Bug description

I am labelling videos of different frame sizes in sleap v1.3.3, and when I move a large frame to a small frame and add a new instance, the labels are placed based (it seems to me) based on the previous positions in pixels. However, because the new frame is much small, some labels are moved outside the new frame and are impossible to re-position (there's the Alt/Option trick from https://github.com/talmolab/sleap/issues/896#issuecomment-1208440415, to move the entire instance as a workaround ).

Expected behaviour

I would expect the new labels to be placed in relative position. So if the previous frame was 2000pixels wide and a point was at 1000, then in a frame 500px wide, it should be placed at 250. Similarly the other way around.

Actual behaviour

In the above example it seems to be placed at px 1000 still in the new frame, which is then outside the frame.

Your personal set up

Environment packages ``` (sleap) ➜ ~ conda list # packages in environment at /Users/roaldarbol/mambaforge/envs/sleap: # # Name Version Build Channel abseil-cpp 20211102.0 hddbf539_3 conda-forge absl-py 2.0.0 pyhd8ed1ab_0 conda-forge aiohttp 3.8.6 py39ha09f3b3_1 conda-forge aiosignal 1.3.1 pyhd8ed1ab_0 conda-forge aom 3.5.0 hf0c8a7f_0 conda-forge astunparse 1.6.3 pyhd8ed1ab_0 conda-forge async-timeout 4.0.3 pyhd8ed1ab_0 conda-forge attrs 23.1.0 pyh71513ae_1 conda-forge blinker 1.6.3 pyhd8ed1ab_0 conda-forge blosc 1.21.5 heccf04b_0 conda-forge brotli 1.0.9 hb7f2c08_9 conda-forge brotli-bin 1.0.9 hb7f2c08_9 conda-forge brotli-python 1.0.9 py39h7a8716b_9 conda-forge brunsli 0.1 h046ec9c_0 conda-forge bzip2 1.0.8 h0d85af4_4 conda-forge c-ares 1.20.1 h10d778d_0 conda-forge c-blosc2 2.10.5 h354e526_0 conda-forge ca-certificates 2023.7.22 h8857fd0_0 conda-forge cached-property 1.5.2 hd8ed1ab_1 conda-forge cached_property 1.5.2 pyha770c72_1 conda-forge cachetools 5.3.1 pyhd8ed1ab_0 conda-forge cairo 1.16.0 h904041c_1014 conda-forge cattrs 1.1.1 pyhd8ed1ab_0 conda-forge certifi 2023.7.22 pyhd8ed1ab_0 conda-forge cffi 1.16.0 py39h18ef598_0 conda-forge cfitsio 4.2.0 hd56cc12_0 conda-forge charls 2.3.4 he49afe7_0 conda-forge charset-normalizer 3.3.0 pyhd8ed1ab_0 conda-forge click 8.1.7 unix_pyh707e725_0 conda-forge contourpy 1.1.1 py39h8ee36c8_1 conda-forge cryptography 39.0.0 py39h7eb6a14_0 conda-forge cycler 0.12.1 pyhd8ed1ab_0 conda-forge dav1d 1.2.1 h0dc2134_0 conda-forge efficientnet 1.0.0 pypi_0 pypi expat 2.5.0 hf0c8a7f_1 conda-forge ffmpeg 4.4.2 gpl_h8b4fe81_112 conda-forge font-ttf-dejavu-sans-mono 2.37 hab24e00_0 conda-forge font-ttf-inconsolata 3.000 h77eed37_0 conda-forge font-ttf-source-code-pro 2.038 h77eed37_0 conda-forge font-ttf-ubuntu 0.83 hab24e00_0 conda-forge fontconfig 2.14.2 h5bb23bf_0 conda-forge fonts-conda-ecosystem 1 0 conda-forge fonts-conda-forge 1 0 conda-forge fonttools 4.43.1 py39ha09f3b3_0 conda-forge freetype 2.12.1 h60636b9_2 conda-forge frozenlist 1.4.0 py39hdc70f33_1 conda-forge gast 0.4.0 pyh9f0ad1d_0 conda-forge geos 3.12.0 he965462_0 conda-forge gettext 0.21.1 h8a4c099_0 conda-forge giflib 5.2.1 hb7f2c08_3 conda-forge glib 2.78.0 h7d26f99_0 conda-forge glib-tools 2.78.0 h7d26f99_0 conda-forge gmp 6.2.1 h2e338ed_0 conda-forge gnutls 3.7.8 h207c4f0_0 conda-forge google-auth 2.23.3 pyhca7485f_0 conda-forge google-auth-oauthlib 0.4.6 pyhd8ed1ab_0 conda-forge google-pasta 0.2.0 pyh8c360ce_0 conda-forge graphite2 1.3.13 h2e338ed_1001 conda-forge grpc-cpp 1.46.4 hea9d06c_3 conda-forge grpcio 1.46.4 py39h389b050_3 conda-forge gst-plugins-base 1.22.6 hb5d3a86_2 conda-forge gstreamer 1.22.6 h840fbdc_2 conda-forge h5py 3.8.0 nompi_py39h7d40783_100 conda-forge harfbuzz 5.3.0 h08f8713_0 conda-forge hdf5 1.12.2 nompi_h70e9adf_101 conda-forge icu 70.1 h96cf925_0 conda-forge idna 3.4 pyhd8ed1ab_0 conda-forge image-classifiers 1.0.0 pypi_0 pypi imagecodecs 2022.9.26 py39h8036967_4 conda-forge imageio 2.31.5 pyh8c1a49c_0 conda-forge imgaug 0.4.0 pyhd8ed1ab_1 conda-forge imgstore 0.2.9 pypi_0 pypi importlib-metadata 6.8.0 pyha770c72_0 conda-forge importlib-resources 6.1.0 pyhd8ed1ab_0 conda-forge importlib_resources 6.1.0 pyhd8ed1ab_0 conda-forge jasper 2.0.33 h7c6fec8_1 conda-forge joblib 1.3.2 pyhd8ed1ab_0 conda-forge jpeg 9e hb7f2c08_3 conda-forge jsmin 3.0.1 pyhd8ed1ab_0 conda-forge jsonpickle 1.2 py_0 conda-forge jsonschema 4.19.0 pypi_0 pypi jsonschema-specifications 2023.7.1 pypi_0 pypi jxrlib 1.1 h35c211d_2 conda-forge keras 2.9.0 pyhd8ed1ab_0 conda-forge keras-applications 1.0.8 pypi_0 pypi keras-preprocessing 1.1.2 pyhd8ed1ab_0 conda-forge kiwisolver 1.4.5 py39h8ee36c8_1 conda-forge krb5 1.20.1 h0165f36_0 conda-forge lame 3.100 hb7f2c08_1003 conda-forge lazy_loader 0.3 pyhd8ed1ab_0 conda-forge lcms2 2.14 h90f4b2a_0 conda-forge lerc 4.0.0 hb486fe8_0 conda-forge libabseil 20211102.0 cxx17_h844d122_3 conda-forge libaec 1.1.2 he965462_1 conda-forge libavif 0.11.1 h22361c6_2 conda-forge libblas 3.9.0 18_osx64_openblas conda-forge libbrotlicommon 1.0.9 hb7f2c08_9 conda-forge libbrotlidec 1.0.9 hb7f2c08_9 conda-forge libbrotlienc 1.0.9 hb7f2c08_9 conda-forge libcblas 3.9.0 18_osx64_openblas conda-forge libclang 13.0.1 root_62804_h2961583_3 conda-forge libclang13 16.0.1 default_h395b6a0_0 conda-forge libcurl 7.87.0 haf73cf8_0 conda-forge libcxx 16.0.6 hd57cbcb_0 conda-forge libdeflate 1.14 hb7f2c08_0 conda-forge libedit 3.1.20191231 h0678c8f_2 conda-forge libev 4.33 haf1e3a3_1 conda-forge libexpat 2.5.0 hf0c8a7f_1 conda-forge libffi 3.4.2 h0d85af4_5 conda-forge libgfortran 5.0.0 13_2_0_h97931a8_1 conda-forge libgfortran5 13.2.0 h2873a65_1 conda-forge libglib 2.78.0 hc62aa5d_0 conda-forge libiconv 1.17 hac89ed1_0 conda-forge libidn2 2.3.4 hb7f2c08_0 conda-forge liblapack 3.9.0 18_osx64_openblas conda-forge liblapacke 3.9.0 18_osx64_openblas conda-forge libllvm13 13.0.1 h64f94b2_2 conda-forge libllvm16 16.0.1 h7001e86_0 conda-forge libnghttp2 1.51.0 h0dd9d14_0 conda-forge libogg 1.3.4 h35c211d_1 conda-forge libopenblas 0.3.24 openmp_h48a4ad5_0 conda-forge libopencv 4.6.0 py39h93c2317_3 conda-forge libopus 1.3.1 hc929b4f_1 conda-forge libpng 1.6.39 ha978bb4_0 conda-forge libpq 15.1 hdbdeef1_3 conda-forge libprotobuf 3.20.3 hbc0c0cd_0 conda-forge libsodium 1.0.18 hbcb3906_1 conda-forge libsqlite 3.43.2 h92b6c6a_0 conda-forge libssh2 1.10.0 h7535e13_3 conda-forge libtasn1 4.19.0 hb7f2c08_0 conda-forge libtiff 4.4.0 h6268bbc_5 conda-forge libunistring 0.9.10 h0d85af4_0 conda-forge libvorbis 1.3.7 h046ec9c_0 conda-forge libvpx 1.11.0 he49afe7_3 conda-forge libwebp-base 1.3.2 h0dc2134_0 conda-forge libxcb 1.13 h0d85af4_1004 conda-forge libxml2 2.10.3 h201ad9d_4 conda-forge libxslt 1.1.37 h5d22bc9_0 conda-forge libzlib 1.2.13 h8a1eda9_5 conda-forge libzopfli 1.0.3 h046ec9c_0 conda-forge llvm-openmp 17.0.2 hff08bdf_0 conda-forge lz4-c 1.9.4 hf0c8a7f_0 conda-forge markdown 3.5 pyhd8ed1ab_0 conda-forge markdown-it-py 3.0.0 pyhd8ed1ab_0 conda-forge markupsafe 2.1.3 py39hdc70f33_1 conda-forge matplotlib-base 3.8.0 py39h7070ae8_2 conda-forge mdurl 0.1.0 pyhd8ed1ab_0 conda-forge multidict 6.0.4 py39ha30fb19_1 conda-forge munkres 1.1.4 pyh9f0ad1d_0 conda-forge mysql-common 8.0.32 h7ebae80_0 conda-forge mysql-libs 8.0.32 hc37e033_0 conda-forge ncurses 6.4 hf0c8a7f_0 conda-forge ndx-pose 0.1.1 pypi_0 pypi nettle 3.8.1 h96f3785_1 conda-forge networkx 3.1 pyhd8ed1ab_0 conda-forge nixio 1.5.3 pypi_0 pypi nspr 4.35 hea0b92c_0 conda-forge nss 3.94 hd6ac835_0 conda-forge numpy 1.22.4 py39h677350a_0 conda-forge oauthlib 3.2.2 pyhd8ed1ab_0 conda-forge opencv 4.6.0 py39h6e9494a_3 conda-forge openh264 2.3.1 hf0c8a7f_2 conda-forge openjpeg 2.5.0 h5d0d7b0_1 conda-forge openssl 1.1.1w h8a1eda9_0 conda-forge opt_einsum 3.3.0 pyhc1e730c_2 conda-forge p11-kit 0.24.1 h65f8906_0 conda-forge packaging 23.2 pyhd8ed1ab_0 conda-forge pandas 2.1.1 py39hcc9a0c8_1 conda-forge patsy 0.5.3 pyhd8ed1ab_0 conda-forge pcre2 10.40 h1c4e4bc_0 conda-forge pillow 9.2.0 py39h35d4919_3 conda-forge pip 23.2.1 pyhd8ed1ab_0 conda-forge pixman 0.42.2 he965462_0 conda-forge protobuf 3.20.3 py39h7a8716b_1 conda-forge psutil 5.9.5 py39hdc70f33_1 conda-forge pthread-stubs 0.4 hc929b4f_1001 conda-forge py-opencv 4.6.0 py39h71a6800_3 conda-forge pyasn1 0.5.0 pyhd8ed1ab_0 conda-forge pyasn1-modules 0.3.0 pyhd8ed1ab_0 conda-forge pycparser 2.21 pyhd8ed1ab_0 conda-forge pygments 2.16.1 pyhd8ed1ab_0 conda-forge pyjwt 2.8.0 pyhd8ed1ab_0 conda-forge pykalman 0.9.5 py_1 conda-forge pynwb 2.5.0 pypi_0 pypi pyopenssl 23.2.0 pyhd8ed1ab_1 conda-forge pyparsing 3.1.1 pyhd8ed1ab_0 conda-forge pyside2 5.15.8 py39heedc0ee_2 conda-forge pysocks 1.7.1 pyha2e5f31_6 conda-forge python 3.9.15 h531fd05_0_cpython conda-forge python-dateutil 2.8.2 pyhd8ed1ab_0 conda-forge python-flatbuffers 1.12 pyhd8ed1ab_1 conda-forge python-rapidjson 1.12 py39h110ca85_0 conda-forge python-tzdata 2023.3 pyhd8ed1ab_0 conda-forge python_abi 3.9 4_cp39 conda-forge pytz 2023.3.post1 pyhd8ed1ab_0 conda-forge pyu2f 0.1.5 pyhd8ed1ab_0 conda-forge pywavelets 1.4.1 py39hd01001f_1 conda-forge pyyaml 6.0.1 py39hdc70f33_1 conda-forge pyzmq 25.1.1 py39hdac9eea_1 conda-forge qimage2ndarray 1.10.0 pypi_0 pypi qt-main 5.15.8 h1d3b3f8_6 conda-forge qtpy 2.4.0 pyhd8ed1ab_0 conda-forge re2 2022.06.01 hb486fe8_1 conda-forge readline 8.2 h9e318b2_1 conda-forge referencing 0.30.2 pypi_0 pypi requests 2.31.0 pyhd8ed1ab_0 conda-forge requests-oauthlib 1.3.1 pyhd8ed1ab_0 conda-forge rich 13.6.0 pyhd8ed1ab_0 conda-forge rpds-py 0.10.3 pypi_0 pypi rsa 4.9 pyhd8ed1ab_0 conda-forge ruamel-yaml 0.17.32 pypi_0 pypi ruamel-yaml-clib 0.2.7 pypi_0 pypi scikit-image 0.22.0 py39h5d65943_2 conda-forge scikit-learn 1.0 py39hd2caeff_1 conda-forge scikit-video 1.1.11 pyh24bf2e0_0 conda-forge scipy 1.9.0 py39h29d19b3_0 conda-forge seaborn 0.13.0 hd8ed1ab_0 conda-forge seaborn-base 0.13.0 pyhd8ed1ab_0 conda-forge segmentation-models 1.0.1 pypi_0 pypi setuptools 68.2.2 pyhd8ed1ab_0 conda-forge shapely 2.0.2 py39h808d968_0 conda-forge six 1.16.0 pyh6c4a22f_0 conda-forge sleap 1.3.3 pypi_0 pypi snappy 1.1.10 h225ccf5_0 conda-forge sqlite 3.43.2 h7461747_0 conda-forge statsmodels 0.14.0 py39h5b4affa_2 conda-forge svt-av1 1.4.1 hf0c8a7f_0 conda-forge tensorboard 2.9.0 pyhd8ed1ab_0 conda-forge tensorboard-data-server 0.6.1 py39h7eb6a14_4 conda-forge tensorboard-plugin-wit 1.8.1 pyhd8ed1ab_0 conda-forge tensorflow 2.9.1 cpu_py39hb0a6171_0 conda-forge tensorflow-base 2.9.1 cpu_py39hdd40284_0 conda-forge tensorflow-estimator 2.9.1 cpu_py39hd441b4c_0 conda-forge tensorflow-hub 0.12.0 pyhca92ed8_0 conda-forge termcolor 2.3.0 pyhd8ed1ab_0 conda-forge threadpoolctl 3.2.0 pyha21a80b_0 conda-forge tifffile 2022.10.10 pyhd8ed1ab_0 conda-forge tk 8.6.13 hef22860_0 conda-forge typing-extensions 4.8.0 hd8ed1ab_0 conda-forge typing_extensions 4.8.0 pyha770c72_0 conda-forge tzdata 2023c h71feb2d_0 conda-forge tzlocal 5.0.1 pypi_0 pypi unicodedata2 15.1.0 py39hdc70f33_0 conda-forge urllib3 2.0.6 pyhd8ed1ab_0 conda-forge werkzeug 3.0.0 pyhd8ed1ab_0 conda-forge wheel 0.41.2 pyhd8ed1ab_0 conda-forge wrapt 1.15.0 py39hdc70f33_1 conda-forge x264 1!164.3095 h775f41a_2 conda-forge x265 3.5 hbb4e6a2_3 conda-forge xorg-libxau 1.0.11 h0dc2134_0 conda-forge xorg-libxdmcp 1.1.3 h35c211d_0 conda-forge xz 5.2.6 h775f41a_0 conda-forge yaml 0.2.5 h0d85af4_2 conda-forge yarl 1.9.2 py39ha09f3b3_1 conda-forge zeromq 4.3.5 h93d8f39_0 conda-forge zfp 1.0.0 hf3d7188_4 conda-forge zipp 3.17.0 pyhd8ed1ab_0 conda-forge zlib 1.2.13 h8a1eda9_5 conda-forge zlib-ng 2.0.7 hb7f2c08_0 conda-forge zstd 1.5.5 h829000d_0 conda-forge ```

Screenshots

Screenshot 2023-10-14 at 18 13 19

How to reproduce

  1. Load two videos of different frame sizes.
  2. Label a frame from the biggest one first
  3. Add instance in a frame from the smaller video
roaldarbol commented 1 year ago

I guess the root issue is that it's expecting frames to be similar. I'm also getting some weird rotations on new instances that are supposed to be the same as the previous frame:

Screenshot 2023-10-14 at 19 30 13
roomrys commented 1 year ago

Hi @roaldarbol,

I have an inkling that this shifting of predictions likely has to do with the resize_and_pad_to_target and related parameters in the training_config.json. This parameter should be set to true, but what I believe might be happening is that the target_height and target_width are set to the incorrect dimensions. I see you have variable sized videos... so, what I would try is:

  1. Find the training_config.json that you have been selecting for use in the GUI
  2. Find the resize_and_pad_to_target, target_height, and target_width parameters
  3. Ensure that target_height and target_width are both set to "null" (no quotes)
  4. Save the training_config.json

  5. Repeat 1-4 for each training_config.json to be used (2 for the top-down)

  6. (Re)Open the Training Widget GUI
  7. Select the edited training_config.json
  8. Optionally, if on 1.3.3+ "Copy to Clipboard" and paste the config info somewhere to verify your changes were applied

  9. Repeat 6-8 for each training_config.json to be used

  10. Rerun training

Let me know if you need more guidance/clarification.

Thanks, Liezl

roaldarbol commented 1 year ago

Hi @roomrys! Thanks for the reply! I am not doing predictions just yet, this is just labelling frames and using Add instance (or Cmd+I). So I wouldn't assume that should be affected by training_config.json - in fact I don't even have such a file yet as I haven't trained.

roomrys commented 1 year ago

Ah yes, thanks for clarifying! I was definitely adding in my own assumptions (and overlooked the number of labeled frames in the project). This is a case that we haven't covered! While we have a bit of code to ensure that non-visible points are initialized within the frame, SLEAP just copies the visible points from the copy_instance https://github.com/talmolab/sleap/blob/1e0627aae2e69e664d8eca0dd3af04ba8dd9b13c/sleap/gui/commands.py#L2951-L2957

We would need to add a check here that ensures the point is within the current frame... maybe shifting all points over to fit in the current frame.

Until this is fixed and released, using the force-directed method to add an instance should add the points at the location you are right-clicking.

roaldarbol commented 1 year ago

Yeah, that sounds just right - I think you've nailed the root of the issue, and your proposed fix sounds exactly like what I'd expect! I think you could just include a scaling factor like new_x = old_x * (new_width/old_width) and the same for y coordinates and height. And I think the check could just be new_width == old_width (and again similar for height).