add a viewer for multi-animal tracking to correct identity switches

[JulianAlvarezdeGiorgi]

Feature request. I'm on a multi-animal setup, training with top-view mice videos, without using the identity layer ('identity = False'), because of the similarities between mice. When running DLC with two mice, I run into an identity swap that I believe would be very difficult for DLC to detect automatically, therefore I plan to build a GUI to correct these swaps manually.

Here is an example:

Describe the solution you'd like I would like to add to the existing "Refinement Tracklets GUI" a new button allowing fast correction of identity swaps.

Pressing this button swaps 2 animal identities from the current time until the movie ends.

Example: after pressing the "SWAP" button, we get:

If working with more than 2 animals, we might need additional controls to specify which animals have an identity swap. We could also add more buttons such as "NEXT ANIMAL CROSSING" to jump to the next potential swap due to animals crossing.

Describe alternatives you've considered I also considered making a separate GUI dedicated to identity swap, rather than adding new features to the existing Refinement Tracklets GUI.

My questions Can this be added to DLC? I would be happy to try developing it myself. In this case, probably I'll need help to know which code to edit, what to pay attention to, etc. Furthermore; I'd appreciate any advice for doing it in a way that I can later push this new functionality to the DLC project.

[jeylau]

Hi @JulianAlvarezdeGiorgi, I believe this is something that can already be done, but please correct me if I'm wrong. You can flag start and end frames to delimit a time window during which a swap occurred (https://github.com/DeepLabCut/DeepLabCut/blob/main/docs/maDLC_UserGuide.md#refine-tracklets), and then click at the top on the target animal's name (which is convenient, as you described, to handle more than 2 animals in the scene).

[JulianAlvarezdeGiorgi]

Hi @jeylau. Yess, you're correct, this can be done, but you need to be select the points you want to swap with the "Lasso" selector, am I right?

This is not only more laborious to do, but it can also pose a problem if you are working with videos of animals that interact a lot. For example suppose we see an identity swap, but due to the nature of the videos (interaction between animals, or hidden parts), it is difficult to select all the points in once, we will need to do it in parts, selecting the most obvious points, exchange the identity, and then go to find a new frame where the remaining points are visible to repeat the process. While it can be done, depending on the case, it can be quite complicated.

This is the example of the previous case, and shows how complex it can be (in the previous example, you could simply select a frame where both mice are separated and select all points there, but in other examples this may not be possible):

Therefore, I believe that a button that simply changes identity (without the need to select the points to be changed) in a time window may be helpful.

[jeylau]

Ah, thanks for clarifying. You're right, that'd be very helpful! If you feel like contributing this feature to DLC, I'm more than happy to provide you with some guidance. The logic for swapping points is this dozen of lines https://github.com/DeepLabCut/DeepLabCut/blob/9e9d677832a148f2dd19b313b9fd976413f4e271/deeplabcut/gui/tracklet_toolbox.py#L643-L655. What you'd need to determine is the tracklet indices belonging to both individuals: this could be done by looking at self.manager.tracklet2id, which maps tracklet indices to individual indices. For example, you could get all tracklet indices of individual 1 with:

inds1 = [k for k, v in self.manager.tracklet2id.items() if v == 1]

I think this should get you started 😊

[JulianAlvarezdeGiorgi]

Perfect, thanks for the help. I've implemented it by using the manager.swap_tracklets method. I've done it as follows, and it works fine:

def swap_tracklets(self, event):
        if self.swap_id1 is not None and self.swap_id2 is not None:

            # Get tracklet indices for each individual
            inds1 = [k for k in range(len(self.manager.tracklet2id)) if self.manager.tracklet2id[k] == self.swap_id1]
            inds2 = [k for k in range(len(self.manager.tracklet2id)) if self.manager.tracklet2id[k] == self.swap_id2]

            print(f'Swapping tracklets {self.swap_id1} and {self.swap_id2}')

            # Frames to swap
            frames = []
            if len(self.cuts) == 2:
                frames = list(range(min(self.cuts), max(self.cuts) + 1))
            elif len(self.cuts) == 1:
                frames = [self.cuts[0]] 
            else:
                frames = list(range(self.curr_frame, self.manager.nframes))

            # Swap the tracklets
            for i in range(min(len(inds1), len(inds2))):
                self.manager.swap_tracklets(inds1[i], inds2[i], frames)
                self.display_traces()
                self.slider.set_val(self.curr_frame)    

    def set_swap_id1(self, val):
        # check that the input is a valid from the list of individuals
        if int(val) in self.manager.tracklet2id:
            self.swap_id1 = int(val)
            print('ID 1 set.')
        else:
            print(f'Invalid ID. Please select a valid ID from the list of individuals: {set(self.manager.tracklet2id)}')
            self.swap_id1 = None

    def set_swap_id2(self, val):
        # check that the input is a valid from the list of individuals
        if int(val) in self.manager.tracklet2id:
            self.swap_id2 = int(val)
            print('ID 2 set.')
        else:
            print(f'Invalid ID. Please select a valid ID from the list of individuals: {set(self.manager.tracklet2id)}')
            self.swap_id2 = None

I've added text boxes to choose the animal identities to swap, and some options to select the frames to swap:

• A time window selected with the flags (when there are two flags fixed)
• A single frame (when there's a single flag fix)
• And from the current frame till the end (when there are no flags fixed).

The problem I'm having right now is installing the DLC locally to make modifications (the modifications I've made I tested them by changing the code installed in a conda environment, from pip). To install it from source, I've forked the DLC repository, then cloned it and I've installed it on an environment with Python 3.10 as follows:

pip install -e .[gui,tf]

The problem I'm getting is while importing deeplabcut:

In [1]: import deeplabcut as dlc
2024-07-18 12:53:05.125632: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'cudart64_110.dll'; dlerror: cudart64_110.dll not found
2024-07-18 12:53:05.126058: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
Loading DLC 2.3.10...
Traceback (most recent call last):

  File ~\AppData\Local\anaconda3\envs\dlc_local\lib\tokenize.py:332 in find_cookie
    line_string = line.decode('utf-8')

UnicodeDecodeError: 'utf-8' codec can't decode byte 0x90 in position 2: invalid start byte

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

  File ~\AppData\Local\anaconda3\envs\dlc_local\lib\site-packages\IPython\core\interactiveshell.py:3577 in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)

  Cell In[1], line 1
    import deeplabcut as dlc

  File c:\users\jalvarez\documents\code\deeplabcut\deeplabcut\__init__.py:25
    from deeplabcut.gui.tracklet_toolbox import refine_tracklets

  File c:\users\jalvarez\documents\code\deeplabcut\deeplabcut\gui\tracklet_toolbox.py:18
    from deeplabcut.refine_training_dataset.tracklets import TrackletManager

  File c:\users\jalvarez\documents\code\deeplabcut\deeplabcut\refine_training_dataset\__init__.py:13
    from deeplabcut.refine_training_dataset.tracklets import *

  File c:\users\jalvarez\documents\code\deeplabcut\deeplabcut\refine_training_dataset\tracklets.py:15
    from deeplabcut.post_processing import columnwise_spline_interp

  File c:\users\jalvarez\documents\code\deeplabcut\deeplabcut\post_processing\__init__.py:21
    from deeplabcut.post_processing.analyze_skeleton import analyzeskeleton

  File c:\users\jalvarez\documents\code\deeplabcut\deeplabcut\post_processing\analyze_skeleton.py:24
    from deeplabcut.utils import auxiliaryfunctions, auxfun_multianimal

  File c:\users\jalvarez\documents\code\deeplabcut\deeplabcut\utils\__init__.py:11
    from deeplabcut.utils.auxfun_multianimal import *

  File c:\users\jalvarez\documents\code\deeplabcut\deeplabcut\utils\auxfun_multianimal.py:34
    from deeplabcut.utils import auxiliaryfunctions, conversioncode

  File c:\users\jalvarez\documents\code\deeplabcut\deeplabcut\utils\auxiliaryfunctions.py:32
    from deeplabcut.pose_estimation_tensorflow.lib.trackingutils import TRACK_METHODS

  File c:\users\jalvarez\documents\code\deeplabcut\deeplabcut\pose_estimation_tensorflow\__init__.py:25
    from deeplabcut.pose_estimation_tensorflow.predict_videos import *

  File c:\users\jalvarez\documents\code\deeplabcut\deeplabcut\pose_estimation_tensorflow\predict_videos.py:36
    from deeplabcut.pose_estimation_tensorflow.lib import inferenceutils, trackingutils

  File c:\users\jalvarez\documents\code\deeplabcut\deeplabcut\pose_estimation_tensorflow\lib\trackingutils.py:20
    from numba import jit

  File ~\AppData\Local\anaconda3\envs\dlc_local\lib\site-packages\numba\__init__.py:92
    from numba.core.decorators import (cfunc, jit, njit, stencil,

  File ~\AppData\Local\anaconda3\envs\dlc_local\lib\site-packages\numba\core\decorators.py:12
    from numba.stencils.stencil import stencil

  File ~\AppData\Local\anaconda3\envs\dlc_local\lib\site-packages\numba\stencils\stencil.py:11
    from numba.core import types, typing, utils, ir, config, ir_utils, registry

  File ~\AppData\Local\anaconda3\envs\dlc_local\lib\site-packages\numba\core\registry.py:6
    from numba.core import utils, typing, dispatcher, cpu

  File ~\AppData\Local\anaconda3\envs\dlc_local\lib\site-packages\numba\core\dispatcher.py:22
    from numba.core.caching import NullCache, FunctionCache

  File ~\AppData\Local\anaconda3\envs\dlc_local\lib\site-packages\numba\core\caching.py:19
    from numba.misc.appdirs import AppDirs

  File ~\AppData\Local\anaconda3\envs\dlc_local\lib\site-packages\numba\misc\appdirs.py:507
    import win32com.shell

  File ~\AppData\Local\anaconda3\envs\dlc_local\lib\site-packages\win32com\__init__.py:8
    import pythoncom

  File ~\AppData\Local\anaconda3\envs\dlc_local\lib\site-packages\pythoncom.py:2
    import pywintypes

  File shibokensupport/signature/loader.py:61 in feature_imported

  File shibokensupport/feature.py:137 in feature_imported

  File shibokensupport/feature.py:148 in _mod_uses_pyside

  File ~\AppData\Local\anaconda3\envs\dlc_local\lib\inspect.py:1147 in getsource
    lines, lnum = getsourcelines(object)

  File ~\AppData\Local\anaconda3\envs\dlc_local\lib\inspect.py:1129 in getsourcelines
    lines, lnum = findsource(object)

  File ~\AppData\Local\anaconda3\envs\dlc_local\lib\inspect.py:954 in findsource
    lines = linecache.getlines(file, module.__dict__)

  File ~\AppData\Local\anaconda3\envs\dlc_local\lib\linecache.py:46 in getlines
    return updatecache(filename, module_globals)

  File ~\AppData\Local\anaconda3\envs\dlc_local\lib\linecache.py:136 in updatecache
    with tokenize.open(fullname) as fp:

  File ~\AppData\Local\anaconda3\envs\dlc_local\lib\tokenize.py:396 in open
    encoding, lines = detect_encoding(buffer.readline)

  File ~\AppData\Local\anaconda3\envs\dlc_local\lib\tokenize.py:373 in detect_encoding
    encoding = find_cookie(first)

  File ~\AppData\Local\anaconda3\envs\dlc_local\lib\tokenize.py:337 in find_cookie
    raise SyntaxError(msg)

  File <string>
SyntaxError: invalid or missing encoding declaration for 'C:\\Users\\jalvarez\\AppData\\Local\\anaconda3\\envs\\dlc_local\\lib\\site-packages\\pywin32_system32\\pywintypes310.dll'

Do you have any idea of what the problem could be?

[MMathisLab]

@JulianAlvarezdeGiorgi could you make a PR that we can review, as that would be easiest for our de-bugging, thanks again!

[JulianAlvarezdeGiorgi]

Hi @MMathisLab . Yes, just did it. However, in case it was not clear, this problem is from before making modifications, it was while installing DLC locally.