Open JulianAlvarezdeGiorgi opened 1 month ago
Hi @JulianAlvarezdeGiorgi, I believe this is something that can already be done, but please correct me if I'm wrong. You can flag start and end frames to delimit a time window during which a swap occurred (https://github.com/DeepLabCut/DeepLabCut/blob/main/docs/maDLC_UserGuide.md#refine-tracklets), and then click at the top on the target animal's name (which is convenient, as you described, to handle more than 2 animals in the scene).
Hi @jeylau. Yess, you're correct, this can be done, but you need to be select the points you want to swap with the "Lasso" selector, am I right?
This is not only more laborious to do, but it can also pose a problem if you are working with videos of animals that interact a lot. For example suppose we see an identity swap, but due to the nature of the videos (interaction between animals, or hidden parts), it is difficult to select all the points in once, we will need to do it in parts, selecting the most obvious points, exchange the identity, and then go to find a new frame where the remaining points are visible to repeat the process. While it can be done, depending on the case, it can be quite complicated.
This is the example of the previous case, and shows how complex it can be (in the previous example, you could simply select a frame where both mice are separated and select all points there, but in other examples this may not be possible):
Therefore, I believe that a button that simply changes identity (without the need to select the points to be changed) in a time window may be helpful.
Ah, thanks for clarifying. You're right, that'd be very helpful! If you feel like contributing this feature to DLC, I'm more than happy to provide you with some guidance. The logic for swapping points is this dozen of lines https://github.com/DeepLabCut/DeepLabCut/blob/9e9d677832a148f2dd19b313b9fd976413f4e271/deeplabcut/gui/tracklet_toolbox.py#L643-L655. What you'd need to determine is the tracklet indices belonging to both individuals: this could be done by looking at self.manager.tracklet2id
, which maps tracklet indices to individual indices. For example, you could get all tracklet indices of individual 1 with:
inds1 = [k for k, v in self.manager.tracklet2id.items() if v == 1]
I think this should get you started 😊
Perfect, thanks for the help. I've implemented it by using the manager.swap_tracklets
method. I've done it as follows, and it works fine:
def swap_tracklets(self, event):
if self.swap_id1 is not None and self.swap_id2 is not None:
# Get tracklet indices for each individual
inds1 = [k for k in range(len(self.manager.tracklet2id)) if self.manager.tracklet2id[k] == self.swap_id1]
inds2 = [k for k in range(len(self.manager.tracklet2id)) if self.manager.tracklet2id[k] == self.swap_id2]
print(f'Swapping tracklets {self.swap_id1} and {self.swap_id2}')
# Frames to swap
frames = []
if len(self.cuts) == 2:
frames = list(range(min(self.cuts), max(self.cuts) + 1))
elif len(self.cuts) == 1:
frames = [self.cuts[0]]
else:
frames = list(range(self.curr_frame, self.manager.nframes))
# Swap the tracklets
for i in range(min(len(inds1), len(inds2))):
self.manager.swap_tracklets(inds1[i], inds2[i], frames)
self.display_traces()
self.slider.set_val(self.curr_frame)
def set_swap_id1(self, val):
# check that the input is a valid from the list of individuals
if int(val) in self.manager.tracklet2id:
self.swap_id1 = int(val)
print('ID 1 set.')
else:
print(f'Invalid ID. Please select a valid ID from the list of individuals: {set(self.manager.tracklet2id)}')
self.swap_id1 = None
def set_swap_id2(self, val):
# check that the input is a valid from the list of individuals
if int(val) in self.manager.tracklet2id:
self.swap_id2 = int(val)
print('ID 2 set.')
else:
print(f'Invalid ID. Please select a valid ID from the list of individuals: {set(self.manager.tracklet2id)}')
self.swap_id2 = None
I've added text boxes to choose the animal identities to swap, and some options to select the frames to swap:
The problem I'm having right now is installing the DLC locally to make modifications (the modifications I've made I tested them by changing the code installed in a conda environment, from pip). To install it from source, I've forked the DLC repository, then cloned it and I've installed it on an environment with Python 3.10 as follows:
pip install -e .[gui,tf]
The problem I'm getting is while importing deeplabcut:
In [1]: import deeplabcut as dlc
2024-07-18 12:53:05.125632: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'cudart64_110.dll'; dlerror: cudart64_110.dll not found
2024-07-18 12:53:05.126058: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
Loading DLC 2.3.10...
Traceback (most recent call last):
File ~\AppData\Local\anaconda3\envs\dlc_local\lib\tokenize.py:332 in find_cookie
line_string = line.decode('utf-8')
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x90 in position 2: invalid start byte
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File ~\AppData\Local\anaconda3\envs\dlc_local\lib\site-packages\IPython\core\interactiveshell.py:3577 in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
Cell In[1], line 1
import deeplabcut as dlc
File c:\users\jalvarez\documents\code\deeplabcut\deeplabcut\__init__.py:25
from deeplabcut.gui.tracklet_toolbox import refine_tracklets
File c:\users\jalvarez\documents\code\deeplabcut\deeplabcut\gui\tracklet_toolbox.py:18
from deeplabcut.refine_training_dataset.tracklets import TrackletManager
File c:\users\jalvarez\documents\code\deeplabcut\deeplabcut\refine_training_dataset\__init__.py:13
from deeplabcut.refine_training_dataset.tracklets import *
File c:\users\jalvarez\documents\code\deeplabcut\deeplabcut\refine_training_dataset\tracklets.py:15
from deeplabcut.post_processing import columnwise_spline_interp
File c:\users\jalvarez\documents\code\deeplabcut\deeplabcut\post_processing\__init__.py:21
from deeplabcut.post_processing.analyze_skeleton import analyzeskeleton
File c:\users\jalvarez\documents\code\deeplabcut\deeplabcut\post_processing\analyze_skeleton.py:24
from deeplabcut.utils import auxiliaryfunctions, auxfun_multianimal
File c:\users\jalvarez\documents\code\deeplabcut\deeplabcut\utils\__init__.py:11
from deeplabcut.utils.auxfun_multianimal import *
File c:\users\jalvarez\documents\code\deeplabcut\deeplabcut\utils\auxfun_multianimal.py:34
from deeplabcut.utils import auxiliaryfunctions, conversioncode
File c:\users\jalvarez\documents\code\deeplabcut\deeplabcut\utils\auxiliaryfunctions.py:32
from deeplabcut.pose_estimation_tensorflow.lib.trackingutils import TRACK_METHODS
File c:\users\jalvarez\documents\code\deeplabcut\deeplabcut\pose_estimation_tensorflow\__init__.py:25
from deeplabcut.pose_estimation_tensorflow.predict_videos import *
File c:\users\jalvarez\documents\code\deeplabcut\deeplabcut\pose_estimation_tensorflow\predict_videos.py:36
from deeplabcut.pose_estimation_tensorflow.lib import inferenceutils, trackingutils
File c:\users\jalvarez\documents\code\deeplabcut\deeplabcut\pose_estimation_tensorflow\lib\trackingutils.py:20
from numba import jit
File ~\AppData\Local\anaconda3\envs\dlc_local\lib\site-packages\numba\__init__.py:92
from numba.core.decorators import (cfunc, jit, njit, stencil,
File ~\AppData\Local\anaconda3\envs\dlc_local\lib\site-packages\numba\core\decorators.py:12
from numba.stencils.stencil import stencil
File ~\AppData\Local\anaconda3\envs\dlc_local\lib\site-packages\numba\stencils\stencil.py:11
from numba.core import types, typing, utils, ir, config, ir_utils, registry
File ~\AppData\Local\anaconda3\envs\dlc_local\lib\site-packages\numba\core\registry.py:6
from numba.core import utils, typing, dispatcher, cpu
File ~\AppData\Local\anaconda3\envs\dlc_local\lib\site-packages\numba\core\dispatcher.py:22
from numba.core.caching import NullCache, FunctionCache
File ~\AppData\Local\anaconda3\envs\dlc_local\lib\site-packages\numba\core\caching.py:19
from numba.misc.appdirs import AppDirs
File ~\AppData\Local\anaconda3\envs\dlc_local\lib\site-packages\numba\misc\appdirs.py:507
import win32com.shell
File ~\AppData\Local\anaconda3\envs\dlc_local\lib\site-packages\win32com\__init__.py:8
import pythoncom
File ~\AppData\Local\anaconda3\envs\dlc_local\lib\site-packages\pythoncom.py:2
import pywintypes
File shibokensupport/signature/loader.py:61 in feature_imported
File shibokensupport/feature.py:137 in feature_imported
File shibokensupport/feature.py:148 in _mod_uses_pyside
File ~\AppData\Local\anaconda3\envs\dlc_local\lib\inspect.py:1147 in getsource
lines, lnum = getsourcelines(object)
File ~\AppData\Local\anaconda3\envs\dlc_local\lib\inspect.py:1129 in getsourcelines
lines, lnum = findsource(object)
File ~\AppData\Local\anaconda3\envs\dlc_local\lib\inspect.py:954 in findsource
lines = linecache.getlines(file, module.__dict__)
File ~\AppData\Local\anaconda3\envs\dlc_local\lib\linecache.py:46 in getlines
return updatecache(filename, module_globals)
File ~\AppData\Local\anaconda3\envs\dlc_local\lib\linecache.py:136 in updatecache
with tokenize.open(fullname) as fp:
File ~\AppData\Local\anaconda3\envs\dlc_local\lib\tokenize.py:396 in open
encoding, lines = detect_encoding(buffer.readline)
File ~\AppData\Local\anaconda3\envs\dlc_local\lib\tokenize.py:373 in detect_encoding
encoding = find_cookie(first)
File ~\AppData\Local\anaconda3\envs\dlc_local\lib\tokenize.py:337 in find_cookie
raise SyntaxError(msg)
File <string>
SyntaxError: invalid or missing encoding declaration for 'C:\\Users\\jalvarez\\AppData\\Local\\anaconda3\\envs\\dlc_local\\lib\\site-packages\\pywin32_system32\\pywintypes310.dll'
Do you have any idea of what the problem could be?
@JulianAlvarezdeGiorgi could you make a PR that we can review, as that would be easiest for our de-bugging, thanks again!
Hi @MMathisLab . Yes, just did it. However, in case it was not clear, this problem is from before making modifications, it was while installing DLC locally.
Feature request. I'm on a multi-animal setup, training with top-view mice videos, without using the identity layer ('identity = False'), because of the similarities between mice. When running DLC with two mice, I run into an identity swap that I believe would be very difficult for DLC to detect automatically, therefore I plan to build a GUI to correct these swaps manually.
Here is an example:![image](https://github.com/user-attachments/assets/f29f0188-c8f5-4dc9-b509-cb856ed637e2)
Describe the solution you'd like I would like to add to the existing "Refinement Tracklets GUI" a new button allowing fast correction of identity swaps.
Pressing this button swaps 2 animal identities from the current time until the movie ends.
Example: after pressing the "SWAP" button, we get:![image](https://github.com/user-attachments/assets/a9fe0b96-4dcb-4f02-bae7-b490f3a087bb)
If working with more than 2 animals, we might need additional controls to specify which animals have an identity swap. We could also add more buttons such as "NEXT ANIMAL CROSSING" to jump to the next potential swap due to animals crossing.
Describe alternatives you've considered I also considered making a separate GUI dedicated to identity swap, rather than adding new features to the existing Refinement Tracklets GUI.
My questions Can this be added to DLC? I would be happy to try developing it myself. In this case, probably I'll need help to know which code to edit, what to pay attention to, etc. Furthermore; I'd appreciate any advice for doing it in a way that I can later push this new functionality to the DLC project.