Issue Analyzing Behaviors

[sgustin1]

Hello,

I hope this message find you well! I recently trained an interactive basic categorizer (with pattern recognizer + animation analyzer) to quantify aggressive behaviors in 5 minute video clips with 2 drosophila at a time. I am using background subtraction (not Detectron). However, when I press the 'Start to analyze the behaviors' button, I continually receive this error message in the terminal:

Traceback (most recent call last): File "/Users/sgustin/.local/pipx/venvs/labgym/lib/python3.10/site-packages/LabGym/gui_analyzer.py", line 875, in analyze_behaviors AA.acquire_information_interact_basic(background_free=self.background_free) File "/Users/sgustin/.local/pipx/venvs/labgym/lib/python3.10/site-packages/LabGym/analyzebehavior.py", line 445, in acquire_information_interact_basic if self.animal_contours[0][max(0,(frame_count_analyze-self.length+1)):frame_count_analyze+1][n] is None: NameError: name 'n' is not defined. Did you mean: 'np'?

The software is able to generate an accurate image of the background (and stores this in the results folder), but fails to analyze the video inputs. Does anyone have any idea what the issue might be? I tried to run it on two different computers with LabGym installed, but still receive the same error message. I'm not sure if it an issue with my trained categorizer. I attached the training report. Thank you for developing such amazing software!

[yujiahu415]

Hi,

This error is caused by a bug. I just released a new version v2.5.3 for bug fixing, which should fix it. But let me know if it doesn't. You can upgrade LabGym by python3 -m pip install --upgrade LabGym

By the way, the training results looked not good. The accuracies seemed a little bit low. Is it just a quick test or a serious training? If you need help to improve accuracy, please let me know how you did the training, like how many pairs of training examples per category? How many frames for each behavior example? And how did you set the Categorizer?

[sgustin1]

Hello,

It seems I'm still getting this error even after running the update code & restarting my computer:

Preparation started... 2024-10-10 23:36:37.922579 Video fps: 30 The original video framesize: 1024 X 1280 Extracting the static background... Background extraction completed! Estimating the animal size... 2024-10-10 23:36:44.741218 Estimation completed! Single animal size: 1999.2953281423804 Preparation completed! Acquiring information in each frame... 2024-10-10 23:37:02.810033 Traceback (most recent call last): File "/Users/sgustin/.local/pipx/venvs/labgym/lib/python3.10/site-packages/LabGym/gui_analyzer.py", line 875, in analyze_behaviors AA.acquire_information_interact_basic(background_free=self.background_free) File "/Users/sgustin/.local/pipx/venvs/labgym/lib/python3.10/site-packages/LabGym/analyzebehavior.py", line 445, in acquire_information_interact_basic if self.animal_contours[0][max(0,(frame_count_analyze-self.length+1)):frame_count_analyze+1][n] is None: NameError: name 'n' is not defined. Did you mean: 'np'?

I was hoping just to analyze one video as a test and then go back and improve the accuracy. I used between 60-150 20 frame clips for each behavior (with at least one of the two flies doing the behavior per clip). I had less clips for the behaviors that performed the worst in the test report. I trained the categorizer as interactive basic with both pattern recognition (complexity 4, input shapes: 32,32,3) and animation analyzer (complexity 4, input shapes: 32,32,1). I set the augmentation settings to default. If you have any recommendations to improve the categorizer's accuracy, I would greatly appreciate it. Thank you again for all your help!

[yujiahu415]

Hi,

I just realized that you were using pipx. So you need to use pipx upgrade LabGym to upgrade LabGym in your pipx virtual environment. After upgrading, make sure the version number is 2.5.3 in the welcome interface of the GUI.

To improve the accuracy, you can select and sort more behavior examples to ensure at least 100 pairs of well-selected behavior examples in each category. Well-selected means examples are not redundant and cover most of the behavior variants for a category and behavior examples in different categories look differently. The setting of the Categorizer is good for now.

[sgustin1]

Thank you for clarifying. I updated with pipx, but now I am receiving this error when I try to analyze the same video with the same settings:

Single animal size: 1999.6354282536151 Preparation completed! Acquiring information in each frame... 2024-10-11 11:57:33.478190 Traceback (most recent call last): File "/Users/sgustin/.local/pipx/venvs/labgym/lib/python3.10/site-packages/LabGym/gui_analyzer.py", line 884, in analyze_behaviors AA.acquire_information_interact_basic(background_free=self.background_free) File "/Users/sgustin/.local/pipx/venvs/labgym/lib/python3.10/site-packages/LabGym/analyzebehavior.py", line 440, in acquire_information_interact_basic pattern_image=generate_patternimage_all(frame,y_bt,y_tp,x_lf,x_rt,temp_contours,temp_inners,std=self.std) File "/Users/sgustin/.local/pipx/venvs/labgym/lib/python3.10/site-packages/LabGym/tools.py", line 800, in generate_patternimage_all for inners in inners_list[n]: TypeError: 'NoneType' object is not iterable

The background extraction seems to work well, so I don't think this error is associated with the video input. I'm not exactly sure what NoneType is referring to.

[yujiahu415]

Hi,

This is another bug in the 'interactive basic' mode. I just released a new version v2.5.4 to fix it. But let me know if the issue is still there. Thanks!

[sgustin1]

Hi again,

It almost worked this time. It processed all of the frames, but then returned this error messgae:

Preparation started... 2024-10-12 00:06:42.719220 Video fps: 30 The original video framesize: 1024 X 1280 Extracting the static background... Background extraction completed! Estimating the animal size... 2024-10-12 00:08:00.189929 Estimation completed! Single animal size: 1999.6354282536151 Preparation completed! Acquiring information in each frame... 2024-10-12 00:08:18.041006 1000 frames processed... 2024-10-12 00:42:27.715867 2000 frames processed... 2024-10-12 10:54:11.431266 3000 frames processed... 2024-10-12 11:18:56.377133 4000 frames processed... 2024-10-12 11:37:33.406391 5000 frames processed... 2024-10-12 12:07:25.430863 6000 frames processed... 2024-10-12 12:21:59.287039 7000 frames processed... 2024-10-12 12:47:40.330814 8000 frames processed... 2024-10-12 13:00:39.634947

Information acquisition completed! Categorizing behaviors... 2024-10-12 13:10:26.774127 562/562 [==============================] - 361s 641ms/step Traceback (most recent call last): File "/Users/sgustin/.local/pipx/venvs/labgym/lib/python3.10/site-packages/LabGym/gui_analyzer.py", line 887, in analyze_behaviors AA.categorize_behaviors(self.path_to_categorizer,uncertain=self.uncertain) File "/Users/sgustin/.local/pipx/venvs/labgym/lib/python3.10/site-packages/LabGym/analyzebehavior.py", line 562, in categorize_behaviors i=self.length+self.register_counts[n] TypeError: unsupported operand type(s) for +: 'int' and 'NoneType'

Could it be that there is another bug?

[yujiahu415]

Hi,

Yes, this is indeed another bug. I have uploaded the new v2.5.5. Let me know if the issue is still there or if new errors arise. Thank you for helping me finding these bugs!

By the way, it seemed you include body parts in pattern images. I would suggest you not include body parts for adult fly behaviors, which may increase the behavior categorization accuracy. But that requires you to re-generate the behavior examples and re-sort them. By the way, did you include background in animations?

[sgustin123]

Hi again,

Sorry it took me a few days to get back to you. I was out of town, so I did not have access to a computer with LabGym installed. I am still receiving this error after updating it again:

The user interface initialized! Preparation started... 2024-10-16 16:30:18.512813 Video fps: 30 The original video framesize: 1024 X 1280 Extracting the static background... Background extraction completed! Estimating the animal size... 2024-10-16 16:30:25.818440 Estimation completed! Single animal size: 1999.6354282536151 Preparation completed! Acquiring information in each frame... 2024-10-16 16:30:43.717394 1000 frames processed... 2024-10-16 16:37:18.379481 2000 frames processed... 2024-10-16 16:43:42.529243 3000 frames processed... 2024-10-16 16:51:03.475143 4000 frames processed... 2024-10-16 17:00:02.544236 5000 frames processed... 2024-10-16 17:06:10.347666 6000 frames processed... 2024-10-16 17:12:55.099997 7000 frames processed... 2024-10-16 17:20:44.457992 8000 frames processed... 2024-10-16 17:28:21.116245 Information acquisition completed! Categorizing behaviors... 2024-10-16 17:33:48.586383 2024-10-16 17:33:48.745687: I tensorflow/core/platform/cpu_feature_guard.cc:151] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX AVX2 To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. 2024-10-16 17:33:49.319327: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1525] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 8943 MB memory: -> device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:01:00.0, compute capability: 7.5 2024-10-16 17:33:56.099178: I tensorflow/stream_executor/cuda/cuda_dnn.cc:368] Loaded cuDNN version 8700 Traceback (most recent call last): File "C:\Users\YLab\AppData\Local\Programs\Python\Python310\lib\site-packages\LabGym\gui_analyzer.py", line 887, in analyze_behaviors AA.categorize_behaviors(self.path_to_categorizer,uncertain=self.uncertain) File "C:\Users\YLab\AppData\Local\Programs\Python\Python310\lib\site-packages\LabGym\analyzebehavior.py", line 562, in categorize_behaviors i=self.length+self.register_counts[n] TypeError: unsupported operand type(s) for +: 'int' and 'NoneType'

I ran the software on a different computer this time. I will try to generate pattern images without body parts to improve accuracy. Also, I did include backgrounds in animations. Do you think that will pose an issue?

[yujiahu415]

Hi,

This is weird. The v2.5.5 runs well with my fly videos using the 'interactive basic' mode as you did. Although the error occurred in the step of categorizing behavior, it's likely caused by false/miss detections because the error is because the 'register count is none', which means that the pair of flies in your video hasn't been registered for analysis--this is typically due to the detection failure.

I was wondering if it's possible for you to send me the video (to: yujiahu415@gmail.com) that has this problem for me to test and debug. My videos all worked fine so I wasn't able to find where the problem is.

Thanks!

[yujiahu415]

Hi,

Thank you for sending me the video. I have found the bug and fixed it and released a new version v2.5.6. It runs fine on your video now. Please update the new version and let me know if you still encounter the issue or any new issues arise. Thanks!