Validate Model on Single Video selection is bugged in current version

[rfkova]

Describe the bug When I click Browse File next to Select Model File on the [Run Machine Model] page I get a weird error instead of loading the file.

I rolled back to simba-uw-tf-dev==1.73.9 and selection works fine and the issue is resolved.

Error Exception in Tkinter callback Traceback (most recent call last): File "C:\Users\RFK\anaconda3\envs\simba\lib\tkinter__init.py", line 1705, in call__ return self.func(*args) File "C:\Users\RFK\anaconda3\envs\simba\lib\site-packages\simba\ui\tkinter_functions.py", line 130, in setFilePath file_selected = askopenfilename(title=self.title, parent=self.parent, filetypes=self.file_type) File "C:\Users\RFK\anaconda3\envs\simba\lib\tkinter\filedialog.py", line 375, in askopenfilename return Open(**options).show() File "C:\Users\RFK\anaconda3\envs\simba\lib\tkinter\commondialog.py", line 35, in show self._fixoptions() File "C:\Users\RFK\anaconda3\envs\simba\lib\tkinter\filedialog.py", line 302, in _fixoptions self.options["filetypes"] = tuple(self.options["filetypes"]) TypeError: 'NoneType' object is not iterable

Desktop (please complete the following information):

• OS: Win 10, 64-bit
• Python Version 3.6.13
• Are you using anaconda? Yes

Thank you! Ryan

[rfkova]

A separate, but possibly related issue, in 1.73.9, I was able to run a model on a features_extracted file, then open the Interactive Probability Plot and scan around, but when I tried to CREATE VALIDATION VIDEO I get the following error...

SIMBA FEATURE NUMBER MISMATCH ERROR: Mismatch in the number of features in input file R:/Basic_Sciences/Phys/Lerner_Lab_tnl2633/Ryan/camera_stuff/SIMBA/dam_nest-c-only/project_folder/csv/features_extracted/LBNF2_Ctrl_P04_4_2021-03-18_19-49-46c.csv, and what is expected by the model dam_in_nest_attentive. The model expects 82 features. The data contains 97 features.

This strikes me as odd, because the could be run on the same file to generate the probabilities... I noticed the model generated a validation folder with a new file in it... this file produced a similar error.

Help! Ryan

[sronilsson]

Thank you @rfkova ! The first issue comes from me being a bit gung-ho yesterday and updating without testing. Many thanks for reporting.

[sronilsson]

Could you update SimBA again (to version 1.75.3) and let me know the error you see in the terminal when trying to create the validation video?

Do you see the same error for all of the files inside the features_extracted directory? Or, do all of your files inside the features_extracted directory contain the same number of columns?

[rfkova]

I have tried two videos previously. They are all using the standard feature extractor and so yes, have the same number of columns. I haven't verified for EVERY file, but those I have tried are the same and produce the same error and I spot checked a few others, so there is mounting evidence, at least, to suggest the files are all the same, haha.

An aside Also, I do not know if it is a bug, but I have to change the .sav file of the model to perfectly match the behavior markers- the model saves with an underscore_number that I had to remove... otherwise I get an error like this: SIMBA VALUE ERROR: The classifier dam_in_nest_attentive_1 is not a classifier in the SimBA project: ['dam_in_nest', 'dam_in_nest_attentive', 'dam_in_nest_active'] 🚨

Back to it! Ok! Different problem now. I still get the above error if I don't change the .sav file --> I CAN run the model on my extracted feature file, but now when I try to open the INTERACTIVE PROBABILITY PLOT a blank GUI window opens and I get this error: SIMBA VIDEO FILE ERROR: Video LBNF2_Ctrl_P04_4_2021-03-18_19-49-46c either does not exist or has fps of 0 (full error video path: R:/Basic_Sciences/Phys/Lerner_Lab_tnl2633/Ryan/camera_stuff/SIMBA\dam_nest-c-only\project_folder\videos\LBNF2_Ctrl_P04_4_2021-03-18_19-49-46c.mp4). 🚨 ... so I opened the same video file afterwards with Label Behavior portion of the GUI and it is totally fine... and again, this part worked in 1.73.9. I do notice the slashes are flipping out, but idk if that's normal.

Let me know if there is anything I can upload for testing :D Ryan

PS @sronilsson, this is a symlink... idk if the labeling code and the INTERACTIVE PROBABILITY PLOT handle those differently or not...

[sronilsson]

Thanks @rfkova - I will test it, using symlinks, but bogged down today with a deadline, I will get back to you asap. If there is a project I can use on a gdrive you could share that would be good (removing any unnessery big files) in the case I can reproduce it using my own projects.

[rfkova]

Thanks for the heads up about timing. Good luck with your deadline and best wishes for health and success!

I will upload something for you today and be back with the a link... However, the model files are kind of large themselves and the videos aren't tiny... I'll trim what I can to provide a workable example including the offending video.

[sronilsson]

Cheers - I did try it with symlinks on a small testing project I have and didn't get any errors on interactive plot. If you have the error traceback from the operating system terminal used to launch simba that would help a lot too to see what functions are fed something they don't like.

[rfkova]

Ohhhhhkay. Not sure what was happening but in the course of troubleshooting, the extracted features for the video I was trying got misplaced - INTERACTIVE PROBABILITY PLOT is back working, but same error with the validation video. Here I:

1. re-ran the model to make sure all the files were in place
2. Checked the probability plot
3. Tried to create validation video - which produced an error like above.

Command Line

[Parallel(n_jobs=16)]: Using backend ThreadingBackend with 16 concurrent workers.
[Parallel(n_jobs=16)]: Done  18 tasks      | elapsed:    0.0s
[Parallel(n_jobs=16)]: Done 168 tasks      | elapsed:    0.2s
[Parallel(n_jobs=16)]: Done 418 tasks      | elapsed:    0.6s
[Parallel(n_jobs=16)]: Done 768 tasks      | elapsed:    1.3s
[Parallel(n_jobs=16)]: Done 1218 tasks      | elapsed:    2.0s
[Parallel(n_jobs=16)]: Done 1768 tasks      | elapsed:    2.9s
[Parallel(n_jobs=16)]: Done 2000 out of 2000 | elapsed:    3.2s finished
Exception in Tkinter callback
Traceback (most recent call last):
  File "C:\Users\rfk416\Anaconda3\envs\simba\lib\tkinter\__init__.py", line 1705, in __call__
    return self.func(*args)
  File "C:\Users\rfk416\Anaconda3\envs\simba\lib\site-packages\simba\SimBA.py", line 388, in <lambda>
    button_generateplot = Button(label_model_validation, text="INTERACTIVE PROBABILITY PLOT", fg='blue', command= lambda: self.launch_interactive_plot())
  File "C:\Users\rfk416\Anaconda3\envs\simba\lib\site-packages\simba\SimBA.py", line 642, in launch_interactive_plot
    interactive_grapher.run()
  File "C:\Users\rfk416\Anaconda3\envs\simba\lib\site-packages\simba\plotting\interactive_probability_grapher.py", line 97, in run
    fig.canvas.draw()
  File "C:\Users\rfk416\Anaconda3\envs\simba\lib\site-packages\matplotlib\backends\backend_tkagg.py", line 10, in draw
    _backend_tk.blit(self._tkphoto, self.renderer._renderer, (0, 1, 2, 3))
  File "C:\Users\rfk416\Anaconda3\envs\simba\lib\site-packages\matplotlib\backends\_backend_tk.py", line 88, in blit
    photoimage.blank()
  File "C:\Users\rfk416\Anaconda3\envs\simba\lib\tkinter\__init__.py", line 3548, in blank
    self.tk.call(self.name, 'blank')
_tkinter.TclError: invalid command name "pyimage398"
Exception in Tkinter callback
Traceback (most recent call last):
  File "C:\Users\rfk416\Anaconda3\envs\simba\lib\tkinter\__init__.py", line 1705, in __call__
    return self.func(*args)
  File "C:\Users\rfk416\Anaconda3\envs\simba\lib\site-packages\simba\mixins\pop_up_mixin.py", line 164, in <lambda>
    self.run_btn = Button(self.run_frm, text=title, fg='blue', command=lambda: run_function())
  File "C:\Users\rfk416\Anaconda3\envs\simba\lib\site-packages\simba\ui\pop_ups\validation_plot_pop_up.py", line 111, in run
    validation_video_creator.run()
  File "C:\Users\rfk416\Anaconda3\envs\simba\lib\site-packages\simba\plotting\single_run_model_validation_video_mp.py", line 146, in run
    self.__run_clf()
  File "C:\Users\rfk416\Anaconda3\envs\simba\lib\site-packages\simba\plotting\single_run_model_validation_video_mp.py", line 86, in __run_clf
    self.in_df[self.prob_col_name] = self.clf_predict_proba(clf=self.clf, x_df=self.in_df, model_name=self.clf_name, data_path=self.feature_file_path)
  File "C:\Users\rfk416\Anaconda3\envs\simba\lib\site-packages\simba\mixins\train_model_mixin.py", line 996, in clf_predict_proba
    raise FeatureNumberMismatchError(f'Mismatch in the number of features in input file {data_path}, and what is expected by the model {model_name}. The model expects {clf_n_features} features. The data contains {len(x_df.columns)} features.', source=self.__class__.__name__)
simba.utils.errors.FeatureNumberMismatchError: SIMBA FEATURE NUMBER MISMATCH ERROR: Mismatch in the number of features in input file R:/Basic_Sciences/Phys/Lerner_Lab_tnl2633/Ryan/camera_stuff/SIMBA/dam_nest-c-only/project_folder/csv/features_extracted/LBNF2_Ctrl_P04_4_2021-03-18_19-49-46c.csv, and what is expected by the model dam_in_nest_active. The model expects 82 features. The data contains 97 features.

GUI log: SIMBA COMPLETE: Loaded project R:/Basic_Sciences/Phys/Lerner_Lab_tnl2633/Ryan/camera_stuff/SIMBA/dam_nest-c-only/project_folder/project_config.ini 🚀 SIMBA COMPLETE: Loaded project R:/Basic_Sciences/Phys/Lerner_Lab_tnl2633/Ryan/camera_stuff/SIMBA/dam_nest-c-only - Copy (2)/project_folder/project_config.ini 🚀 SIMBA COMPLETE: Loaded project R:/Basic_Sciences/Phys/Lerner_Lab_tnl2633/Ryan/camera_stuff/SIMBA/dam_nest-c-only/project_folder/project_config.ini 🚀 Pose-estimation body part setting for feature extraction: 2 animals user_defined body-parts Extracting features from 1 file(s)... Extracting features for video 1/1... Calculating euclidean distances... Calculating movements of all body-parts... Calculating rolling windows data: distances between body-parts... Calculating rolling windows data: animal movements... Feature extraction complete for video LBNF2_Ctrl_P04_4_2021-03-18_19-49-46c (elapsed time: 3.6791s) SIMBA COMPLETE: Feature extraction complete for 1 video(s). Results are saved inside the project_folder/csv/features_extracted directory (elapsed time: 3.7578s) 🚀 SIMBA COMPLETE: Validation predictions generated for "LBNF2_Ctrl_P04_4_2021-03-18_19-49-46c" within the project_folder/csv/validation directory (elapsed time: 81.1842s) 🚀 Click on "Interactive probability plot" to inspect classifier probability thresholds. If satisfactory proceed to specify threshold and minimum bout length and click on "Validate" to create video. SIMBA FEATURE NUMBER MISMATCH ERROR: Mismatch in the number of features in input file R:/Basic_Sciences/Phys/Lerner_Lab_tnl2633/Ryan/camera_stuff/SIMBA/dam_nest-c-only/project_folder/csv/features_extracted/LBNF2_Ctrl_P04_4_2021-03-18_19-49-46c.csv, and what is expected by the model dam_in_nest_active. The model expects 82 features. The data contains 97 features. 🚨

Link to Drive file is incoming, upload is taking forever

[rfkova]

@sronilsson Please hold. I created the validation video without multiprocessing and with no minimum bout length without an error but with a warning that "Some frames appear to be missing in the video vs the data file"... but I can't open the file... let me explore and get back to you

[rfkova]

I was able to get a corrupted video (read unplayable and like 6kb in size) so long as I don't use multiprocessing. Multiprocessing is causing the specific bug I showed earlier with the 97 features.

If I disable Show Pose in Tracking options, it looks like I can get a video out.

I would really like to be able to get the pose estimation on there... kind of an aside, but is there an easy way in the GUI to visualize the poses on a video without the probability (i.e. prior to running a model) out of it for the purpose to evaluate the outlier correction/interpolation.

Here's the drive link. I guess the symlinks aren't transferrable, so the file structure is only so useful. Let me know if I left anything you might need out!

[sronilsson]

@rfkova Yes, I found the error. It comes from when SimBA is trying to calculate how many colors will be needed, in how many different palettes, to draw the images. To do this it gets all the unique body-parts and divides it with the number of animals:

You would get (rounded up for "false" safety) 5 / 2 = 3 colors per color palette and animal. But in your use-case this won't be enough: you have an animal with 4 body-parts and there is no 4th color in the animals pallette so it breaks. I will insert some better logic. The error is kind of badly hidden in a try and except statement also so its tricky to troubleshoot. The try and excpet statement is there because sometimes, users have a mismatch between the frames they have in the video, and the number of frames in the pose-estimation data, but they don't want it to error out, they want to see the video up until the point it errored out. And in your case it errors out on frame 0.

EDIT: Thanks for reporting this btw. It is very helpful. The code runs, I'm just going to make sure your video is created using multiptocessing and I can watch it before pushing an update.

[sronilsson]

Let me know how it goes in version 1.75.6 - you have tons of frames so I recommend using multiprocessing, and maybe hide the animal names as it looks a little weird when the nest is called Animal_2.

[rfkova]

Hey! It's working! Thank you so much :D

One outstanding questions not related to the errors: Is there an easy way in the GUI to visualize the poses on a video without the probability (i.e. prior to running a model) out of it for the purpose to evaluate the outlier correction/interpolation.

[sronilsson]

Yes there is a tool through the Tools drop-down menu called Visualize pose-estimation in folder documented HERE let me know if there are any problems though.

PS. THIS is the code that the GUI calls, when using more than 1 core.

[rfkova]

It worked for the first video in the pile but I got this error in the terminal before it got to the next video:

Exception in Tkinter callback Traceback (most recent call last): File "C:\Users\rfk416\Anaconda3\envs\simba\lib\tkinter\__init__.py", line 1705, in __call__ return self.func(*args) File "C:\Users\rfk416\Anaconda3\envs\simba\lib\site-packages\simba\ui\pop_ups\visualize_pose_in_dir_pop_up.py", line 31, in <lambda> run_btn = Button(self.main_frm, text='VISUALIZE POSE', font=Formats.LABELFRAME_HEADER_FORMAT.value, fg='blue', command= lambda: self.run()) File "C:\Users\rfk416\Anaconda3\envs\simba\lib\site-packages\simba\ui\pop_ups\visualize_pose_in_dir_pop_up.py", line 68, in run pose_plotter.run() File "C:\Users\rfk416\Anaconda3\envs\simba\lib\site-packages\simba\plotting\pose_plotter_mp.py", line 115, in run self.config.animal_bp_dict[animal]['colors'] = tuple(self.config.animal_bp_dict[animal]['colors'][0]) TypeError: 'numpy.float64' object is not iterable

[sronilsson]

Cheers, lets take a look!

[sronilsson]

I recreated the error and inserted a fix in 1.75.7 but let me know if it fails on your end.

[rfkova]

Hm, it got through two this time. I see the colors are changing on the body parts from video to video. Not sure if that is intended. Here's my command line error after 2 successful videos:

Exception in Tkinter callback Traceback (most recent call last): File "C:\Users\rfk416\Anaconda3\envs\simba\lib\tkinter\__init__.py", line 1705, in __call__ return self.func(*args) File "C:\Users\rfk416\Anaconda3\envs\simba\lib\site-packages\simba\ui\pop_ups\visualize_pose_in_dir_pop_up.py", line 31, in <lambda> run_btn = Button(self.main_frm, text='VISUALIZE POSE', font=Formats.LABELFRAME_HEADER_FORMAT.value, fg='blue', command= lambda: self.run()) File "C:\Users\rfk416\Anaconda3\envs\simba\lib\site-packages\simba\ui\pop_ups\visualize_pose_in_dir_pop_up.py", line 68, in run pose_plotter.run() File "C:\Users\rfk416\Anaconda3\envs\simba\lib\site-packages\simba\plotting\pose_plotter_mp.py", line 115, in run self.config.animal_bp_dict[animal]['colors'] = self.config.animal_bp_dict[animal]['colors'][0] IndexError: invalid index to scalar variable.

[sronilsson]

Bugs bugs, sorry about this I should take better care, thanks for sticking with me!

[rfkova]

I'm honestly glowing with the experience. I know the bugs aren't ideal, but like, this is all coming together. Happy to help after all we've been through tailoring things to my application, haha!

[sronilsson]

Thank you :) I have your project so I will test on that instead as I cant replicate this one on my test projects.

[sronilsson]

When you get a chance, how does it look in version 1.75.8?

[rfkova]

Works like a charm!!!!!!! Thank you so much :D

[rfkova]

Hello @sronilsson!

It has been some time :) - since my issue relates directly to this workflow I'm adding on here once again (lmk if this isn't your favored policy).

I'm adapting this workflow to another set of data that is less complicated but it seemed easiest to adapt this than figure out a separate workflow for what I need from command line and I ran into some issues that I don't understand.

1. This should be easy - when I run OutlierCorrecterLocationAdvanced I don't get a csv out I get a file that has csv appended to the filename without a '.' ... I suspect f"{self.file_type} in 252 of OutlierCorrecterLocationAdvanced should be like ["." + self.file_type] from earlier?
2. I can get through my script import_h5_shuttlebox_avoidance.py (see below for code in Drive folder) which imports videos, h5 pose estimation, runs movement outlier correction (if the above is fixed, also location), and interpolation but then I get an error about symlink/video accessibility when I run smoothing... which is confusing since the prior steps all do the same to the same file.

I can actually use the GUI for this project which I will switch to, but I really would like to understand why I got that error so it won't haunt me later. Here is the text of the error from the command line:

smoother.run()
Smoothing data in video Box1_14Jun2022_16-10-21...
warning: Error opening file (/build/opencv/modules/videoio/src/cap_ffmpeg_impl.hpp:901)
warning: R:/Basic_Sciences/Phys/Lerner_Lab_tnl2633/Ryan/Gaby_SIMBA\shuttlebox_avoidance\project_folder\videos\Box1_14Jun2022_16-10-21.mp4 (/build/opencv/modules/videoio/src/cap_ffmpeg_impl.hpp:902)
ackages\simba\data_processors\interpolation_smoothing.py", line 680, in run
    video_meta_data = get_video_meta_data(video_path=video_path)
  File "C:\Users\rfk416\Anaconda3\envs\simba\lib\site-packages\simba\utils\read_write.py", line 447, in get_video_meta_data
    source=get_video_meta_data.__name__,
simba.utils.errors.InvalidVideoFileError: SIMBA VIDEO FILE ERROR: Video Box1_14Jun2022_16-10-21 either does not exist or has fps of 0 (full error video path: R:/Basic_Sciences/Phys/Lerner_Lab_tnl2633/Ryan/Gaby_SIMBA\shuttlebox_avoidance\project_folder\videos\Box1_14Jun2022_16-10-21.mp4).>>> SMOOTHING_SETTINGS
{'Animal_1': {'method': 'Savitzky Golay', 'time_window': 100}}

Here is a link to a Drive folder with the video, h5 file, full cmd line text from the run of the code that made the error, and the offending project file. I tried this with the previous version of simba that was working and also an --upgrade version and both have the same issues.

Anything in my code after the smooth.run() is really my problem... I think it might be partially finished/working at the moment but that's where I'm stuck.

Thanks so much! If other things are pressing this is only moderate priority for me since I do have a path forward. Best, Ryan

[rfkova]

@sronilsson oh, interesting, actually using the GUI also produces the symlink error... This wasn't an issue for me previously and I'm not sure how to proceed. For now I'll skip smoothing, but I would like to have that feature if possible... that said it may be difficult to troubleshoot since my files are on a server... however, the way that the interpolation runs and the smoothing doesn't seems odd.

Here's the log text from the GUI run: 2024-03-27T12:02:59Z|SimbaProjectPopUp.stdout_success||complete||Loaded project R:/Basic_Sciences/Phys/Lerner_Lab_tnl2633/Ryan/Gaby_SIMBA/shuttlebox_avoidance/project_folder/project_config.ini 2024-03-27T12:04:41Z|.PermissionError||error||SIMBA PERMISSION ERROR: Symbolic link privilege not held. Try running SimBA in terminal opened in admin mode 2024-03-27T12:05:37Z|SimbaProjectPopUp.stdout_success||complete||Loaded project R:/Basic_Sciences/Phys/Lerner_Lab_tnl2633/Ryan/Gaby_SIMBA/shuttlebox_avoidance/project_folder/project_config.ini 2024-03-27T12:06:03Z|copy_multiple_videos_to_project.stdout_success||complete||148 videos copied to project. 2024-03-27T12:07:08Z|SLEAPImporterH5||CLASS_INIT||smoothing_settings: {'Method': 'Savitzky Golay', 'Parameters': {'Time_window': '100'}}, interpolation_settings: Animal(s): Quadratic, id_lst: ['Animal_1'], data_folder: R:/Basic_Sciences/Phys/Lerner_Lab_tnl2633/Ryan/Gaby_SIMBA/h5_file_repository/Awaiting Import, config_path: R:/Basic_Sciences/Phys/Lerner_Lab_tnl2633/Ryan/Gaby_SIMBA/shuttlebox_avoidance/project_folder/project_config.ini 2024-03-27T12:07:14Z|Interpolate||CLASS_INIT||initial_import_multi_index: True, method: Animal(s): Quadratic, config_path: R:/Basic_Sciences/Phys/Lerner_Lab_tnl2633/Ryan/Gaby_SIMBA/shuttlebox_avoidance/project_folder/project_config.ini, input_path: R:/Basic_Sciences/Phys/Lerner_Lab_tnl2633/Ryan/Gaby_SIMBA\shuttlebox_avoidance\project_folder\csv\input_csv\Box1_06Jun2022_12-32-46.csv 2024-03-27T12:07:17Z|Interpolate.stdout_success||complete||1 data file(s) interpolated) 2024-03-27T12:07:17Z|Smooth||CLASS_INIT||input_path: R:/Basic_Sciences/Phys/Lerner_Lab_tnl2633/Ryan/Gaby_SIMBA\shuttlebox_avoidance\project_folder\csv\input_csv\Box1_06Jun2022_12-32-46.csv, time_window: 100, smoothing_method: Savitzky Golay, initial_import_multi_index: True 2024-03-27T12:07:20Z|get_video_meta_data.InvalidVideoFileError||error||SIMBA VIDEO FILE ERROR: Video Box1_06Jun2022_12-32-46 either does not exist or has fps of 0 (full error video path: R:/Basic_Sciences/Phys/Lerner_Lab_tnl2633/Ryan/Gaby_SIMBA\shuttlebox_avoidance\project_folder\videos\Box1_06Jun2022_12-32-46.mp4).

[sronilsson]

Hey @rfkova!

A little background to see if we can fix it.

When performing smoothing and providing a time window, we need to know the FPS of the video so we know how many frames represent a 100 milliseconds so we can smooth the data according to your settings. One possibility is to read the FPS from the appropriate row in the project_folder/logs/video_info.csv file, which SimBA often does, but in some cases, like this one, we can't do that - because we are importing the video and we can't assume that Box1_06Jun2022_12-32-46 exist in the project_folder/logs/video_info.csv as yet.

Alternative is to get the FPS from reading the meta data from the actual video file, which we make the assumption to be in the videos directory of the SimBa project, at location project_folder\videos\Box1_06Jun2022_12-32-46.mp4. THIS is the function which that path passes through. However, sometimes this file doesn't exist, or is corrupted, and then we have nowhere to turn to get the FPS, and this error comes up.

Just to check, does the R:/Basic_Sciences/Phys/Lerner_Lab_tnl2633/Ryan/Gaby_SIMBA\shuttlebox_avoidance\project_folder\videos\Box1_06Jun2022_12-32-46.mp4 file exist?

[rfkova]

@sronilsson,

Well, yes, a symlink exists for project_folder\videos\Box1_06Jun2022_12-32-46.mp4 but not the video (I wasn't able to upload the symlink in the Drive folder provided). The way this is set up, the video will be imported first, so in this case we can assume the info will be in project_folder/logs/video_info.csv if that helps.

The way my code (import_h5_shuttlebox_avoidance.py) is set up the video is imported first which I kind of thought was a prerequisite for smoothing but you imply may not be. The video is indeed at the location indicated by the symlink, if it wasn't the symlink wouldn't be created.

Let me know if there is anything else I should check or try! Best, Ryan

[rfkova]

@sronilsson,

Just tested and it works if I import the videos directly. Would be nice to use symlinks so I don't waste a ton of time moving/copying videos... this is more relevant to another project with like 1000 videos. This one only has 148 so it wouldn't be the worst, but would like symlinks to work too!

[sronilsson]

@rfkova Yeah of course, definitely this should be fixed - I'll see if I can recreate it with symlinks, but I can only do it locally 😬 I don't have a server available to me so can't completely try to recreate - could there be any permission errors that fail silently, if you run the conda environment in admin mode, you get the same error?

[rfkova]

@sronilsson Yeah, I had errors earlier in the process if I didn't run in admin mode so I am now... I also had to run this in my terminal to get admin-terminal to see my server folders too for some reason at earlier parts of the process (importing videos and generating the symlinks I think): net use R: \fsmresfiles.fsm.northwestern.edu\fsmresfiles /savecred /p:yes

After I found that fix everything worked before... but not in this new project... it's possible I'm doing something wrong but the way everything else seems to work has me thinking it might not be me.

I'm skeptical it's a permission issue, but it isn't impossible... if you can think of any other way to check that lmk...

[sronilsson]

@rfkova One thing for troubleshooting, to find out if it is symlink, or symlink + server issue: if you place the symlink in the videos folder, where the symlink points to a video file also located locally on the same machine as running simba, does it still fail?

[rfkova]

@sronilsson ,

1. Needed admin for importing the video via symlink (lol), even locally.
2. Oh... oh no... you were right. It works with a symlink to a local file. It's so odd because this pipeline worked for me before on a different project. Dang... maybe something IT changed. Any advice? Seems like a me-problem you might not be able to help with/troubleshoot.

[sronilsson]

There are a few ways to dig....

You could check if it is SimBA+server related, or just server, by trying to read the FPS of the symlink on server outside SimBA.

E.g., does this give you the correct fps printed?

symlink_path = "R:/Basic_Sciences/Phys/Lerner_Lab_tnl2633/Ryan/Gaby_SIMBA\shuttlebox_avoidance\project_folder\videos\Box1_06Jun2022_12-32-46.mp4"

import cv2
cap = cv2.VideoCapture(symlink_path)
fps = cap.get(cv2.CAP_PROP_FPS)
print(fps)

Otherwise, could the videos have changed location to a folder or directory tree with restrictions that didn't exist previously?

[rfkova]

Hmmm, it's just the server. When I run that (different video) I get the following. I'll maybe ask some IT people about it, something must have changed.

>>> symlink_path = r"R:\Basic_Sciences\Phys\Lerner_Lab_tnl2633\Ryan\Gaby_SIMBA\shuttlebox_avoidance\project_folder\videos\Box1_14Jun2022_16-10-21"
>>> import cv2
>>> cap = cv2.VideoCapture(symlink_path)
warning: Error opening file (/build/opencv/modules/videoio/src/cap_ffmpeg_impl.hpp:901)
warning: R:\Basic_Sciences\Phys\Lerner_Lab_tnl2633\Ryan\Gaby_SIMBA\shuttlebox_avoidance\project_folder\videos\Box1_14Jun2022_16-10-21 (/build/opencv/modules/videoio/src/cap_ffmpeg_impl.hpp:902)

Thank you so much for your help <3<3<3 Ryan

[sronilsson]

Ah just to check - I see mp4 extension missing in symlink:

symlink_path = r"R:\Basic_Sciences\Phys\Lerner_Lab_tnl2633\Ryan\Gaby_SIMBA\shuttlebox_avoidance\project_folder\videos\Box1_14Jun2022_16-10-21"

If you add .mp4 does it run?

[rfkova]

Great eye, sadly did not fix the issue:

>>> symlink_path = "R:\Basic_Sciences\Phys\Lerner_Lab_tnl2633\Ryan\Gaby_SIMBA\shuttlebox_avoidance\project_folder\videos\Box1_14Jun2022_16-10-21.mp4"
>>> import cv2
>>> cap = cv2.VideoCapture(symlink_path)
warning: Error opening file (/build/opencv/modules/videoio/src/cap_ffmpeg_impl.hpp:901)
warning: R:\Basic_Sciences\Phys\Lerner_Lab_tnl2633\Ryan\Gaby_SIMBA\shuttlebox_avoidance\project_folder♂ideos\Box1_14Jun2022_16-10-21.mp4 (/build/opencv/modules/videoio/src/cap_ffmpeg_impl.hpp:902

[rfkova]

@sronilsson While I have you (and I can create another Issue for this if you like) - I'm playing with ROIs visualizations and interested in the velocity by time bin... goal is to plot velocity on top of some dopamine signals but the 1s minimum in the GUI is kind of restrictive... can I get below 1s? I have 30fps so a reasonable minimum would be 0.1s or 3 frames I would think...

[sronilsson]

👍🏻 Let me take a look at the logic, to see if I had some reason for restricting it to integers, because I cant remember on the top of my head.

Just to confirm, it's velocity by ROI in time bins?

[rfkova]

Yep! Really just velocity in all time bins, but I think if my ROIs span the whole arena it will give me that.

[sronilsson]

Got it - one more question, I think there is an option to calculate sum of movement in time-bins that is non-ROI based, would that work if I made it possible to input 0.N sized time-bin size?

[rfkova]

The output I most need is velocity in small time bins over the whole video, it does not need to be ROI-related. If "sum of movement" refers to distance over time and not an aggregation of velocity bins, then yes, that's exactly right! (phrasing was a little difficult to parse but I think we are on the same page).

Thank you so much for helping with this!

[sronilsson]

@rfkova Yeah I think I forced integers as when I wrote this code, I didn’t have much error handling, and users set time-bin size to say 0.5, and had some videos at say 25fps, but some other videos they analyzed at the same time where 1fps, and when it hit that 1fps video, it errored out so I stopped them for doing that.. 😬

Anyway, there is better error handling now. If you use this menu in latest version, do you get what you want?

Watch out though, if you have 1000s of videos and they are fairly long, and we want to compute movements in time-bins of 100ms bins, I can see this going pretty slow as currently written. We can speed it up though if the wait is unacceptable long. Also let me know if you need an example for running it from the command line.

[rfkova]

Hi @sronilsson!

Sorry, got completely side tracked so couldn't test until just now. It works well and at a reasonable pace for this project! I'm concerned though that the velocity and distance (cm/s and cm) are identical for all bins I looked at... that can't be right, can it?

Different bins have different numbers but within a bin (say bin 12) both the cm/s and cm are identical and this was true for every bin.

Thanks again for looking into this!!!

[sronilsson]

@rfkova Ah yeah that makes sense and thanks for pointing that ou!

Velocity is sum displacement over time and, I hardcoded time as seconds. So if you have a 5s window I check the displacement in each 1s sub time-window within each 5s bin, and return the average of those 5 values as cm/s.

If you have a time-window which is 1s, then the velocity (cm/s) and sum of movement (cm) will be the same. If you have a 100ms time-windows, we can't compute cm/s as there is not one second of data and I think the code just takes what is available which is 100ms.. Not sure how best to fix it on top how my head, any ideas welcome!

[rfkova]

@sronilsson wooooooord.

This might be silly without looking at the code, but couldn't you just... not hard code it? So if an animal moves 1cm in 0.1s 1cm/0.1s just comes to 10cm/s which is the accurate velocity despite only moving 1cm. If the bin length comes in variable form you should just be able to tinker with your sub-window iteration based on its length...

If that doesn't help, can you direct me to the code so I can examine the issue directly? There could easily be a reason why the hard coding makes this not super easy but it isn't intuitive based on your description <3

[sronilsson]

Of course! The code is right HERE But I can fix that.

[rfkova]

So I think the main issue is line 92... something like this might get at least part way there... I just did this in my head though, testing empirically would be better but I'm not 100% sure how to do that and would have to muddle through (and have never used pandas before)...

If you are unlikely to have time today I can test this empricially and get you working code but if it isn't too much trouble I would appreciate the leg up.

for bin, results in enumerate(results_df_lists):
            time_bin_per_s = [results[i : i + bin_length_fr] for i in range(0, results.shape[0], bin_length_fr)]
            for second, df in enumerate(time_bin_per_s):
                df["Time bin #"] = bin
                df["Second"] = second*bin_length_fr
                indexed_df.append(df)

[sronilsson]

Yep! I think I got it... the code was written a while back when my head was working differently :) I dont't think that index think is necessery...

            results = []
            for animal_data in self.bp_dict.values():
                name, bps = list(animal_data.keys())[0], list(animal_data.values())[0]
                bp_time_1, bp_time_2 = (data_df[bps].values, data_df[[f"{bps[0]}_shifted", f"{bps[1]}_shifted"]].values)
                movement_data = pd.DataFrame(self.framewise_euclidean_distance(location_1=bp_time_1, location_2=bp_time_2, px_per_mm=px_per_mm, centimeter=True,), columns=["VALUE"])
                movement_df_lists = [movement_data[i : i + bin_length_frames] for i in range(0, movement_data.shape[0], bin_length_frames)]
                for bin, movement_df in enumerate(movement_df_lists):
                    time_bin_sliced = [movement_df[i : i + fps] for i in range(0, movement_df.shape[0], fps)]
                    bin_velocities = []
                    for slice, df in enumerate(time_bin_sliced):
                        bin_velocities.append(df['VALUE'].sum() * (1 / (len(df) / fps)))
                    results.append({"VIDEO": video_name, 'TIME BIN #': bin, 'ANIMAL': name, "BODY-PART": bps[0][:-2], "MEASUREMENT": "Movement (cm)", 'VALUE': round(movement_df.sum(), 4)})
                    results.append({"VIDEO": video_name, 'TIME BIN #': bin, 'ANIMAL': name, "BODY-PART": bps[0][:-2], "MEASUREMENT": "Velocity (cm/s)", 'VALUE': round(np.mean(bin_velocities), 4)})
            results = pd.DataFrame(results).reset_index(drop=True)
            out_df_lst.append(results)

This part slices the dataframe into one dataframe per use-defined bin:

movement_df_lists = [movement_data[i : i + bin_length_frames] for i in range(0, movement_data.shape[0], bin_length_frames)]

Next, that bin is split into one sub-bin per second of the bin:

time_bin_sliced = [movement_df[i : i + fps] for i in range(0, movement_df.shape[0], fps)]

Next, we calculate the velocity in each of those "seconds" sub-bins, but that into account of if the subbin is shorter than one second:

bin_velocities = []
for slice, df in enumerate(time_bin_sliced):
   bin_velocities.append(df['VALUE'].sum() * (1 / (len(df) / fps)))

Finally, calculate the mean velocity in that bin for that animal and body-part and append it to some list

results.append({"VIDEO": video_name, 'TIME BIN #': bin, 'ANIMAL': name, "BODY-PART": bps[0][:-2], "MEASUREMENT": "Velocity (cm/s)", 'VALUE': round(np.mean(bin_velocities), 4)})

I can push an update

[rfkova]

That seems great! Thanks, python is not my native language haha, so it's a slight struggle, especially testing things in actual workspaces :/

Honestly, though, I really enjoy learning coding languages, I just haven't given python the time it needs to become easily usable, so I appreciate the opportunity to look at your code and see your reasoning!!!

[sronilsson]

Yeah this reasoning is not much to write home about, it will work but these kind of nested for loops, avoid them, don't do it! :)

Let me know how ot looks like on your end in the latest version with pip install simba-uw-tf-dev --upgrade its 1.88.8