Validate Model on Single Video selection is bugged in current version

[rfkova]

Describe the bug When I click Browse File next to Select Model File on the [Run Machine Model] page I get a weird error instead of loading the file.

I rolled back to simba-uw-tf-dev==1.73.9 and selection works fine and the issue is resolved.

Error Exception in Tkinter callback Traceback (most recent call last): File "C:\Users\RFK\anaconda3\envs\simba\lib\tkinter__init.py", line 1705, in call__ return self.func(*args) File "C:\Users\RFK\anaconda3\envs\simba\lib\site-packages\simba\ui\tkinter_functions.py", line 130, in setFilePath file_selected = askopenfilename(title=self.title, parent=self.parent, filetypes=self.file_type) File "C:\Users\RFK\anaconda3\envs\simba\lib\tkinter\filedialog.py", line 375, in askopenfilename return Open(**options).show() File "C:\Users\RFK\anaconda3\envs\simba\lib\tkinter\commondialog.py", line 35, in show self._fixoptions() File "C:\Users\RFK\anaconda3\envs\simba\lib\tkinter\filedialog.py", line 302, in _fixoptions self.options["filetypes"] = tuple(self.options["filetypes"]) TypeError: 'NoneType' object is not iterable

Desktop (please complete the following information):

• OS: Win 10, 64-bit
• Python Version 3.6.13
• Are you using anaconda? Yes

Thank you! Ryan

[rfkova]

Hi @sronilsson!

I took the time to do some frame by frame validation today (manually measuring/calculating pixels -> distance -> velocity) and everything looks like it is working as intended!!! Thanks again so for your help! I'll look more into the symlink problem in the future and let you know if there is anything useful I learn :)

[sronilsson]

Nice one, thank you - I appreciate that check!! Yes, please let me know how goes. Regardless I will try to borrow a remote computer with a server come the weekend and see if I can recreate any of the issues when using symlinks.

[rfkova]

@sronilsson, wait... does this take frame 0-3 instead of 0-2 as intended with bin_length_frames = 3 (for bin 0)? results_df_lists = [movement_data[i : i + bin_length_frames] for i in range(0, movement_data.shape[0], bin_length_frames)]

I'm noticing some mismatches between the computed velocities and the videos themselves of a few frames with large movements (artifacts). I'd like to be able to match frames to bins to verify things and sync up some neural data with this behavior data. I can't get a match even shifting things around... I think the way this should work is bin 0 has frames 0-2, bin 1 has frames 3-5... so to get the center frame of the bin I could take the bin values and multiply by 3 and add 1 (bin#*3+1).

To be clear, when I varified I assumed small variations were due to my own manual labeling, but it could be it was also wrapping in adjacent (and similar) frames I wasn't aware of.

[rfkova]

@sronilsson

belay that... reasoning things out... it's this way so we can get differences between frames... we need 4 points for 3 differences... let me work on this some more trying to match up frames to data... sorry!

[sronilsson]

Yeah these things always do my head in in python - the off-by-one errors. I just did a quick double check and it seems to be OK? Note that in python the last index in each bin is not inclusive while the first one is inclusive, so the first bin would be frames 0->4 while the second would be values 5->9.

import numpy as np
one_bin_size_in_frames = 5
fake_movement_data_for_50_frames = np.random.randint(0, 100, (50,))
for bin_start_frm in range(0, fake_movement_data_for_50_frames.shape[0], one_bin_size_in_frames):
    bin_end_frm = bin_start_frm + one_bin_size_in_frames
    print(bin_start_frm, bin_end_frm)
0 5
5 10
10 15
15 20
20 25
25 30
30 35
35 40
40 45
45 50

[rfkova]

I agree, the code looks like it's doing the right thing... but that revelation hasn't completely solved my issue. I'm just doing a little more validation, specifically making sure I can align time in neural recordings to velocities.

Basically I have 7 frames with lots of movement flanked by practically no movement and I can't figure out why the velocity bins aren't capturing that in a way that lines up with the frames in your video annotation of that body part. Possibly it's something I'm doing wrong lining things up based on my understanding of the code, but I'm tracking what you are saying above just fine so maybe it's something less obvious than a blunt misunderstanding.

I'll attach an excerpt of the velocity output with some additional columns of my estimated start and end frames for the bin and a description of which frame differences contain large movements... maybe you can help me understand what I'm seeing in the output. example frames.xlsx

Frame 37404-419 from this video with the smoothed data exported from the ROI GUI tab. As noted in the excel doc frames 37407-37413 have a ton of movement (artifacts) that I want to align to bins but right now it doesn't line up.

[sronilsson]

Hang on, I will read this properly later!

[sronilsson]

@rfkova Alright lets see if I understand this, looking at the excel sheet, I see a lot of movement and high velocity happening in bin 12471 and 12472, what you are saying, is that these bursts of movement, is actually happening in bins 12469 and 12470 and that the bin counts have been shifted by 1?

[rfkova]

Close with some misalignment of bins - frame 37407-37408 should be the first big movement of the bp tracked which should fall in bin 12470 (at the start of this bin). There are consecutive large movements of the bp every frame until after 37413 (end of bin 12471). But the velocity output continues to log large movements in 12472. I feel like it's under-counting movement in 12470 and it's ending up in 12472 somehow...

The surrounding movements I initially thought were smaller than what is shown in the spreadsheet, but honestly, they are in the ballpark... the odd one is bin 12472 which should be similar to the flanking bins (12469 and 12473).

[rfkova]

Sorry, the spreadsheet was hurting my eyes... to simplify - It appears shifted by a bin with large movements in this example appearing one bin too late, but going over the code, that doesn't make sense to me.

In other related news, is it normal for the videos that go through the VISUALIZE ROI TRACKING to be shorter than the original by a few frames? This particular video I sent is 7 frames shorter than the original video - is it dues to smoothing (Savitzky Golay 100ms) and will it likely affect time estimates like I'm doing now (could this be because I'm looking at a very slightly shorter video than the original)?

^I am using the VISUALIZE ROI TRACKING video to compare to the spreadsheets which is how I'm determining which frame the movements are on since it is plotting body part location...

[sronilsson]

One more question @rfkova just so I get this right - is it only shifted when the movements are very large? Or is it shifted throughout - or is it just noticeable when the movements are large? I am going to test it using some tes data where I know the results in a bit.

And yes, someone pointed this out a month ish back and I thought I had fixed it - is it possible that you ran the visualization using 7 cores? It used to be that each core missed a frame..

[sronilsson]

Ah just saw the first message sorry!

[rfkova]

Hah! Yes, I ran with 7 cores!!!

It is just noticeable when they are large. Probably all the time, but I wouldn't know. I picked this example because the artifacts make it so stark.

[sronilsson]

Thanks @rfkova - I appreciate this, please keep issues like this coming! The code has sprawled like crazy and it is difficult to keep on-top off on my own. I will fixed the missing frames issue first, and then look into this shifted bins business.

[sronilsson]

Yes I think there is an while current_frm < end_frm: that should be while current_frm <= end_frm: right here

[sronilsson]

One more question, what FPS of the video do you have, and what is the bin length? I can't see the issue as yet, but I will try to reproduce with whatever values you have.

I'm thinking there could be some rounding errors tht have large effect when bins are so small:

The bin length in frames is calculated as bin_length_frames = int(fps * self.bin_length) and it will be rounded to floor by default.

[rfkova]

Rad!!!

If the processor bug was dropping a few frames, couldn't that account for the offset I was seeing depending on where they were dropped? It would only need to be off by a few frames to shift a bin since (in answer to your question) I'm using 30fps and 0.1s bins (3 frames).

Edit: @sronilsson I was reading frame #s off the video that got shortened, so that's what I mean by "account for"... I don't have a better way to visualize the bp locations than the video output which was dropping frames. lmk if you think the video annotation is working and I can check the current version of the binning script against an accurate video representation.

[sronilsson]

I should have fixed it in the latest version, I pushed yesterday, but please let me know!

[rfkova]

Confirmed! It IS fixed, and now the data line up perfectly :D

Thank you so much!

[sronilsson]

Thanks @rfkova !