Path plots artefeacts

[DorianBattivelli]

Hello,

As shown bellow, I'm having an issue when I generate a path-plot (from H5 multi animal project):

Is there a way to remove the 0, 0 frames with SimBA? In case I'd like to try to troubleshot this on my own, which CSV data file should I use in the Log folder? Outlier location one?

FYI: I did not apply interpolation, but I applied outlier correction (movement = 1, and movement = 2) I applied smoother correction (Savitzky Golay = 400)

Thank you !

[sronilsson]

Hi @DorianBattivelli -

Yes the path plotter grabs the data from the project_folder/csv/outlier_corrected_movement_location directory.

Yes - you can remove missing data (0,0) frames using interpolation - choose "Body-part: Some method" for this case. You can either (i) interpolate at import, or (ii) interpolate after import using menu below and selecting the `outlier_corrected_movement_location" directory.

FYI: There are some notebooks etc to create path plots and other stuff outside of GUI now, should you prefer: https://simba-uw-tf-dev.readthedocs.io/en/latest/nb/create_path_plot.html

[DorianBattivelli]

@sronilsson thank you for the help, unfortunately it did not solve the problem, I obtained exactly the same plot

[sronilsson]

Odd! Do you still see (0,0) in the data after running the interpolation and in the data you are plotting?

[DorianBattivelli]

Yes I do

[sronilsson]

Let me check if I can recreate, did you use the interpolation method in the screengrab avove?

[DorianBattivelli]

Yes

[sronilsson]

Ive tested a few projects now, and I can't recreate the missing interpolation... :/ Any chance you can share your project? Just the (i) project_folder/logs/measures/pose_configs/bp_names/project_bp_names.csv file, (ii) the project_config.ini, and (iii) and the file you are trying to interpolate?

[DorianBattivelli]

Shared.zip

[sronilsson]

Intresting... the interpolation seems to run fine on my end on your data. Which version of SimBA do you have pip show simba-uw-tf-dev ?

[DorianBattivelli]

1.59.3

[sronilsson]

If you upgrade pip install simba-uw-tf-dev --upgrade to version 1.59.7 is how does it run on your end?

[DorianBattivelli]

Do I need to re run the outlier correction after the interpolation or can I directly jump to path plots?

[sronilsson]

You should be able to go straight to path plots.

[sronilsson]

If you run outlier correction again your interpolated data might be overwritten again with the original data

[DorianBattivelli]

:/ Still the same error

[DorianBattivelli]

I mean it seems that the interpolation applied, but the plot still displays artefacts

[sronilsson]

Which path plot function are you using? Do you have a screen grab of the button / menu you are using?

[DorianBattivelli]

[sronilsson]

Ah I think I can see what is going on, there are some negative values in there for some reason, because smoothing has been run without interpolation I suspect. Let me insert a fix for that, but will be an hour or so before I am back at my computer.

[DorianBattivelli]

Wo let's hope it will fix the problem! Thank you

Il giorno gio 18 mag 2023 alle ore 23:41 Simon Nilsson < @.***> ha scritto:

Ah I think I can see what is going on, there are some negative values in there for some reason, because smoothing has been run without interpolation I suspect. Let me insert a fix for that, but will be an hour or so before I am back at my computer.

— Reply to this email directly, view it on GitHub https://github.com/sgoldenlab/simba/issues/261#issuecomment-1553689122, or unsubscribe https://github.com/notifications/unsubscribe-auth/AP6JZZFWMH4BDIMF5Z5H223XG2JQPANCNFSM6AAAAAAYGSO7OU . You are receiving this because you were mentioned.Message ID: @.***>

[sronilsson]

@DorianBattivelli - when you get a chance, try upgrading to version 1.58.9 and let me know how the path plot looks after interpolation!

[DorianBattivelli]

I only have access to 1.58.8, is that ok?

[sronilsson]

Ah sorry, my typo - https://pypi.org/project/Simba-UW-tf-dev/1.59.8/ - 1.59.8

[DorianBattivelli]

unfortunatly, it is still happening..

[DorianBattivelli]

Yet, it seems that the artefact lines are fewer than before, but maybe there is a solution to clean them better?

[sronilsson]

If you import the video again, and do interpolation + smoothing, how does it look like then?

[DorianBattivelli]

Still the same. For precision, I created a project from scratch, applied interpolation and smoothing, outlier correction, and generate path plot:

[sronilsson]

So any of the body-part coordinate values in the file point to (0,0) or coordinates below 0?

[DorianBattivelli]

Yes, to 0, but I did not see negative values. L column in the "center" body part, that I choose for plotting

[sronilsson]

Ah so it doesn't seem to have interpolated correctly, just to confirm, this screen grab above, comes from a file you have run interpolation on?

[DorianBattivelli]

yes

[sronilsson]

Gosh I don't know at the moment, I can't recreate it - I don't see any 0,0 in HybCD1-B2-D6-Urine.csv after running interpolation.. can you zip up and share the video file for this data as well?

[DorianBattivelli]

The video is 7gb.. maybe on drive?

[sronilsson]

Oh, maybe just tell me the resolution of the video?

[DorianBattivelli]

Resolution is 2020x2034, 5fps, 1 hour long

[sronilsson]

I don't know how arena looks like, but appears I do see some "artifacts" similar to you... I've got an idea though. Can you send me the project_csv/input_csv version of the file also? And tell me what the pixels per millimeter is for this video?

[DorianBattivelli]

HybCD1-B2-D6-Urine.csv

Pixel / mm: 1,54947

[sronilsson]

Last question, have you got a picture of arena?

[DorianBattivelli]

Is that enough a screenshot or should I propeerly extract a frame from the video?

[sronilsson]

That's good enough

[DorianBattivelli]

For precision, mice can only access the diagonal including upper left and lower right quadrants

[sronilsson]

Is this more what you expect?

[DorianBattivelli]

Absolutely! What was wrong?

[sronilsson]

haha yeah.... not 100% sure, have to back-trace my steps, weird interaction of events with negative values produced by smoothing and the outlier corrector freaking out.

[sronilsson]

I will push an update, but won't be immidiate, want to make sure I'm not missing anything.

[sronilsson]

@DorianBattivelli - if you update to version 1.59.9, and run outlier correction again, how does it look?

[DorianBattivelli]

I run one video, and I reproduced the last graph you sent without the artefact lines :D
I'll check other videos tomorrow, but I think you solved the probleem! Thank you so much, I keep you updated

[sronilsson]

👍🏻 👍🏻 👍🏻 nice, let me know how it goes !

[sronilsson]

fyi if you want to speed things up significantly, manually copy your files from the project_folder/csv/outlier_corrected_movement_location folder to the project_folder/csv/machine_results folder and use this menu, it allows you to create path plots using all of your computers cores or last frm only.

[DorianBattivelli]

I tested anoher video and it looks good too :) Thank you again!