Error when creating Validation video [out#0/mp4 @ 00000294EB792500]

[beepboop174]

Hi, can anyone please help me find out how to resolve this issue: When I press create validation video it creates this error message in the terminal:

[out#0/mp4 @ 00000294EB792500] Output file does not contain any stream Error opening output file C:/Users/-/Desktop/Test/SimBA\test1\project_folder\frames\output\validation\Trial5downsampled.mp4. Error opening output files: Invalid argument

Weirdly, the CPU fans are still running hard after this error comes, until I close simba.

ffmpeg is installed and added to PATH.

It creates a folder in output > validation > temp with a few mp4 files that cant be opened and a log saying: file 'C:\Users-\Desktop\Test\SimBA\test1\project_folder\frames\output\validation\temp\0.mp4' file 'C:\Users-\Desktop\Test\SimBA\test1\project_folder\frames\output\validation\temp\1.mp4' ... and so on.

Thanks a lot for any help!

[sronilsson]

Hi @beepboop174! - which version of Simba are you running - pip show simba-uw-tf-dev?

[beepboop174]

Hi @sronilsson thanks for the quick response! I use 1.87.5.

[sronilsson]

Thanks @beepboop174 - can you check in the latest version of simba with pip install simba-uw-tf-dev --upgrade if you're still seeing the same issue? If you do I can dig to try and replicate the issue.

[beepboop174]

@sronilsson First of all, this makes sense, right?

So. I updated it and tried it same settings as always (multicore processing 10 cores) and still the same issue: Btw im running this on an Intel i9, 10 cores. And for some reason after the error, the Simba GUI is unresponsive, CPU usage is at like 10-20% and the fans go crazy, so it does seem like its doing something, not sure what though.

[Parallel(n_jobs=20)]: Using backend ThreadingBackend with 20 concurrent workers. [Parallel(n_jobs=20)]: Done 10 tasks | elapsed: 0.0s [Parallel(n_jobs=20)]: Done 160 tasks | elapsed: 0.0s [Parallel(n_jobs=20)]: Done 410 tasks | elapsed: 0.0s [Parallel(n_jobs=20)]: Done 760 tasks | elapsed: 0.2s [Parallel(n_jobs=20)]: Done 1210 tasks | elapsed: 0.3s [Parallel(n_jobs=20)]: Done 1760 tasks | elapsed: 0.5s [Parallel(n_jobs=20)]: Done 2000 out of 2000 | elapsed: 0.5s finished Trailing option(s) found in the command: may be ignored. Input #0, concat, from 'C:\Users\taylorma\Desktop\Test\SimBA\test1\project_folder\frames\output\validation\temp\files.txt': Duration: N/A, bitrate: N/A Output #0, mp4, to 'C:/Users/taylorma/Desktop/Test/SimBA\test1\project_folder\frames\output\validation\Trial1downsampled.mp4': [out#0/mp4 @ 000002B0A70EE0C0] Output file does not contain any stream Error opening output file C:/Users/taylorma/Desktop/Test/SimBA\test1\project_folder\frames\output\validation\Trial1downsampled.mp4. Error opening output files: Invalid argument

Now, the weird thing is that its very different when I disable multicore processing. It also doesnt work, but for a different reason. It loads super fast and I get this message:

[Parallel(n_jobs=20)]: Using backend ThreadingBackend with 20 concurrent workers. [Parallel(n_jobs=20)]: Done 10 tasks | elapsed: 0.0s [Parallel(n_jobs=20)]: Done 160 tasks | elapsed: 0.0s [Parallel(n_jobs=20)]: Done 410 tasks | elapsed: 0.0s [Parallel(n_jobs=20)]: Done 760 tasks | elapsed: 0.2s [Parallel(n_jobs=20)]: Done 1210 tasks | elapsed: 0.3s [Parallel(n_jobs=20)]: Done 1760 tasks | elapsed: 0.5s [Parallel(n_jobs=20)]: Done 2000 out of 2000 | elapsed: 0.5s finished Exception in Tkinter callback Traceback (most recent call last): File "C:\Users\taylorma\AppData\Local\anaconda3\envs\SimBA\lib\tkinter__init.py", line 1705, in call return self.func(*args) File "C:\Users\taylorma\AppData\Local\anaconda3\envs\SimBA\lib\site-packages\simba\mixins\pop_up_mixin.py", line 308, in self.run_frm, text=title, fg="blue", command=lambda: run_function() File "C:\Users\taylorma\AppData\Local\anaconda3\envs\SimBA\lib\site-packages\simba\ui\pop_ups\validation_plot_pop_up.py", line 169, in run validation_video_creator.run() File "C:\Users\taylorma\AppData\Local\anaconda3\envs\SimBA\lib\site-packages\simba\plotting\single_run_model_validation_video.py", line 320, in run self.create_video() File "C:\Users\taylorma\AppData\Local\anaconda3\envs\SimBA\lib\site-packages\simba\plotting\single_run_model_validation_video.py", line 204, in create_video self.in_df.index[frm_cnt], [x_header, y_header] File "C:\Users\taylorma\AppData\Local\anaconda3\envs\SimBA\lib\site-packages\pandas\core\indexing.py", line 1418, in getitem return self._getitem_tuple(key) File "C:\Users\taylorma\AppData\Local\anaconda3\envs\SimBA\lib\site-packages\pandas\core\indexing.py", line 805, in _getitem_tuple return self._getitem_lowerdim(tup) File "C:\Users\taylorma\AppData\Local\anaconda3\envs\SimBA\lib\site-packages\pandas\core\indexing.py", line 961, in _getitem_lowerdim return getattr(section, self.name)[new_key] File "C:\Users\taylorma\AppData\Local\anaconda3\envs\SimBA\lib\site-packages\pandas\core\indexing.py", line 1424, in getitem__ return self._getitem_axis(maybe_callable, axis=axis) File "C:\Users\taylorma\AppData\Local\anaconda3\envs\SimBA\lib\site-packages\pandas\core\indexing.py", line 1839, in _getitem_axis return self._getitem_iterable(key, axis=axis) File "C:\Users\taylorma\AppData\Local\anaconda3\envs\SimBA\lib\site-packages\pandas\core\indexing.py", line 1133, in _getitem_iterable keyarr, indexer = self._get_listlike_indexer(key, axis, raise_missing=False) File "C:\Users\taylorma\AppData\Local\anaconda3\envs\SimBA\lib\site-packages\pandas\core\indexing.py", line 1092, in _get_listlike_indexer keyarr, indexer, o._get_axis_number(axis), raise_missing=raise_missing File "C:\Users\taylorma\AppData\Local\anaconda3\envs\SimBA\lib\site-packages\pandas\core\indexing.py", line 1177, in _validate_read_indexer key=key, axis=self.obj._get_axis_name(axis) KeyError: "None of [Index(['Animal 1_Ear_left_1_x', 'Animal 1_Ear_left_1_y'], dtype='object')] are in the [index]"

The CPU fans are silent and the SImba GUI is responsive. I get a mp4 file in the frames>output>validation folder, but its 5,54kb and doesnt work.

By the way when I do a validation plot everything works fine and seems good:

So yeah, seems like two differnt issues, one with multicore processing and one without.

[sronilsson]

Thanks @beepboop174 - yes, two different errors and I will look into and catch the errors, but just in brief:

The first error typically happens when we say we want to create say a 100x100 pixel video, but then we start to stack 101x105 frames into that video. OpenCV and FFMpeg doesn't complain, the video appears to be created, but if you look closely, the videos are all 0-1kb. Next, we want to join all those small bits of video that has been created on each core in parallel into a single video. Thats when the error pops up - can't concatenate videos that don't have any frames!

[sronilsson]

Hi @beepboop174 thanks again for reporting this.

Unfortunately I could not recreate your issues, but I went over the code..

I inserted some checks to ensure that any misalignment in the video and frame sizes are corrected for, and control for events when there are different number of frames in the pose-estimation data and the videos, etc and checks for all the arguments going into the functions to ensure they are valid, for my one and two animal test projects.

Could you update simba again using pip install simba-uw-tf-dev --upgrade, and let me know how it looks when you run it in your end? If not, could you send me the traceback errors you see again?

Simon

[beepboop174]

Hi @sronilsson thanks again, Just tried it and the error message I got isnt there anymore, but its still not working :(

Without multicore processing: (SimBA) C:\Users\taylorma>simba [Parallel(n_jobs=20)]: Using backend ThreadingBackend with 20 concurrent workers. [Parallel(n_jobs=20)]: Done 10 tasks | elapsed: 0.0s [Parallel(n_jobs=20)]: Done 160 tasks | elapsed: 0.0s [Parallel(n_jobs=20)]: Done 410 tasks | elapsed: 0.1s [Parallel(n_jobs=20)]: Done 760 tasks | elapsed: 0.2s [Parallel(n_jobs=20)]: Done 1210 tasks | elapsed: 0.3s [Parallel(n_jobs=20)]: Done 1760 tasks | elapsed: 0.5s [Parallel(n_jobs=20)]: Done 2000 out of 2000 | elapsed: 0.6s finished Exception in Tkinter callback Traceback (most recent call last): File "C:\Users\taylorma\AppData\Local\anaconda3\envs\SimBA\lib\tkinter__init.py", line 1705, in call return self.func(*args) File "C:\Users\taylorma\AppData\Local\anaconda3\envs\SimBA\lib\site-packages\simba\mixins\pop_up_mixin.py", line 308, in self.run_frm, text=title, fg="blue", command=lambda: run_function() File "C:\Users\taylorma\AppData\Local\anaconda3\envs\SimBA\lib\site-packages\simba\ui\pop_ups\validation_plot_pop_up.py", line 178, in run threading.Thread(target=validation_video_creator.run()).start() File "C:\Users\taylorma\AppData\Local\anaconda3\envs\SimBA\lib\site-packages\simba\plotting\single_run_model_validation_video.py", line 227, in run self.data_df.index[frm_cnt], [x_header, y_header] File "C:\Users\taylorma\AppData\Local\anaconda3\envs\SimBA\lib\site-packages\pandas\core\indexing.py", line 1418, in getitem__ return self._getitem_tuple(key) File "C:\Users\taylorma\AppData\Local\anaconda3\envs\SimBA\lib\site-packages\pandas\core\indexing.py", line 805, in _getitem_tuple return self._getitem_lowerdim(tup) File "C:\Users\taylorma\AppData\Local\anaconda3\envs\SimBA\lib\site-packages\pandas\core\indexing.py", line 961, in _getitem_lowerdim return getattr(section, self.name)[new_key] File "C:\Users\taylorma\AppData\Local\anaconda3\envs\SimBA\lib\site-packages\pandas\core\indexing.py", line 1424, in getitem return self._getitem_axis(maybe_callable, axis=axis) File "C:\Users\taylorma\AppData\Local\anaconda3\envs\SimBA\lib\site-packages\pandas\core\indexing.py", line 1839, in _getitem_axis return self._getitem_iterable(key, axis=axis) File "C:\Users\taylorma\AppData\Local\anaconda3\envs\SimBA\lib\site-packages\pandas\core\indexing.py", line 1133, in _getitem_iterable keyarr, indexer = self._get_listlike_indexer(key, axis, raise_missing=False) File "C:\Users\taylorma\AppData\Local\anaconda3\envs\SimBA\lib\site-packages\pandas\core\indexing.py", line 1092, in _get_listlike_indexer keyarr, indexer, o._get_axis_number(axis), raise_missing=raise_missing File "C:\Users\taylorma\AppData\Local\anaconda3\envs\SimBA\lib\site-packages\pandas\core\indexing.py", line 1177, in _validate_read_indexer key=key, axis=self.obj._get_axis_name(axis) KeyError: "None of [Index(['Animal 1_left_ear_x', 'Animal 1_left_ear_y'], dtype='object')] are in the [index]"

I think the problem may be that in the features_extracted files etc., the bodypart names are represented with different names, like "center", while in my DLC H5 pose file its called "back", for example. But changing the project_bp_names file didnt help. Then I tried making an entirely new proejct but that also didnt help, same error (also shows up when training the model). Then I delted all files like extracted frames or movementoutliers, changed the bp_name log and re ran them, but still the same error. And still the names like"center" in these files, even though my names are in the bp log.

So do you think this is the issue and how might I solve it? Create a new project again and first thing I do is change the bp log file?

Thanks for any help!

[sronilsson]

Hi @beepboop174! Yes nearly there now.. :) Fastest way to solve it, if possible, is to share a little bit of the project, enough for me to replicate the error, so I can see how this has happened, and fix it, and I can make sure no-one else bumps into it? Perhaps on a gdrive or similar?

Exactly as you say, SimBA loops through the body-parts it expects in the project to place a little circle on each frame and each body-part, the first body-part it expects is the left ear for Animal 1. But in year case, it's not there, and it fails. If you are using one of the standard built-in default body-part configurations, SimBa should make sure that you have a body-part called left ear, so not sure how this would come about.

One possible way that comes to me as I am typing: when you select a features file to run the validation on, are you selecting a file in the project_folder/csv/features_extracted directory or are you selecting a different file?

Simon

[beepboop174]

Hey @sronilsson thanks alot! Yes Here I uploaded my project files except for the videos via wetransfer: https://we.tl/t-aBodyFADfL

Yes, I am using the features_extracted csv files for the Validation. And yes I use the normal 2 animal 8 bodypart configuration, my boidyparts I labelled in maDLC are in the same order and lcoation as the SimBA config suggests. I only have different names, like "left_ear" instead of "Ear_left". Also btw why is there Ear_left_x, Ear_left_y and Ear_left_p in my files?

Thanks!

[sronilsson]

Thanks @beepboop174 - I will take a look. At first glance, I can't see any video files, would you mind sharing one for one of the data files so I can try to replicate the error in full?

I only have different names, like "left_ear" instead of "Ear_left". Also btw why is there Ear_left_x, Ear_left_y and Ear_left_p in my files

When you choose one of the default body-part configurations in SimBA, then SimBA will use hard-coded names for each one of your body-part, like "Ear_left" instead of your name "left_ear". The reason for this is to standardize the naming and make all the functions downstream know, that if you want to compute anything using the left ear, then look in the "Ear_left" column, regardless if users named them "my_LEFT_eAr" or "head_left_Side", it will be forced to "Ear_left".

[beepboop174]

@sronilsson Sure, heres the video folder: https://we.tl/t-YX4IxUyPC5 for some reason I cant upload an entire project folder but you should have all the pieces now.

Thnak you so much!

[sronilsson]

No problem @beepboop174 !

I can see what is happening, but not sure how this could happen - I will insert some more informative error messages at least

We might have chatted about it above but regardless..

i) When you create a SimBA project with 16 body-parts, the project contains a CSV at project_folder/logs/measures/pose_configs/bp_names/project_bp_names.csv that lists a set body-part names in the project.

ii) Your project data files indeed contained all these body-part names as listed in the expected CSV (that is produced when using a 16-body-part project). However, the expected CSV had changed, into other names. So there was a mismatch in the expected and actual column names, causing the issue.

To fix it, replace the project_folder/logs/measures/pose_configs/bp_names/project_bp_names.csv in your project with the below file (just unzip it first). And rerun your validation. Let me know how it goes please!

project_bp_names.csv.zip

NOTE:

I see in a single validation video I made, that your body-parts appear "stuck" and sometimes not move with the rest of the animal.

https://github.com/sgoldenlab/simba/assets/34761092/a3a8d06a-eeec-45c6-9be0-1bcb5d935ea0

This can be produced by using a too stringent outlier critera, see POINT 3 HERE.

To fix it, consider skipping heuristic outlier critera fixing, if your pose is good, or alternatively setting higher outlier criteria.

[beepboop174]

@sronilsson Thank you soo much, its finally mostly working! So far, the validation video works, and path plot, gantt plot, heatmap work! Yet for some reason running data plot gives me this:

Exception in Tkinter callback Traceback (most recent call last): File "C:\Users\taylorma\AppData\Local\anaconda3\envs\SimBA\lib\tkinter__init.py", line 1705, in call return self.func(*args) File "C:\Users\taylorma\AppData\Local\anaconda3\envs\SimBA\lib\site-packages\simba\ui\pop_ups\data_plot_pop_up.py", line 115, in command=lambda: self.create_data_plots(multiple_videos=False), File "C:\Users\taylorma\AppData\Local\anaconda3\envs\SimBA\lib\site-packages\simba\ui\pop_ups\data_plot_pop_up.py", line 221, in create_data_plots video_setting=self.data_videos_var.get(), File "C:\Users\taylorma\AppData\Local\anaconda3\envs\SimBA\lib\site-packages\simba\plotting\data_plotter.py", line 60, in init__ self.process_movement() File "C:\Users\taylorma\AppData\Local\anaconda3\envs\SimBA\lib\site-packages\simba\plotting\data_plotter.py", line 90, in process_movement movement_processor.run() File "C:\Users\taylorma\AppData\Local\anaconda3\envs\SimBA\lib\site-packages\simba\data_processors\movement_calculator.py", line 101, in run self.data_df = self.data_df[self.bp_list] File "C:\Users\taylorma\AppData\Local\anaconda3\envs\SimBA\lib\site-packages\pandas\core\frame.py", line 3001, in getitem indexer = self.loc._convert_to_indexer(key, axis=1, raise_missing=True) File "C:\Users\taylorma\AppData\Local\anaconda3\envs\SimBA\lib\site-packages\pandas\core\indexing.py", line 1285, in _convert_to_indexer return self._get_listlike_indexer(obj, axis, **kwargs)[1] File "C:\Users\taylorma\AppData\Local\anaconda3\envs\SimBA\lib\site-packages\pandas\core\indexing.py", line 1092, in _get_listlike_indexer keyarr, indexer, o._get_axis_number(axis), raise_missing=raise_missing File "C:\Users\taylorma\AppData\Local\anaconda3\envs\SimBA\lib\site-packages\pandas\core\indexing.py", line 1177, in _validate_read_indexer key=key, axis=self.obj._get_axis_name(axis) KeyError: "None of [Index(['Nose_1_x', 'Nose_1_y', 'Nose_1_p', 'Nose_2_x', 'Nose_2_y', 'Nose_2_p'], dtype='object')] are in the [columns]"

Similiarly, distance plot gives me this:

Exception in Tkinter callback Traceback (most recent call last): File "C:\Users\taylorma\AppData\Local\anaconda3\envs\SimBA\lib\site-packages\pandas\core\indexes\base.py", line 2897, in get_loc return self._engine.get_loc(key) File "pandas/_libs/index.pyx", line 107, in pandas._libs.index.IndexEngine.get_loc File "pandas/_libs/index.pyx", line 131, in pandas._libs.index.IndexEngine.get_loc File "pandas/_libs/hashtable_class_helper.pxi", line 1607, in pandas._libs.hashtable.PyObjectHashTable.get_item File "pandas/_libs/hashtable_class_helper.pxi", line 1614, in pandas._libs.hashtable.PyObjectHashTable.get_item KeyError: 'Nose_1_x'

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "C:\Users\taylorma\AppData\Local\anaconda3\envs\SimBA\lib\tkinter__init.py", line 1705, in call return self.func(*args) File "C:\Users\taylorma\AppData\Local\anaconda3\envs\SimBA\lib\site-packages\simba\ui\pop_ups\distance_plot_pop_up.py", line 143, in command=lambda: self.create_distance_plots(multiple_videos=False), File "C:\Users\taylorma\AppData\Local\anaconda3\envs\SimBA\lib\site-packages\simba\ui\pop_ups\distance_plot_pop_up.py", line 289, in __create_distance_plots distance_plotter.run() File "C:\Users\taylorma\AppData\Local\anaconda3\envs\SimBA\lib\site-packages\simba\plotting\distance_plotter_mp.py", line 124, in run (self.data_df[data[0] + "_x"] - self.data_df[data[1] + "_x"]) File "C:\Users\taylorma\AppData\Local\anaconda3\envs\SimBA\lib\site-packages\pandas\core\frame.py", line 2995, in getitem indexer = self.columns.get_loc(key) File "C:\Users\taylorma\AppData\Local\anaconda3\envs\SimBA\lib\site-packages\pandas\core\indexes\base.py", line 2899, in get_loc return self._engine.get_loc(self._maybe_cast_indexer(key)) File "pandas/_libs/index.pyx", line 107, in pandas._libs.index.IndexEngine.get_loc File "pandas/_libs/index.pyx", line 131, in pandas._libs.index.IndexEngine.get_loc File "pandas/_libs/hashtable_class_helper.pxi", line 1607, in pandas._libs.hashtable.PyObjectHashTable.get_item File "pandas/_libs/hashtable_class_helper.pxi", line 1614, in pandas._libs.hashtable.PyObjectHashTable.get_item KeyError: 'Nose_1_x'

And: "To fix it, consider skipping heuristic outlier critera fixing, if your pose is good, or alternatively setting higher outlier criteria." Thnaks, yea that seems to be happening. My pose-data isnt amazing, what outlier criterea settings would you suggest?

Edit:

Also, when trying to merge videos, theres no "run" button for me, any idea what that might be?

[sronilsson]

Let me check on the data plot and the run button!

[sronilsson]

The missing run button is a bug that has sneaked in, thanks @beepboop174!

[beepboop174]

@sronilsson Thanks for the effort, youre doing an amazing job with fixing this stuff! FYI, so today I reran everything with new outlier criteria extracted new frames labeld more data etc: All in the same project that you fixed last time. Turns out I guess that replaced the pose log file and I again ran into the same issue, and again had to replace the pose log with the file you uploaded. Hope that helps with finding the issue.

So, since I cant really create a Full video trough ther Visualization Tab due to the the data plot and video merge issues, Is there maybe a way to run the validation video with two classifiers at the same time? Since thats kinda what I need for a presentation soon.

Thanks and have a good day!

[sronilsson]

Hi @beepboop174 - let me take a look what is going on with the pose log as well. I kind of dug myself into a hole: I inserted the run button, but then I looked at the frame merging code and it is not well written, performs a lot of unnecessary slow operations and code duplications, so I want to fix that now.

To create visualizations with multiple classifiers, run your models on your files so you have data inside the project_folder/csv/machine_results directory. Next, go the the visualization tab and use this menu:

Let me know if that doesn't make sense or if issues!

[beepboop174]

@sronilsson thanks a lot! Visualize classifications was the only option I didnt try haha, didnt know that was the one. Works great, thanks for the hard work!

Plese give me a heads up once the run merge button works!

Have a good day!

[sronilsson]

Thanks for letting me know @beepboop174! The frame merging is fixed but I have to update the pip package so you can reach it. I started looking at the distance plotting first , and when I look at code written by myself some time ago, it looks a bit "smelly" - I see that the code leaves doors open for errors that not necessarily will happen.. but could happen. So will fix that too before I update. I will let you know.

[sronilsson]

Can you give it a go in latest version with pip install simba-uw-tf-dev —upgrade and let me know how it works? I

Likely this too much info but info below, ignore it, but at least for myself, if I have to revisit this in future :)

. For the frame merging, issue is we need to check and handle cases where users say they want to concatenate videos that are of different fps and different resolutions, and when they provide uneven number of videos. Because of these checks currently… it’s still not very quick, especially when using mosaic concatenations that involve both vertical and horizontal joins:

Just aheads up, I can see how it is useful to create showcase videos, but if it is taking too long, it might be worth it to just play the videos separately side by side… took me 5-10min to create a vertical join of your videos

To create distance plots, like line plots representing animal distances, it’s typical to use matplotlib. To plot distances between animals for each frame of the video though, we need to create several 1000s of plots and we should split the images and process them in parallel on each available core so we don’t have to wait forever. However, matplotlib doesn’t always play nice multiprocessing, especially on Mac computers, so I needed to dig a little for alternative.

[beepboop174]

Thanks! I get this:

Creating mixed mosaic video... Creating upper right mosaic ... (Step 1/4) Creating lower right mosaic ... (Step 2/4) Joining upper and lower right mosaic... (Step 3/4) Concatenating 2 videos vertically with a 720 pixel width... Vertical concatenation complete. Saved at C:/Users/taylorma/Desktop/Test/SimBA/Dos/dos/project_folder/frames/output/gantt_plots\temp_20240412133906\mosaic.mp4 (Elapsed time: 0.0629s.) Joining left and right mosaic... (Step 4/4) SIMBA VIDEO FILE ERROR: Video mosaic either does not exist or has fps of 0 (full error video path: C:/Users/taylorma/Desktop/Test/SimBA/Dos/dos/project_folder/frames/output/gantt_plots\temp_20240412133906\mosaic.mp4).

[sronilsson]

Hi @beepboop174 ! I tried it a few ways on your data and could not hit this error you are seeing, would you mind take a screenshot of the GUI window filled in so I see your choices just before clicking RUN?