Outlier Correction length mismatch

[blogeman]

I am using the development version of SIMBA (0.61) primarily with the GUI on Windows 10 with Python 3.7 through Anaconda.

My pose_config is very similar to that in #66 such that I have 2 animals with the same body parts and 1 animal with different body parts.

I have successfully imported a multi-animal DLC H5 file and have verified the presence of the .csv file located in the "input_csv" directory. Using the most recent version of DLC I have already corrected for outliers and wish to skip this step in SIMBA. After clicking "Skip outlier correction" I receive the following error:

Exception in Tkinter callback Traceback (most recent call last): File "c:\users\brand\anaconda3\envs\simba\lib\tkinter__init.py", line 1705, in call__ return self.func(*args) File "c:\users\brand\anaconda3\envs\simba\lib\site-packages\simba\SimBA.py", line 3513, in button_skipOC = Button(label_outliercorrection,text='Skip outlier correction (CAUTION)',fg='red', command=lambda:skip_outlier_c(self.projectconfigini)) File "c:\users\brand\anaconda3\envs\simba\lib\site-packages\simba\outlier_scripts\skip_outlierCorrection.py", line 52, in skip_outlier_c csv_df.columns = newHeaders File "c:\users\brand\anaconda3\envs\simba\lib\site-packages\pandas\core\generic.py", line 5192, in setattr return object.setattr(self, name, value) File "pandas/_libs/properties.pyx", line 67, in pandas._libs.properties.AxisProperty.set File "c:\users\brand\anaconda3\envs\simba\lib\site-packages\pandas\core\generic.py", line 690, in _set_axis self._data.set_axis(axis, labels) File "c:\users\brand\anaconda3\envs\simba\lib\site-packages\pandas\core\internals\managers.py", line 183, in set_axis "values have {new} elements".format(old=old_len, new=new_len) ValueError: Length mismatch: Expected axis has 57 elements, new values have 69 elements

Any insight would be appreciated, thanks.

[sgoldenlab]

Hi @blogeman - thanks for reporting - and sorry - I don't have your kind of datasets to troubleshoot with and can't replicate it so will have to ask ask questions to see if we can figure it out!

What did you name your individual body-parts in the SimBA user-defined pose configuration menu? SimBA needs to know which body-part belongs to which animal, and, currently, the process of getting that done is to add a number at the end of each body-part in the user-defined pose-configuration box:

Any body part ending with _1 belongs to the first animal, any body part ending with _2 belongs to the second animal, etc..

The error that you are seeing could be produced by a typo here, which could cause SimBA to expects more (or less) columns than actually exists.

[blogeman]

Hi @sgoldenlab thanks very much for your assistance, much appreciated. After some investigation, I think I am onto something. Per your recommendation, the individual body parts seem to have the correct spelling and possess the "_X" to indicate which animal the point belongs to in the pose-configuration menu. For what its worth, the body parts for animal 3 are exactly the same as for animal 2 but with the "_3" instead of the "_2".

...but when I load the project the opening GUI only displays the body parts for animals 1 and 2 while omitting the bodyparts from animal 3.

The mismatch error I'm getting is expecting 57 columns while the file has 69, a difference of 12 columns. Knowing that this data comes from DLC, which for each body part uses 3 columns (X, Y, and p-value), the error would be consistent with SIMBA not expecting animal_3 which has 4 body parts (a total of 12 columns). This corresponds to the opening GUI not displaying animal_3.......but I don't understand why its missing animal_3.

[sgoldenlab]

Thanks @blogeman - to figure out where it goes wrong:

1. Could you open the project_folder\logs\measures\pose_configs\bp_names csv file and paste a screenshot of the content? I think there might be something there, where the bodyparts of the last animals isn't saved properly at import.

2. Could you open the project_folder/project_config.ini, and tell me what it reads on line 73, under [create ensemble settings], after pose_estimation_body_parts =. It should read user_defined.

3. Also in the project_folder/project_config.ini, could you tell me what it reads after animal_no on line 8 under [General settings]?

Lastly, are you working with DeepLabCut data, or is it from SLEAP?

EDIT: Sorry, scrolled up as saw that it was DLC!

[blogeman]

Thanks for troubleshooting this with me @sgoldenlab. Below are my responses:

1. Here is the screenshot of the "bp_names.csv" file. All looks as it should to my eyes.

2. Line 73 of the project_config.ini file reads "pose_estimation_body_parts = user_defined"

3. Line 8 of the project_config.ini file reads "animal_no = 3"

When I start to play around with the "animal_no" variable in the config file things behave strangely. For starters, even though it says 3 the GUI is only displaying lists for 2 animals (see screenshot from last post). If I change the animal_no to 1, it display all body parts inside the first list and the second list reads "No body parts".

It seems as though the program is stuck creating 2 lists / animals. Does this make sense?

[sgoldenlab]

Thanks @blogeman - ah I think we are dealing with 2 separate issues, I'm onto it.

One more question.

Can you tell me what you can see on Line 122 in the project config? A screenshot would be best, like this:

[sgoldenlab]

I think I got it, but will come back to this a little later on today.

[blogeman]

Hmm, it looks like my id_list is empty...

[sgoldenlab]

Ah, that could be part of it. Let me make a small update first. Hang on.

[sgoldenlab]

@blogeman - something has gone wrong at the import but I think we can fix this error manually.

First, update to Simba-dev==0.63.

Next, in your id_list manually insert names for your three animals. E.g.,

id_list = Animal1,Animal2,Animal3

NOTE: do not use any spaces between your animal names.

Let me know if that helps or not.

[blogeman]

Updated to dev==0.63 and manually edited the config file id_list to contain the names of my 3 animals. After running the "Skip outlier correction" step I get the following error:

The original problem was that at this step SIMBA was only expected 57 columns (corresponding to 2 animals) but my dataset had 69 columns (3 animals). Now SIMBA is expecting 69 columns.....but the dataset contains 115. I can't figure out where the number 115 comes from.....

115 is divisible by 23 (the number of bodyparts in the dataset), but I can't think of what would be represented in the 5 columns per bodypart.

[sgoldenlab]

I see, that did not go as I have planned. I will look into this later today.

Just to confirm, your animals, they do have their respective body parts ending with _1 and _2 and _3 ?

It is the other way around - so your data frame contains 69 columns, and that's how many new column headers are expected. For some reason SimBA generates a list of 115 column names now in your scenario, and tries to use those for you 69 columns.

[blogeman]

Thanks @sgoldenlab. To confirm, all body parts have the "_X" identifier.

[sgoldenlab]

Thanks @blogeman: one final question for now. I'm trying to reproduce your project and error but I am not getting it. Any chance you could send me the input H5 or CSV from DLC? It would be extremely helpful. I am only really interested in the header structure so I am not missing anything. If you don't want to share the entire file just the headers lines and a few rows will do

[blogeman]

Please forgive for my ignorance, but how can I send the file to you? I've tried to drag and drop it into the comment box but it wont allow for an H5 file type. I've tried to google for the answer, but haven't gotten anywhere. I'm also new to Github and not sure what the proper etiquette for this kind of thing is, hope I'm not bothering you.

[sgoldenlab]

No no @blogeman, not at all! People's tracking parameters are becoming more complex, and I need a stable and flexible solution for file import. What limits me is that I don't have tracking files for your situation, so I somewhat shoot in the dark with my own mock-up hand-generated csv/h5 files, so this is helpful.

I'd say you can either try and paste a gdrive link here to the file. Or share it with sgoldenlab@gmail.com

[blogeman]

Hi @sgoldenlab, I e-mailed the CSV and the H5 files to the e-mail address you indicated. If there is anything else you need from me I will happily provide.

[sgoldenlab]

Hi @blogeman - sorry about this - can you try again sending it to goldenneurolab@gmail.com ?

I will try to look at it before end of week.

[adamz10]

Hello, i am having this same problem except my error message states "expected axis has 16 elements, new values have 24 elements." Did you all find a fix to this problem?

[blogeman]

Hi @adamz10. Unfortunately I never figured out how to solve this problem.

[adamz10]

Sorry to hear that @blogeman. I appreciate the update though!