Error during train machine model step/need help understanding hyperparameters

[fon215]

Hello! I completed the labeling of videos in my project and I am trying to do the Train Machine Model step in the pipeline but I am having trouble understanding how it works. I read the documentation and followed the guide, using the generic hyperparameter settings listed in this document BtWGaNP_meta.csv that is provided in the guide but I am getting the error listed below. Could you help me understand this step and the hyperparameters so I can complete this successfully? I am new to SimBA and I just want to make sure I am doing everything correctly! For context, my videos have one mouse in each video where I am using 6 body parts (nose, left forepaw, right forepaw, left hind paw, right hind paw, and tail base) to analyze left and right nape scratching. Thanks!

[Parallel(n_jobs=-1)]: Using backend ThreadingBackend with 20 concurrent workers. [Parallel(n_jobs=-1)]: Done 10 tasks | elapsed: 0.2s [Parallel(n_jobs=-1)]: Done 160 tasks | elapsed: 1.8s [Parallel(n_jobs=-1)]: Done 410 tasks | elapsed: 4.7s [Parallel(n_jobs=-1)]: Done 760 tasks | elapsed: 8.5s [Parallel(n_jobs=-1)]: Done 1210 tasks | elapsed: 13.6s [Parallel(n_jobs=-1)]: Done 1760 tasks | elapsed: 19.8s [Parallel(n_jobs=-1)]: Done 2000 out of 2000 | elapsed: 22.5s finished Exception in Tkinter callback Traceback (most recent call last): File "c:\tools\anaconda3\envs\simbaenv\lib\tkinter\__init__.py", line 1705, in __call__ return self.func(*args) File "c:\tools\anaconda3\envs\simbaenv\lib\site-packages\simba\SimBA.py", line 4088, in <lambda> button_trainmachinemodel = Button(label_trainmachinemodel,text='Train single model from global environment',fg='blue',command = lambda: threading.Thread(target=self.train_single_model(config_path=self.projectconfigini)).start()) File "c:\tools\anaconda3\envs\simbaenv\lib\site-packages\simba\SimBA.py", line 4539, in train_single_model model_trainer.train_model() File "c:\tools\anaconda3\envs\simbaenv\lib\site-packages\simba\train_single_model.py", line 132, in train_model create_x_importance_bar_chart(self.rf_clf, self.feature_names, self.clf_name, self.eval_out_path, feature_importance_bars) File "c:\tools\anaconda3\envs\simbaenv\lib\site-packages\simba\train_model_functions.py", line 217, in create_x_importance_bar_chart create_x_importance_log(rf_clf, x_names, clf_name, save_dir) File "c:\tools\anaconda3\envs\simbaenv\lib\site-packages\simba\train_model_functions.py", line 208, in create_x_importance_log df.to_csv(save_file_path, index=False) File "c:\tools\anaconda3\envs\simbaenv\lib\site-packages\pandas\core\generic.py", line 3228, in to_csv formatter.save() File "c:\tools\anaconda3\envs\simbaenv\lib\site-packages\pandas\io\formats\csvs.py", line 183, in save compression=self.compression, File "c:\tools\anaconda3\envs\simbaenv\lib\site-packages\pandas\io\common.py", line 399, in _get_handle f = open(path_or_buf, mode, encoding=encoding, newline="") FileNotFoundError: [Errno 2] No such file or directory: 'N:/Ross/1Tayler Sheahan/Automated Behavioral Analysis/Madeline Fontana/SimBA\\Full_SimBA\\models\\generated_models\\model_evaluations\\leftnapescratch_feature_importance_log.csv'

I was able to exit still and this error was no longer in the command windows so I should be able to do that step successfully again once I fix the issue.

[sronilsson]

Hi @fon215 ! Thanks for reporting, and, my bad. I was just updating these scripts, and I forgot the line of code that creates the folder, before saving the file in that folder :) Can you update to simba 0.93.6 with pip install simba-uw-tf-dev --upgrade and see if that fixes it? If it doesn't, or any other errors come up, please let me know.

[fon215]

That's alright! I upgraded my version of SimBA but when I use the generic settings above I am still getting the same error. Along with fixing that issue, I was wondering if I will need to create 2 models and train on two different specific models because I am checking for both left and right nape scratch, or is that not how this works?

[sronilsson]

If I understand correctly, "nape scratch" is a behavior you want to classify, and you want to know how much left-side nape scratching the animal is doing, and how much right-side nape scratching the animal is doing? If so then yes, in SimBA we need two classifiers.

[sronilsson]

Oh I noticed you still see the error.. I will have to check again why that is.

can you confirm you are running 0.93.6 by typing pip show simba-uw-tf-dev?

[fon215]

Okay! Yes, I have two behaviors/classifiers because I am looking for when a mouse scratches the nape of their neck on both sides. So do I save settings for a specific model for each and train on a specific model for each as well, selecting each behavior for the settings in this step and saving a model for each?

Also, this is the version listed when I use the command you provided.

However, even after using the pip install simba-uw-tf-dev --upgrade command again this is the message in the command window so I'm not sure what is happening.

Requirement already satisfied: zipp>=0.5 in c:\tools\anaconda3\envs\simbaenv\lib\site-packages (from importlib-metadata->click>=7.1.2->Flask>=1.0.2->dash==1.14.0->simba-uw-tf-dev) (3.6.0)

[sronilsson]

Yes, I think the model settings will be very similar, if the scratching is very similar, just mirrored. But you will need annotations for both behaviors.

That is just a message, no error or warning.

That said, I can't recreate your error. Would you mind running the code again and send me a screengrab of the error msg in the windows terminal?

[fon215]

Okay so this time, I am no longer getting the error because I made two specific models, one for each side, saved those metadata setting files, and trained on multiple models and that seems to have worked and successfully and completed. Let me know if something still does not look right and thanks for your help!

[sronilsson]

Thanks for testing again, and looks good so far! However the # 25 features looks a little low considering the number of body-parts that you are tracking, SimBA should calculate the distance between all of the body-parts, their movements etc.. If you can share a csv file located in the project_folder/csv/features _extracted, zip it up and drag it into this thread if not too big, I can take a look. If too big, see if you can remove some rows (I'm only really intrested in the top rows)

[fon215]

Here is one of the csv files and what the features extracted folder looks like for me. Bottom up comp 48-80 050721 mouse 2 part 2_Trim7.csv.zip

[sronilsson]

False alarm, I think you should be good :)

[fon215]

Okay, thanks so much!