Error: ReTraining on Different TrainingSet - size of output embedding mismatch

[humayun]

Dear TreB1eN,

Thank you for sharing the code. I am trying to retrain (fine tune) previously trained model on different training set. Number of classes (identities) is different in new training set. It gave me following error when I load the previously trained model:

How can I overcome resolve retraining on different training set?

self.head.load_state_dict(torch.load(save_path/'head_{}'.format(fixed_str)))
  File "/home/synthesis/anaconda3/envs/pytorch3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 839, in load_state_dict
    self.__class__.__name__, "\n\t".join(error_msgs)))
RuntimeError: Error(s) in loading state_dict for Arcface:
    size mismatch for kernel: copying a param with shape torch.Size([512, 25]) from checkpoint, the shape in current model is torch.Size([512, 41]).

[tadbeer]

Hey @humayun Ji, did you make any progress with training on your own dataset?

[humayun]

In case of different training dataset (which have different number of classes), you can only load model, you can not load optimizer and head. Modify your load function accordingly.

[ANDRESHZ]

Hi @humayun and @tadbeer, I hope you are having nice results, I am wondering if you finally found a way to tune with your dataset.

Can you share with the community your code?

[tadbeer]

I was unable to finetune with my dataset to my advantage. I ended up using the pretrained weights to compute features for new faces in my dataset and used those for test time comparisons. I dont have the finetuning codes with me now.

[humayun]

@ANDRESHZ Yes, I did fine tune the model with our dataset (Synthetic Faces with custom Dataset) and got slightly better results. That is 4 years ago I did this with a my previous employer. I don't have that code with me after I left the company.

Good luck for your try.

[sankalp-25]

@ANDRESHZ Hey, can you please update me if you'll find a way to fine-tune this model to recognise new faces.

[ANDRESHZ]

I am working on that way, when I get it I will create an issue to add the tutorial to fine tuning with a new data set

[humayun]

If you already have a trained model that covers all the cases you need, there is no need for additional training. Training a model becomes necessary only when the existing trained model is not accurate enough in recognizing faces in the dataset you have. This could be due to different face attributes, such as age, specific accessories like masks, tattoos, or makeup, that were not well-represented in the original training data.

In my case, since I needed to work with synthetic faces and evaluated their similarity to real faces, I required training the model from scratch. The purpose is to demonstrate that the synthetic faces we generated closely resemble real faces. Training from scratch ensures that the model learns specific features of synthetic faces and generalizes well to real faces.