Issue with fine-tuning

[marcosfelt]

I am trying to using a model I already pretrained as a starting point for fine-tuning. However, the process does not seem to work as it does when I start with the rxnfp base models. Below is a rough outline of what I am doing:

name = 'fine_tune_reaxys'
base_model = "pretrained"
output_path = f"models/my_model_{base_model}_{name}"
dropout=0.7987
learning_rate=0.00009659

model_args = {
'num_train_epochs': 10, 'overwrite_output_dir': True,
'learning_rate': learning_rate, 'gradient_accumulation_steps': 1,
'regression': True, "num_labels":1, "fp16": False,
"evaluate_during_training": True, 'manual_seed': 42,
"max_seq_length": 300, "train_batch_size": 16,"warmup_ratio": 0.00,
"config" : { 'hidden_dropout_prob': dropout } 
}

#Load pretrained model
model_path =  "models/my_pretrained_model/checkpoint-6240-epoch-10"
pretrained_bert = SmilesClassificationModel("bert", 
                                            model_path, 
                                            num_labels=1, 
                                            args=model_args,
                                            use_cuda=torch.cuda.is_available())

#Train model
pretrained_bert.train_model(train_df, 
                            output_dir=output_path,
                            eval_df=test_df,
                            r2=r2_score)

When I run this, it starts training but then fails to run any epochs and save any checkpoints.

Edit: I'm using transformers==2.11.0 and simpletransformers==0.34.4

[pschwllr]

This seems to be a frequent issue when fine-tuning on your own models. Depending on the files in the model folder the fine-tuning run does not start.

The best is to copy your pretrained model into a new folder keeping only three files config.json, pytorch_model.bin and vocab.txt, as in https://github.com/rxn4chemistry/rxnfp/tree/master/rxnfp/models/transformers/bert_pretrained.

The fine-tuning can then be started setting the model_path to the freshly created folder.

Thanks to @marcosfelt for helping us on this issue!