Fine-tuning BERT for reaction classification doesn't work on GPU

[v-in-cube]

I am trying to re-run classification on 1k-USPTO and using code from https://rxn4chemistry.github.io/rxnfp/fine_tune_bert_on_uspto_1k_tpl Weirdly, but on CPU it trains slowly but trains, and on GPU it can't train with the following error:

model_path = "/home/varvara/rxnfp/rxnfp/models/transformers/bert_mlm_1k_tpl"
model = SmilesClassificationModel("bert", model_path, num_labels=len(final_train_df.labels.unique()), args=model_args, use_cuda=torch.cuda.is_available())
model.train_model(final_train_df, eval_df=eval_df, acc=sklearn.metrics.accuracy_score, mcc=sklearn.metrics.matthews_corrcoef)

...
Tries to train
...
AttributeError: 'SmilesClassificationModel' object has no attribute 'loss_fct'

Do you know what could be the cause of this issue?

[pschwllr]

Hi! Sorry for the late reply. Did you make sure that you used the exact same version of SimpleTransformers (simpletransformers==0.34.4)? If not, there might be some changes required to make the code work, as newer versions of simpletransformers contain breaking changes.

[v-in-cube]

Dear Phillipe, Actually, with simpletransformers==0.34.4 it gets worse, I was trying out the tutorial https://rxn4chemistry.github.io/rxnfp/fine_tune_bert_on_uspto_1k_tpl/ And inside the rxnfp environment and with the correct version of simpletransformers it can't import all the modules:

import os
import numpy as np
import pandas as pd
import torch
import logging
import random
import pkg_resources
import sklearn

from rxnfp.models import SmilesClassificationModel
logger = logging.getLogger(__name__)

---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
<ipython-input-2-8207b8c433b2> in <module>
      8 import sklearn
      9 
---> 10 from rxnfp.models import SmilesClassificationModel
     11 logger = logging.getLogger(__name__)

~/rxnfp/rxnfp/models.py in <module>
     39 # optional
     40 from simpletransformers.config.global_args import global_args
---> 41 from simpletransformers.language_modeling import (
     42     LanguageModelingModel
     43 

/opt/conda/envs/rxnfp/lib/python3.6/site-packages/simpletransformers/language_modeling/__init__.py in <module>
----> 1 from simpletransformers.language_modeling.language_modeling_model import LanguageModelingModel

/opt/conda/envs/rxnfp/lib/python3.6/site-packages/simpletransformers/language_modeling/language_modeling_model.py in <module>
     26 import torch
     27 from simpletransformers.config.global_args import global_args
---> 28 from simpletransformers.custom_models.models import ElectraForLanguageModelingModel
     29 from simpletransformers.language_modeling.language_modeling_utils import (
     30     SimpleDataset,

/opt/conda/envs/rxnfp/lib/python3.6/site-packages/simpletransformers/custom_models/models.py in <module>
     15     XLNetPreTrainedModel,
     16 )
---> 17 from transformers.configuration_distilbert import DistilBertConfig
     18 from transformers.configuration_roberta import RobertaConfig
     19 from transformers.configuration_xlm_roberta import XLMRobertaConfig

ModuleNotFoundError: No module named 'transformers.configuration_distilbert'

[pschwllr]

Could you try with transformers==4.5.0? It looks like there were breaking changes in the transformers package.

[v-in-cube]

And simpletransformers leave with 0.34.4 version?

[v-in-cube]

Dear Philippe, Could you please specify versions of crucial packages since currently I could not even load the model.

[pschwllr]

Looks like the tutorials were not updated with the rxnfp version. To reproduce the results from the paper, and have the right version I would look into https://github.com/rxn4chemistry/rxnfp/blob/v0.0.8/settings.ini.

transformers==2.5.1,<4 torch==1.3.1 scipy==1.4.1 scikit-learn==0.23.1 matplotlib==3.2.2 faerun==0.3.20

Together with simpletransformers==0.34.4. To make the training work with newer pytorch/simpletransformers versions would take some adaption to the code but should be more or less straightforward. Have you managed to make it run?