Issues while finetuning the RoBERTa model

[Soumyaworks]

Hi,

I am trying to setup sem2vec-BERT based on the instructions given in README. I am facing the following issues while fine-tuning the RoBERTa model running python src/fine_tune.py :

    - Some weights of the model checkpoint at /path to FoBERT/checkpoint-290000 were not used when initializing RobertaModel: 
    ['lm_head.dense.weight', 'lm_head.bias', 'lm_head.layer_norm.bias', 'lm_head.dense.bias', 'lm_head.layer_norm.weight']

    This IS expected if you are initializing RobertaModel from the checkpoint of a model trained on another task or with another 
    architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).

    This IS NOT expected if you are initializing RobertaModel from the checkpoint of a model that you expect to be exactly identical 
    (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
    Some weights of RobertaModel were not initialized from the model checkpoint at /path to FoBERT/checkpoint-290000 and are 
    newly initialized: ['roberta.pooler.dense.bias', 'roberta.pooler.dense.weight']

    - File "/path to transformers/tokenization_utils_base.py", line 1794, in from_pretrained
     raise EnvironmentError(
     OSError: Can't load tokenizer for '/path to FoBERT/checkpoint-290000'. If you were trying to load it from 
    'https://huggingface.co/models', make sure you don't have a local directory with the same name. Otherwise, make sure '/path to 
    FoBERT/checkpoint-290000' is the correct path to a directory containing all relevant files for a RobertaTokenizerFast tokenizer.

It would be of great help if you could kindly help in resolving this.

[whj0401]

It seems the model you are trying to load is different from what was trained. Since we use the model imported from transformers directly, I am not sure if it is caused by an incompatible library. In my environment, sentence-transformers==2.1.0 (This is the version I used to load the model for experiment, but it is not the training environment). If it works, plz reply me and I will update the readme file.

[Soumyaworks]

Hi, So I tried downgrading sentence-transformers to version 2.1.0 since using the Dockerfile provided in README , the version of sentence-transformers was 2.2.2, however currently I am facing this error while running python src/fine_tune.py :

ModuleNotFoundError: No module named 'huggingface_hub.snapshot_download'

I tried installing huggingface_hub separately, however this error persists. Would it be possible for you to share any environment with the specific versions, so that version related issues can be avoided ? It would be of great help if you can kindly help in resolving this issue.

[whj0401]

absl-py==1.3.0
ailment==9.0.10651
angr==9.0.10651
archinfo==9.0.10651
bert-pytorch==0.0.1a4
bitstring==3.1.9
cachetools==4.2.4
capstone==4.0.2
certifi==2021.10.8
cffi==1.15.0
charset-normalizer==2.0.8
claripy==9.0.10651
cle==9.0.10651
click==8.0.3
CppHeaderParser==2.7.4
cycler==0.11.0
dataclasses==0.8
decorator==4.4.2
dgl-cu113==0.7.2
dpkt==1.9.7.2
filelock==3.4.0
future==0.18.2
gitdb==4.0.9
GitPython==3.1.18
google-auth==2.15.0
google-auth-oauthlib==0.4.6
googledrivedownloader==0.4
grpcio==1.37.1
grpcio-tools==1.37.1
huggingface-hub==0.1.2
idna==3.3
importlib-metadata==4.8.2
isodate==0.6.1
itanium-demangler==1.0
Jinja2==3.0.3
joblib==1.1.0
kiwisolver==1.3.1
lark==1.0.0
Markdown==3.3.7
MarkupSafe==2.0.1
matplotlib==3.3.4
mmh3==3.0.0
mulpyplexer==0.9
nampa==0.1.1
networkx==2.5.1
nltk==3.6.5
numpy==1.19.5
oauthlib==3.2.2
packaging==21.3
pandas==1.1.5
pefile==2021.9.3
Pillow==8.4.0
pkg_resources==0.0.0
plumbum==1.7.1
ply==3.11
progressbar2==3.55.0
protobuf==3.19.1
psutil==5.8.0
psycopg2==2.9.3
py-tlsh==4.7.2
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycparser==2.21
pyelftools==0.27
pyparsing==3.0.6
PySMT==0.9.1.dev139
python-dateutil==2.8.2
python-utils==2.5.6
pytorch-metric-learning==1.0.0
pytz==2021.3
pyvex==9.0.10651
PyYAML==6.0
rdflib==5.0.0
regex==2021.11.10
requests==2.26.0
requests-oauthlib==1.3.1
rpyc==5.0.1
rsa==4.9
sacremoses==0.0.46
scikit-learn==0.24.2
scipy==1.5.4
sentence-transformers==2.1.0
sentencepiece==0.1.96
six==1.16.0
smmap==5.0.0
sortedcontainers==2.4.0
tensorboard==2.10.1
tensorboard-data-server==0.6.1
tensorboard-plugin-wit==1.8.1
threadpoolctl==3.0.0
tokenizers==0.10.3
torch==1.8.2+cu111
torch-cluster==1.5.9
torch-geometric==2.0.3
torch-scatter==2.0.9
torch-sparse==0.6.12
torch-spline-conv==1.2.1
torchaudio==0.8.2
torchvision==0.9.2+cu111
tqdm==4.62.3
transformers==4.13.0
typing_extensions==4.0.0
ujson==4.3.0
unicorn==1.0.2rc4
urllib3==1.26.7
Werkzeug==2.0.3
yacs==0.1.8
z3-solver==4.8.13.0
zipp==3.6.0

This is the version information with pip freeze. The OS is Ubuntu 18.04 and the python version is 3.6.9

Sorry for the late reply. I am trapped with something else.

[Soumyaworks]

Hi, apologies for the delay, was occupied with something. So, I am trying to setup sem2vec-BERT, and have created a new environment based on the version information that you have provided above. I am still facing the same issue, even though the versions match. Can you shed some light on the issue? And will it be possible to share the pre-trained model ?

[whj0401]

I am sorry that we cannot share the pretrained model to you directly due to privacy issues. If you are still facing the issue half years ago, I guess merely change the version at the loading side is useless. As it mentioned,

 - Some weights of the model checkpoint at /path to FoBERT/checkpoint-290000 were not used when initializing RobertaModel:

It means the saved checkpoint to be loaded is not compatible to the model being used. You need to check how you get the saved checkpoint, and make sure the checkpoint is compatible with the model you are using.