search

JohnGiorgi / DeCLUTR

The corresponding code from our paper "DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations". Do not hesitate to open an issue if you run into any trouble!
https://aclanthology.org/2021.acl-long.72/
Apache License 2.0
378 stars 33 forks source link

load pretrained tf1 model with pytorch #241

Closed ZahraGhasemi-AI closed 2 years ago

ZahraGhasemi-AI commented 2 years ago

Hi. I want to load a pretrained model that trained with fensorflow 1.15. Is there any solution for it? I found some solutions but all of them was used for converting tensorflow 2 to pytorch.bin. I used tensorflow1.15.

JohnGiorgi commented 2 years ago

Hi, could you provide more detail? Could you share the stacktrace of the error for example?

ZahraGhasemi-AI commented 2 years ago

I got this error at first: ... File "/media/2TB_2/ZGH/DontDelet/venv3.6.9/lib/python3.6/site-packages/transformers/modeling_utils.py", line 625, in from_pretrained pretrained_model_name_or_path, OSError: Error no file named ['pytorch_model.bin', 'tf_model.h5', 'model.ckpt.index'] found in directory electra-small or from_tf set to False

then I set from_tf to True and I got another error: ... File "/media/2TB_2/ZGH/DontDelet/venv3.6.9/lib/python3.6/site-packages/transformers/modeling_electra.py", line 100, in load_tf_weights_in_electra assert pointer.shape == array.shape, original_name AssertionError: ('electra/encoder/layer_0/attention/self/key/bias', torch.Size([252]), (256,))

and when I used bert-base-multilingual-cased(tf version) as a transformer, I got this: ... model = load_tf2_checkpoint_in_pytorch_model(model, resolved_archive_file, allow_missing_keys=True) File "/media/2TB_2/ZGH/DontDelet/venv3.6.9/lib/python3.6/site-packages/transformers/modeling_tf_pytorch_utils.py", line 252, in load_tf2_checkpoint_in_pytorch_model tf_model_class = getattr(transformers, tf_model_class_name) AttributeError: module 'transformers' has no attribute 'TFBertForMaskedLM'

but when I used bert-base-multilingual-cased(pytorch version), it run correctly, so the problem is about load tf1...

JohnGiorgi commented 2 years ago

Hmm, these errors are coming from the transformers package, not declutr. The stack trace shows the following files:

What is the value of pretrained_model_name_or_path? Can you try loading it outside of declutr, e.g.

from transformers import AutoModel, AutoTokenizer

model = AutoModel.from_pretrained(pretrained_model_name_or_path)
tokenizer = AutoTokenizer.from_pretrained(pretrained_model_name_or_path)
ZahraGhasemi-AI commented 2 years ago

I tried it and got same error: Traceback (most recent call last): File..............., line 3, in model = AutoModel.from_pretrained('/media/2TB_2/ZGH/DontDelet/MyDeCLUTR/DeCLUTR/bert-base-multilingual-cased') File "/media/2TB_2/ZGH/DontDelet/venv3.6.9/lib/python3.6/site-packages/transformers/modeling_auto.py", line 502, in from_pretrained return model_class.from_pretrained(pretrained_model_name_or_path, *model_args, config=config, **kwargs) File "/media/2TB_2/ZGH/DontDelet/venv3.6.9/lib/python3.6/site-packages/transformers/modeling_utils.py", line 696, in from_pretrained model = load_tf2_checkpoint_in_pytorch_model(model, resolved_archive_file, allow_missing_keys=True) File "/media/2TB_2/ZGH/DontDelet/venv3.6.9/lib/python3.6/site-packages/transformers/modeling_tf_pytorch_utils.py", line 252, in load_tf2_checkpoint_in_pytorch_model tf_model_class = getattr(transformers, tf_model_class_name) AttributeError: module 'transformers' has no attribute 'TFBertModel'

JohnGiorgi commented 2 years ago

Right, so this suggests the problem is with transformers, not declutr. You may have to ask for help on the transformers repo!