search

allenai / longformer

Longformer: The Long-Document Transformer
https://arxiv.org/abs/2004.05150
Apache License 2.0
2.05k stars 276 forks source link

AttributeError: 'RobertaEmbeddings' object has no attribute 'position_ids' #238

Closed XueqiYang closed 2 years ago

XueqiYang commented 2 years ago

Got this error when I was converting a roberta-base model into roberta-base-4096 which is an instance of RobertaLong, then save it to the disk.

INFO:main:Converting roberta-base into roberta-base-4096 INFO:transformers.configuration_utils:loading configuration file https://s3.amazonaws.com/models.huggingface.co/bert/roberta-base-config.json from cache at /root/.cache/torch/transformers/e1a2a406b5a05063c31f4dfdee7608986ba7c6393f7f79db5e69dcd197208534.117c81977c5979de8c088352e74ec6e70f5c66096c28b61d3c50101609b39690 INFO:transformers.configuration_utils:Model config RobertaConfig { "architectures": [ "RobertaForMaskedLM" ], "attention_probs_dropout_prob": 0.1, "bos_token_id": 0, "eos_token_id": 2, "gradient_checkpointing": false, "hidden_act": "gelu", "hidden_dropout_prob": 0.1, "hidden_size": 768, "initializer_range": 0.02, "intermediate_size": 3072, "layer_norm_eps": 1e-05, "max_position_embeddings": 514, "model_type": "roberta", "num_attention_heads": 12, "num_hidden_layers": 12, "pad_token_id": 1, "type_vocab_size": 1, "vocab_size": 50265 }

INFO:transformers.modeling_utils:loading weights file https://cdn.huggingface.co/roberta-base-pytorch_model.bin from cache at /root/.cache/torch/transformers/80b4a484eddeb259bec2f06a6f2f05d90934111628e0e1c09a33bd4a121358e1.49b88ba7ec2c26a7558dda98ca3884c3b80fa31cf43a1b1f23aef3ff81ba344e INFO:transformers.modeling_utils:All model checkpoint weights were used when initializing RobertaForMaskedLM.

WARNING:transformers.modeling_utils:Some weights of RobertaForMaskedLM were not initialized from the model checkpoint at roberta-base and are newly initialized: ['lm_head.decoder.bias'] You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference. INFO:transformers.tokenization_utils_base:loading file https://s3.amazonaws.com/models.huggingface.co/bert/roberta-base-vocab.json from cache at /root/.cache/torch/transformers/d0c5776499adc1ded22493fae699da0971c1ee4c2587111707a4d177d20257a2.ef00af9e673c7160b4d41cfda1f48c5f4cba57d5142754525572a846a1ab1b9b INFO:transformers.tokenization_utils_base:loading file https://s3.amazonaws.com/models.huggingface.co/bert/roberta-base-merges.txt from cache at /root/.cache/torch/transformers/b35e7cd126cd4229a746b5d5c29a749e8e84438b14bcdb575950584fe33207e8.70bec105b4158ed9a1747fea67a43f5dee97855c64d62b6ec3742f4cfdb5feda

AttributeError Traceback (most recent call last) in () 5 logger.info(f'Converting roberta-base into roberta-base-{model_args.max_pos}') 6 model, tokenizer = create_long_model( ----> 7 save_model_to=model_path, attention_window=model_args.attention_window, max_pos=model_args.max_pos)

1 frames /usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py in getattr(self, name) 1184 return modules[name] 1185 raise AttributeError("'{}' object has no attribute '{}'".format( -> 1186 type(self).name, name)) 1187 1188 def setattr(self, name: str, value: Union[Tensor, 'Module']) -> None:

AttributeError: 'RobertaEmbeddings' object has no attribute 'position_ids'

XueqiYang commented 2 years ago

Never mind. I saw someone else had addressed this issue in Issue#152. Thanks1