convert pre-trained tf model to pytorch error

[ericlormul]

I used following script to convert pre-trained model tanda_bert_base_asnq

export BERT_BASE_DIR=/path/to/bert/uncased_L-12_H-768_A-12

transformers-cli convert --model_type bert \
  --tf_checkpoint $BERT_BASE_DIR/bert_model.ckpt \
  --config $BERT_BASE_DIR/bert_config.json \
  --pytorch_dump_output $BERT_BASE_DIR/pytorch_model.bin

but it gives error:

  File ".../anaconda3/bin/transformers-cli", line 30, in <module>
    service.run()
  File ".../anaconda3/lib/python3.7/site-packages/transformers/commands/convert.py", line 62, in run
    convert_tf_checkpoint_to_pytorch(self._tf_checkpoint, self._config, self._pytorch_dump_output)
  File ".../anaconda3/lib/python3.7/site-packages/transformers/convert_bert_original_tf_checkpoint_to_pytorch.py", line 36, in convert_tf_checkpoint_to_pytorch
    load_tf_weights_in_bert(model, config, tf_checkpoint_path)
  File ".../anaconda3/lib/python3.7/site-packages/transformers/modeling_bert.py", line 99, in load_tf_weights_in_bert
    pointer = getattr(pointer, 'bias')
  File ".../anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 591, in __getattr__
    type(self).__name__, name))
AttributeError: 'BertForPreTraining' object has no attribute 'bias'

I'm using transformers 2.3.0 and pytorch 1.2.0

Thanks!

[cooscao]

Have you solved it? I have meet the same problem.

[sid7954]

This error is due to using the convert_tf_checkpoint_to_pytorch script directly which has been written to convert Google's pre-trained BERT models (which are initialised as BertForPreTraining objects). The models we have released are objects of BertForSequenceClassification and thus there are some changes that need to be made to the convert_tf_checkpoint_to_pytorch script to make it work correctly.

The same issues was raised on the huggingface transformers repository, and can you can refer here for the solution.

[Olivia-fsm]

Step1: Clone the pytorch-pretrained-BERT toolkit from here --> pytorch-pretrained-BERT Step2: Manually change the convert_tf_checkpoint_to_pytorch.py script as shared here --> Personal Manually Changed Script Step3: Run in the terminal python convert_tf_checkpoint_to_pytorch.py --tf_checkpoint_path [your TF model ckpt] --bert_config_file [your TF config] --pytorch_dump_path [target save path of pytorch model] Step4: Get something delicious for celebration!

[xiaojiev587]

Step1: Clone the pytorch-pretrained-BERT toolkit from here --> pytorch-pretrained-BERT Step2: Manually change the convert_tf_checkpoint_to_pytorch.py script as shared here --> Personal Manually Changed Script Step3: Run in the terminal python convert_tf_checkpoint_to_pytorch.py --tf_checkpoint_path [your TF model ckpt] --bert_config_file [your TF config] --pytorch_dump_path [target save path of pytorch model] Step4: Get something delicious for celebration!

hi，同学，请问你使用转换后的pytorch权重可以复现出作者在wikiQA上的表现吗？我加载权重进行预测指标相差很多，特来请教~