BERT: TensorFlow Model Garden Conversion scripts

stefan-it commented 1 year ago

Feature request

Hi,

after working some time with the TensorFlow Model Garden Repository and training BERT models, I found out the following things that could be changed in Transformers library:

I added the Token Dropping BERT Conversion script a while ago, see #17142. Now I found out, that latest BERT models pretrained with Model Garden Repository repository can also be converted with this script.

For this reason I would propose to rename the script convert_bert_token_dropping_original_tf2_checkpoint_to_pytorch.py just to convert_bert_original_tf2_checkpoint_to_pytorch.py.

However, this script also exists, but it is no longer working, as this was deprecated in Model Garden Repository a while ago, I added this notice in #16171.

I see now two possibilities to proceed with the different conversion scripts:

Rename the current convert_bert_original_tf2_checkpoint_to_pytorch.py to something like convert_deprecated_bert_original_tf2_checkpoint_to_pytorch.py so that this name is free for the "new" conversion script that supports Token Dropping BERT und latest BERT models from Model Garden Repository.
Delete the old script completely

Motivation

More recent BERT and Token Dropping BERT models can be pretrained with TensorFlow Model Garden repository.

There should be one script that does these conversions, the old one that is only working with deprecated models from Model Garden repo should be renamed or deleted.

Your contribution

I can take care of renaming/deletion and extending the conversion script to have better documentation.

amyeroberts commented 1 year ago

cc @Rocketknight1

Rocketknight1 commented 1 year ago

This isn't really my area either! A lot of this code goes back to the earliest code in transformers when it was a port of TF code for BERT to PyTorch. Pinging @LysandreJik - do you know if the code is intended to support ports from recent versions of the TF Model Garden?

LysandreJik commented 1 year ago

As long as everything is correctly documented, I'm all for having up to date scripts that work with the most recent BERT releases.

Rocketknight1 commented 1 year ago

In that case @stefan-it I think it's okay to delete the old script entirely and replace it with a modern one, since it's no longer usable and people who need it for historical purposes can always find it in past release branches.

stefan-it commented 1 year ago

Thanks for your feedback! I will prepare PR for this soon.

Please unstale :robot:

huggingface / transformers