Not support for loading pretrained HuggingFace Transformers model from local path?

[WangHexie]

Failed to load pretrained HuggingFace Transformers model from my local machine. It seems only the hard-code models in the code can be loaded.

MODEL_NAME = "D:\programming\models\tf_rbtl"
t = text.Transformer(MODEL_NAME, maxlen=500,  
                     classes=["0", "1"])

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-15-e20b30887588> in <module>
      1 t = text.Transformer(MODEL_NAME, maxlen=500,  
----> 2                      classes=["0", "1"])

d:\anaconda3.5\envs\adverse\lib\site-packages\ktrain\text\preprocessor.py in __init__(self, model_name, maxlen, classes, batch_size, multilabel, use_with_learner)
    838             raise ValueError('classes argument is required when multilabel=True')
    839         super().__init__(model_name,
--> 840                          maxlen, max_features=10000, classes=classes, multilabel=multilabel)
    841         self.batch_size = batch_size
    842         self.use_with_learner = use_with_learner

d:\anaconda3.5\envs\adverse\lib\site-packages\ktrain\text\preprocessor.py in __init__(self, model_name, maxlen, max_features, classes, lang, ngram_range, multilabel)
    719         self.name = model_name.split('-')[0]
    720         if self.name not in TRANSFORMER_MODELS:
--> 721             raise ValueError('uknown model name %s' % (model_name))
    722         self.model_type = TRANSFORMER_MODELS[self.name][1]
    723         self.tokenizer_type = TRANSFORMER_MODELS[self.name][2]

ValueError: uknown model name D:\programming\models\tf_rbtl

[amaiya]

The Transformer constructor currently only accepts a pretrained model name in the form of a string.

Can you clarify what tf_rbtl is in your example?

                    - a string with the `shortcut name` of a pre-trained model configuration to load from cache or download, e.g.: ``bert-base-uncased``.
                    - a string with the `identifier name` of a pre-trained model configuration that was user-uploaded to our S3, e.g.: ``dbmdz/bert-base-german-cased``.
                    - a path to a `directory` containing a configuration file saved using the :func:`~transformers.PretrainedConfig.save_pretrained` method, e.g.: ``./my_model_directory/``.
                    - a path or url to a saved configuration JSON `file`, e.g.: ``./my_model_directory/configuration.json``.

[WangHexie]

It's

- a path to a `directory` containing a configuration file saved using the :func:`~transformers.PretrainedConfig.save_pretrained` method, e.g.: ``./my_model_directory/``.

And the file structure in the directory is like this:

• tf_rbtl
  • bert_config_rbtl3.json
  • bert_model.ckpt.data-00000-of-00001
  • bert_model.ckpt.index
  • bert_model.ckpt.meta
  • vocab.txt

Changing the string to - a path to a saved configuration JSON file, e.g.: /my_model_directory/configuration.json. didn't work too.

I check the source code of ktrain, it seems it only checks if string is in these ones

I downloaded model from this repo Chinese-BERT-wwm and this is the model link.I extracted the model directly.

[amaiya]

OK - thanks for the information. We'll plan to add support for this.

[mdavis95]

I would love to use the models from https://huggingface.co/models. For example https://huggingface.co/monologg/scibert_scivocab_uncased.

tokenizer = AutoTokenizer.from_pretrained("monologg/scibert_scivocab_uncased")

model = AutoModelWithLMHead.from_pretrained("monologg/scibert_scivocab_uncased")

Is this something that could be supported?

[amaiya]

@mdavis95 Yes, we are planning to support this in an upcoming version, which will address the use cases of both you and @WangHexie

Thanks.

[amaiya]

@WangHexie I downloaded the model from the link you provided and tried to load it using the transformers library directly. None of the following work:

fpath = '/tmp/tf_rbtl/'
model = TFAutoModelForSequenceClassification.from_pretrained(fpath)
tok = AutoTokenizer.from_pretrained(fpath)
config = AutoConfig.from_pretrained(fpath)

Have you verified that you can load this model directly using the transformers library? If so, please post the steps.

[amaiya]

@mdavis95 I just looked into loading scibert using the transformers library directly. The PyTorch versions (both 'monologg/scibert_scivocab_uncased' and the model downloaded from here) both work perfectly in PyTorch. However, the TensorFlow version from here does not work. And, the monologg model does not appear to work with TensorFlow, as well.

Do you have a working example of loading scibert in TensorFlow using transformers directly?

Here is what I tried:

# WORKS (PyTorch)
model = AutoModelForSequenceClassification.from_pretrained('monologg/scibert_scivocab_uncased')

# WORKS (Pytorch)
# download HF Pytorch model from here: https://github.com/allenai/scibert/blob/master/README.md
fpath = '/tmp/pytorch_scibert/scibert_scivocab_uncased/'
model = AutoModelForSequenceClassification.from_pretrained(fpath)

# DOES NOT WORK (TensorFlow)
model = TFAutoModelWithLMHead.from_pretrained("monologg/scibert_scivocab_uncased")
model = TFAutoModelForSequenceClassification.from_pretrained('monologg/scibert_scivocab_uncased')

# DOES NOT WORK (TensorFlow)
# download TF scibert model from here: https://github.com/allenai/scibert/blob/master/README.md
fpath = '/tmp/scibert_scivocab_uncased/'
model = TFAutoModelForSequenceClassification.from_pretrained(fpath+'/bert_config.json')

[mdavis95]

@amaiya looking at the files from monologg/scibert_scivocab_uncased I can definitely see it is a pytorch model. I will look into if there is a tensorflow compatible model somewhere.

[mdavis95]

what about

model = TFAutoModelForSequenceClassification.from_pretrained('monologg/scibert_scivocab_uncased', from_pt=True)

[mdavis95]

if that works #68 might be solved by

model = TFAutoModelForSequenceClassification.from_pretrained('monologg/biobert_v1.1_pubmed', from_pt=True)

[amaiya]

Thanks for the suggestion. I tried it and got this error, unfortunately:

OSError: Loading a TF model from a PyTorch checkpoint is not supported when using a model identifier name.

Error was generated from this:

model = TFAutoModelForSequenceClassification.from_pretrained('monologg/scibert_scivocab_uncased', from_pt=True)

[mdavis95]

Weird I wrote the following three lines in a python file

from transformers import AutoModelForSequenceClassification, TFAutoModelForSequenceClassification, AutoConfig

model = TFAutoModelForSequenceClassification.from_pretrained('monologg/biobert_v1.1_pubmed', from_pt=True)
print(model)

and it gives a lot of logging then

<transformers.modeling_tf_bert.TFBertForSequenceClassification object at 0x7f47d276f7d0>

[mdavis95]

Which version of transformers are you using?

[amaiya]

@mdavis95 Thanks - I upgraded to 2.5.1 and it works now. I was able to successfully load both scibert and biobert in TF2. We'll now plan on adding support for such models in an upcoming release. Thanks again for the assistance.

[mdavis95]

@amaiya any time, thanks for the great library

[amaiya]

@mdavis95 Thanks again for the help. Support for community-uploaded transformer models is included in v0.10.0 of ktrain (just released).

Example:

import ktrain
from ktrain import text
MODEL_NAME = 'monologg/scibert_scivocab_uncased'
t = text.Transformer(MODEL_NAME, maxlen=500, class_names=label_list)
trn = t.preprocess_train(x_train, y_train)
val = t.preprocess_test(x_test, y_test)
model = t.get_classifier()
learner = ktrain.get_learner(model, train_data=trn, val_data=val, batch_size=6)
learner.fit_onecycle(3e-5, 1)

[atowey01]

@amaiya does the latest release include support for models saved with save_pretrained() from transformers as mentioned by @WangHexie or just community uploaded models?

[amaiya]

@atowey01 Yes, this is supported.

In fact, when you do learner.save_model('/tmp/mymodel') and then learner.load_model('/tmp/mymodel'), that is exactly what it is doing (i.e., calling save_pretrained and from_pretrained).

If you encounter issues, please do open an issue.

[atowey01]

@amaiya thanks for your response. Is there a way to use a pretrained model (a huggingface bert model which I have finetuned using my own text and saved with save_pretrained()) in text.transformer? Ideally I would like to be able to give a path to my pretrained model folder to model_name in text.transformer but it only seems to accept predefined huggingface or community uploaded models.

[amaiya]

Yes, that should work.

What is the error you're getting?

Have you verified that the model loads correctly when directly using the transformers library using commands like the following?

from transformers import * 
tokenizer = AutoTokenizer.from_pretrained(path_to_your_model)
model =TFAutoModelForSequenceClassification.from_pretrained(path_to_your_model, from_pt=<VALUE>)

Try both from_pt=True and from_pt=False.

What is the code you used to create and save your fine-tuned model?

What version of transformers are you using?

[atowey01]

@amaiya Apologies for the delay in getting back. Realised it was package versions causing the issues and when updated it works, thank you!

[meliksahturker]

I had the same issue with GPU enabled CUDA. I had to turn off GPU in tensorflow then I was able to use the function get_classifier() to create model.