What to do about this warning message: "Some weights of the model checkpoint at bert-base-uncased were not used when initializing BertForSequenceClassification"

[ohmeow]

model = AutoModelForSequenceClassification.from_pretrained("bert-base-uncased")

returns this warning message:

Some weights of the model checkpoint at bert-base-uncased were not used when initializing BertForSequenceClassification: ['cls.predictions.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.decoder.weight', 'cls.seq_relationship.weight', 'cls.seq_relationship.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.LayerNorm.bias']
- This IS expected if you are initializing BertForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPretraining model).
- This IS NOT expected if you are initializing BertForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of BertForSequenceClassification were not initialized from the model checkpoint at bert-base-uncased and are newly initialized: ['classifier.weight', 'classifier.bias']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.

This just started popping up with v.3 so I'm not sure what is the recommended action to take here. Please advise if you can. Basically, any of my code using the AutoModelFor<X> is throwing up this warning now.

Thanks.

[julien-c]

Not sure what's happening with the multiple duplicate opened issues, @ohmeow?

Is GitHub flaky again? :)

[fliptrail]

I am also encountering the same warning.

When loading the model

Some weights of the model checkpoint at bert-base-uncased were not used when initializing TFBertModel: ['nsp___cls', 'mlm___cls']
- This IS expected if you are initializing TFBertModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPretraining model).
- This IS NOT expected if you are initializing TFBertModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
All the weights of TFBertModel were initialized from the model checkpoint at bert-base-uncased.
If your task is similar to the task the model of the ckeckpoint was trained on, you can already use TFBertModel for predictions without further training.

When attempting to fine tune it:

WARNING:tensorflow:Gradients do not exist for variables ['tf_bert_model_1/bert/pooler/dense/kernel:0', 'tf_bert_model_1/bert/pooler/dense/bias:0'] when minimizing the loss.
WARNING:tensorflow:Gradients do not exist for variables ['tf_bert_model_1/bert/pooler/dense/kernel:0', 'tf_bert_model_1/bert/pooler/dense/bias:0'] when minimizing the loss.
WARNING:tensorflow:Gradients do not exist for variables ['tf_bert_model_1/bert/pooler/dense/kernel:0', 'tf_bert_model_1/bert/pooler/dense/bias:0'] when minimizing the loss.
WARNING:tensorflow:Gradients do not exist for variables ['tf_bert_model_1/bert/pooler/dense/kernel:0', 'tf_bert_model_1/bert/pooler/dense/bias:0'] when minimizing the loss.

Is the model correctly fine-tuning? Are the pre-trained model weights also getting updated (fine-tuned) or only the layers outside(above) the pre-trained model are changing their weights while training?

[ohmeow]

Not sure what's happening with the multiple duplicate opened issues, @ohmeow?

Is GitHub flaky again? :)

I noticed the same thing. Not sure what is going on ... but I swear I only opened this one :)

[LysandreJik]

@ohmeow you're loading the bert-base-cased checkpoint (which is a checkpoint that was trained using a similar architecture to BertForPreTraining) in a BertForSequenceClassification model.

This means that:

• The layers that BertForPreTraining has, but BertForSequenceClassification does not have will be discarded
• The layers that BertForSequenceClassification has but BertForPreTraining does not have will be randomly initialized.

This is expected, and tells you that you won't have good performance with your BertForSequenceClassification model before you fine-tune it :slightly_smiling_face:.

@fliptrail this warning means that during your training, you're not using the pooler in order to compute the loss. I don't know how you're finetuning your model, but if you're not using the pooler layer then there's no need to worry about that warning.

[fliptrail]

@LysandreJik Thank you for your response. I am using the code:

def main_model():
  encoder = ppd.TFBertModel.from_pretrained("bert-base-uncased")
  input_ids = tf.keras.layers.Input(shape=(max_seq_len,), dtype=tf.int32)
  token_type_ids = tf.keras.layers.Input(shape=(max_seq_len,), dtype=tf.int32)
  attention_mask = tf.keras.layers.Input(shape=(max_seq_len,), dtype=tf.int32)

  embedding = encoder(input_ids, token_type_ids=token_type_ids, attention_mask=attention_mask)[0]

  pooling = tf.keras.layers.GlobalAveragePooling1D()(embedding)
  normalization = tf.keras.layers.BatchNormalization()(pooling)
  dropout = tf.keras.layers.Dropout(0.1)(normalization)

  out = tf.keras.layers.Dense(1, activation="sigmoid", name="final_output_bert")(dropout)

  model = tf.keras.Model(inputs=[input_ids, token_type_ids, attention_mask], outputs=out)

  loss = tf.keras.losses.BinaryCrossentropy(from_logits=True)
  optimizer = tf.keras.optimizers.Adam(lr=2e-5)
  metrics=['accuracy', tf.keras.metrics.FalseNegatives(), tf.keras.metrics.FalsePositives()]

  model.compile(optimizer=optimizer, loss=loss, metrics=metrics)
  return model

model = main_model()
model.summary()

I am only using the TFBertModel.from_pretrained("bert-base-uncased") pre-built class. I am not initializing it from any other class. Still, I am encountering the warning. From what I can understand this should only appear when initializing given pre-trained model inside another class. Am I fine-tuning correctly? Are the BERT layer weights also getting updated?

Warning while loading model:

Some weights of the model checkpoint at bert-base-uncased were not used when initializing TFBertModel: ['nsp___cls', 'mlm___cls']
- This IS expected if you are initializing TFBertModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPretraining model).
- This IS NOT expected if you are initializing TFBertModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
All the weights of TFBertModel were initialized from the model checkpoint at bert-base-uncased.
If your task is similar to the task the model of the ckeckpoint was trained on, you can already use TFBertModel for predictions without further training.

While attempting to train:

WARNING:tensorflow:Gradients do not exist for variables ['tf_bert_model_1/bert/pooler/dense/kernel:0', 'tf_bert_model_1/bert/pooler/dense/bias:0'] when minimizing the loss.

This warning only started to appear from yesterday in all my codes and other sample codes given.

[VaibhavBhatnagar17]

Hello everyone, I also start getting this error today. before today it was working fine. Are there any changes that take place in colab? This is the code I am using:

!pip install transformers
import TensorFlow as to
import transformers
from transformers import TFBertForSequenceClassification, BertConfig
tokenizer = transformers.BertTokenizer('gdrive/My Drive/Colab Notebooks/vocab.txt', do_lower_case=True)

max_seq_length = 128

bert = 'bert-large-uncased'
config = BertConfig.from_pretrained('bert-large-uncased', output_hidden_states=True, hidden_dropout_prob=0.2, 
attention_probs_dropout_prob=0.2)

transformer_model = TFBertForSequenceClassification.from_pretrained(bert, config=config)

input_ids_in = tf.keras.layers.Input(shape=(max_seq_length,), name='input_token', dtype='int32')
input_masks_in = tf.keras.layers.Input(shape=(max_seq_length,), name='masked_token', dtype='int32')
input_segments_in = tf.keras.layers.Input(shape=(max_seq_length,), name='segment_ids', dtype='int32') 

embedding_layer = transformer_model(input_ids_in, attention_mask=input_masks_in, token_type_ids=input_segments_in)

I have been using this same code for more than 2 weeks and no problem till yesterday. Please if anyone finds the solution, share it. Thank you

[ohmeow]

Thanks @LysandreJik

This is expected, and tells you that you won't have good performance with your BertForSequenceClassification model before you fine-tune it

Makes sense.

Now, how do we know what checkpoints are available that were trained on BertForSequenceClassification?

[LysandreJik]

@fliptrail in your code you have the following:

embedding = encoder(input_ids, token_type_ids=token_type_ids, attention_mask=attention_mask)[0]

which means you're only getting the first output of the model, and using that to compute the loss. The first output of the model is the hidden states:

https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_tf_bert.py#L716-L738

    Returns:
        :obj:`tuple(tf.Tensor)` comprising various elements depending on the configuration (:class:`~transformers.BertConfig`) and inputs:
        last_hidden_state (:obj:`tf.Tensor` of shape :obj:`(batch_size, sequence_length, hidden_size)`):
            Sequence of hidden-states at the output of the last layer of the model.
        pooler_output (:obj:`tf.Tensor` of shape :obj:`(batch_size, hidden_size)`):
            Last layer hidden-state of the first token of the sequence (classification token)
            further processed by a Linear layer and a Tanh activation function. The Linear
            layer weights are trained from the next sentence prediction (classification)
            objective during Bert pretraining. This output is usually *not* a good summary
            of the semantic content of the input, you're often better with averaging or pooling
            the sequence of hidden-states for the whole input sequence.
        hidden_states (:obj:`tuple(tf.Tensor)`, `optional`, returned when ``output_hidden_states=True`` is passed or when ``config.output_hidden_states=True``):
            tuple of :obj:`tf.Tensor` (one for the output of the embeddings + one for the output of each layer)
            of shape :obj:`(batch_size, sequence_length, hidden_size)`.
            Hidden-states of the model at the output of each layer plus the initial embedding outputs.
        attentions (:obj:`tuple(tf.Tensor)`, `optional`, returned when ``output_attentions=True`` is passed or when ``config.output_attentions=True``):
            tuple of :obj:`tf.Tensor` (one for each layer) of shape
            :obj:`(batch_size, num_heads, sequence_length, sequence_length)`:
            Attentions weights after the attention softmax, used to compute the weighted average in the self-attention
            heads.
        """

You're ignoring the second value which is the pooler output. The warnings are normal in your case.

[LysandreJik]

@VaibhavBhatnagar17, these are warnings, not errors. What exact warning are you not understanding?

[LysandreJik]

@ohmeow that really depends on what you want to do! Sequence classification is a large subject, with many different tasks. Here's a list of all available checkpoints fine-tuned on sequence classification (not all are for BERT, though!)

Please be aware that if you have a specific task in mind, you should fine-tune your model to that task.

[VaibhavBhatnagar17]

@LysandreJik Hey, What I am not able to understand is that I was using this code for more than 2 weeks and no warning came up till yesterday. I haven't changed anything but suddenly this warning came up is confusing. I am not getting the same output dimension as before and not able to complete my project.

[LysandreJik]

The warning came up yesterday because version 3.0.0 was released yesterday. It's weird that you saw an output dimension changed since yesterday. What's the error you get?

[tarskiandhutch]

I see this same warning when initializing BertForMaskedLM, pasted in below for good measure. As other posters have mentioned, this warning began appearing only after upgrading to v3.0.0.

Some weights of the model checkpoint at bert-large-uncased-whole-word-masking were not used when initializing BertForMaskedLM: ['cls.seq_relationship.weight', 'cls.seq_relationship.bias']
- This IS expected if you are initializing BertForMaskedLM from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPretraining model).
- This IS NOT expected if you are initializing BertForMaskedLM from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of BertForMaskedLM were not initialized from the model checkpoint at bert-large-uncased-whole-word-masking and are newly initialized: ['cls.predictions.decoder.bias']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.

Note that my module imports/initializations essentially duplicate the snippet demonstrating cloze task usage at https://huggingface.co/bert-large-uncased-whole-word-masking?text=Paris+is+the+%5BMASK%5D+of+France.

from transformers import BertTokenizer, BertForMaskedLM

_tokenizer = BertTokenizer.from_pretrained(
    'bert-large-uncased-whole-word-masking')
_model = BertForMaskedLM.from_pretrained(
    'bert-large-uncased-whole-word-masking')

Am I correct in assuming that nothing has changed in the behavior of the relevant model, but that perhaps this warning should have been being printed all along?

[LysandreJik]

You're right, this has always been the behavior of the models. It wasn't clear enough before, so we've clarified it with this warning.

[tarskiandhutch]

Thanks, @LysandreJik .

[ehalit]

Anyone knows how to suppress this warning? I am aware that the model needs fine-tuning and I am fine-tuning it so, it becomes annoying to see this over and over again.

[LysandreJik]

You can manage the warnings with the logging utility introduced in version 3.1.0:

from transformers import logging

logging.set_verbosity_warning()

[ehalit]

@LysandreJik Thanks for the rapid response, I set it with set_verbosity_error()

[s4sarath]

@LysandreJik - So , by default bert-base-uncased loading from TFBertModel has 199 variables [ 3embedding + 2 layer norms + (16 x 12 layers) + 2 (pooler kernel and bias )].

But when loading from TFBertForMaskedLM, it has 204 variables. Below are the 5 extra variables

tf_bert_for_masked_lm_1/mlm___cls/predictions/bias:0
tf_bert_for_masked_lm_1/mlm___cls/predictions/transform/dense/kernel:0
tf_bert_for_masked_lm_1/mlm___cls/predictions/transform/dense/bias:0
tf_bert_for_masked_lm_1/mlm___cls/predictions/transform/LayerNorm/gamma:0
tf_bert_for_masked_lm_1/mlm___cls/predictions/transform/LayerNorm/beta:0

So that means , these 5 variables are randomly initialising right. Are these 5 variables required for MLM ( is this how it is in official tensorflow models )

OR

can we take output token embeddings ( before passing to mlm___cls ) ( batch x sequence x embedding_dimension ), multiply it with word_embedding matrix to produce ( batch x sequence x vocab_size ) and then use that for MLM loss .

[veronica320]

@LysandreJik I'm having a slightly different issue here - I'm loading a sequence classification checkpoint in a AutoModelForSequenceClassification model. But I still get the warning. Here's my code:

from transformers import AutoTokenizer, AutoModelForSequenceClassification
model = AutoModelForSequenceClassification.from_pretrained('roberta-large-mnli')

Output:

Some weights of the model checkpoint at roberta-large-mnli were not used when initializing RobertaForSequenceClassification: ['roberta.pooler.dense.weight', 'roberta.pooler.dense.bias']
- This IS expected if you are initializing RobertaForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPretraining model).
- This IS NOT expected if you are initializing RobertaForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).

I believe it's NOT expected because I'm indeed initializing from a model that I expect to be exactly identical.

I'm only starting to get this warning after upgrading to transformers v3 as well. I'm using 3.3.1 currently. Could you please help? Thanks!

[LysandreJik]

@s4sarath I'm not sure I understand your question.

@veronica320, the pooler layer is not used when doing sequence classification, so there's nothing to be worried about.

The pooler is the second output of the RobertaModel: https://github.com/huggingface/transformers/blob/v3.4.0/src/transformers/modeling_roberta.py#L691

But only the first output is used in the sequence classification model: https://github.com/huggingface/transformers/blob/v3.4.0/src/transformers/modeling_roberta.py#L1002

[veronica320]

Thanks a lot!

[s4sarath]

@LysandreJik - Sorry to make you confused .

tf_bert_for_masked_lm_1/mlm___cls/predictions/bias:0
tf_bert_for_masked_lm_1/mlm___cls/predictions/transform/dense/kernel:0
tf_bert_for_masked_lm_1/mlm___cls/predictions/transform/dense/bias:0
tf_bert_for_masked_lm_1/mlm___cls/predictions/transform/LayerNorm/gamma:0
tf_bert_for_masked_lm_1/mlm___cls/predictions/transform/LayerNorm/beta:0

The above 4 variables are randomly initialising right, means they were not a part of official BERT . Am i right?

[LysandreJik]

Thank you for your explanation.

Actually these four variables shouldn't be initialized randomly, as they're part of BERT. The official BERT checkpoints contain two heads: the MLM head and the NSP head.

You can see it here:

>>> from transformers import TFBertForMaskedLM
>>> model = TFBertForMaskedLM.from_pretrained("bert-base-cased")

Among the logging, you should find this:

Some layers from the model checkpoint at bert-base-cased were not used when initializing TFBertForMaskedLM: ['nsp___cls']
- This IS expected if you are initializing TFBertForMaskedLM from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPretraining model).
- This IS NOT expected if you are initializing TFBertForMaskedLM from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
All the layers of TFBertForMaskedLM were initialized from the model checkpoint at bert-base-cased.

This tells you two things:

• Some layers of the checkpoints are not used. These are ['nsp___cls'], corresponding to the CLS head. Since we're using a ***ForMaskedLM, it makes sense not to use the CLS head
• All the layers of the model were initialized from the model checkpoint, as both the transformer layers and the MLM head were present in the checkpoint.

If you're getting those variables randomly initialized:

tf_bert_for_masked_lm_1/mlm___cls/predictions/bias:0
tf_bert_for_masked_lm_1/mlm___cls/predictions/transform/dense/kernel:0
tf_bert_for_masked_lm_1/mlm___cls/predictions/transform/dense/bias:0
tf_bert_for_masked_lm_1/mlm___cls/predictions/transform/LayerNorm/gamma:0
tf_bert_for_masked_lm_1/mlm___cls/predictions/transform/LayerNorm/beta:0

then it means you're using a checkpoint that does not contain these variables. These are the MLM layers, so you're probably loading a checkpoint that was saved using an architecture that does not contain these layers. This can happen if you do the following:

>>> from transformers import TFBertModel, TFBertForMaskedLM
>>> model = TFBertModel.from_pretrained("bert-base-cased")
>>> model.save_pretrained(directory)
>>> mlm_model = TFBertForMaskedLM.from_pretrained(directory)

I hope this answers your question!

[s4sarath]

Oh okay. Thank you so much for the clarification. When I looked at bert models from tf-hub , these 4 variables were not present. That was the reason for the confusion .

On Tue, Oct 27, 2020, 7:02 PM Lysandre Debut notifications@github.com wrote:

Thank you for your explanation.

Actually these four variables shouldn't be initialized randomly, as they're part of BERT. The official BERT checkpoints contain two heads: the MLM head and the NSP head.

You can see it here:

from transformers import TFBertForMaskedLM>>> model = TFBertForMaskedLM.from_pretrained("bert-base-cased")

Among the logging, you should find this:

Some layers from the model checkpoint at bert-base-cased were not used when initializing TFBertForMaskedLM: ['nsp___cls']

• This IS expected if you are initializing TFBertForMaskedLM from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPretraining model).
• This IS NOT expected if you are initializing TFBertForMaskedLM from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model). All the layers of TFBertForMaskedLM were initialized from the model checkpoint at bert-base-cased.

This tells you two things:

• Some layers of the checkpoints are not used. These are ['nsp___cls'], corresponding to the CLS head. Since we're using a ***ForMaskedLM, it makes sense not to use the CLS head
• All the layers of the model were initialized from the model checkpoint, as both the transformer layers and the MLM head were present in the checkpoint.

If you're getting those variables randomly initialized:

tf_bert_for_masked_lm_1/mlm_cls/predictions/bias:0 tf_bert_for_masked_lm1/mlmcls/predictions/transform/dense/kernel:0 tf_bert_for_masked_lm_1/mlm_cls/predictions/transform/dense/bias:0 tf_bert_for_masked_lm1/mlmcls/predictions/transform/LayerNorm/gamma:0 tf_bert_for_masked_lm_1/mlm___cls/predictions/transform/LayerNorm/beta:0

then it means you're using a checkpoint that does not contain these variables. These are the MLM layers, so you're probably loading a checkpoint that was saved using an architecture that does not contain these layers. This can happen if you do the following:

from transformers import TFBertModel, TFBertForMaskedLM>>> model = TFBertModel.from_pretrained("bert-base-cased")>>> model.save_pretrained(directory)>>> mlm_model = TFBertForMaskedLM.from_pretrained(directory)

I hope this answers your question!

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/huggingface/transformers/issues/5421#issuecomment-717245807, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACRE6KEEQACWSAEO3GK3CL3SM3DYNANCNFSM4OM5S2SQ .

[s4sarath]

Hi @LysandreJik . I had a look at the official BERT repo . There are only 199 variables in the official model checkpoints. Which means, of 204 variables ( last 5 variables for MLM layer ) is initialised randomly. These variables are not a part of official checkpoints I think.

[rkunani]

@ohmeow you're loading the bert-base-cased checkpoint (which is a checkpoint that was trained using a similar architecture to BertForPreTraining) in a BertForSequenceClassification model.

This means that:

• The layers that BertForPreTraining has, but BertForSequenceClassification does not have will be discarded
• The layers that BertForSequenceClassification has but BertForPreTraining does not have will be randomly initialized.

This is expected, and tells you that you won't have good performance with your BertForSequenceClassification model before you fine-tune it 🙂.

@fliptrail this warning means that during your training, you're not using the pooler in order to compute the loss. I don't know how you're finetuning your model, but if you're not using the pooler layer then there's no need to worry about that warning.

Where does the random initialization of the missing parameters occur? I don't see any calls to _init_weights.

[PremalMatalia]

@rkunani - did you get answer to this? I am also facing the same issue....

[rkunani]

@PremalMatalia I looked into it myself and found that the initialization of the nn.Linear layer on line 1469 here is where the parameters are randomly initialized (see the nn.Linear documentation).

[s4sarath]

There is something wrong. There is nothing to be randomly initia;ized, unless it is a new layer out of architecture.

On Sun, Apr 4, 2021 at 5:02 AM Raguvir Kunani @.***> wrote:

@PremalMatalia https://github.com/PremalMatalia I looked into it myself and found that the initialization of the nn.Linear layer on line 1469 here https://github.com/huggingface/transformers/blob/master/src/transformers/models/bert/modeling_bert.py is where the parameters are randomly initialized (see the nn.Linear documentation https://pytorch.org/docs/stable/generated/torch.nn.Linear.html).

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/huggingface/transformers/issues/5421#issuecomment-812940724, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACRE6KHJ4RZMPNOT6KWU7HTTG6QPDANCNFSM4OM5S2SQ .

[TingNLP]

Hi, is there any solution? I have a same problem.

the warning as below:

Some weights of the model checkpoint at bert-base-uncased were not used when initializing BertForMultiLabelSequenceClassification: ['cls.predictions.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.decoder.weight', 'cls.seq_relationship.weight', 'cls.seq_relationship.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.LayerNorm.bias']
- This IS expected if you are initializing BertForMultiLabelSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPretraining model).
- This IS NOT expected if you are initializing BertForMultiLabelSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of BertForMultiLabelSequenceClassification were not initialized from the model checkpoint at bert-base-uncased and are newly initialized: ['classifier.weight', 'classifier.bias']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.

the learner still can fit and predict, but the prediction is not consistent every time

[s4sarath]

I don't know brother. I really can't understand those warnings Because it doesn't make sense.

Check github.com/legacyai/tf-tranaformers . A new and improved version is on the way.

On Tue, Apr 13, 2021, 2:48 PM TingNLP @.***> wrote:

Hi, is there any solution? I have a same problem.

339 https://github.com/huggingface/transformers/issues/339 #18

https://github.com/huggingface/transformers/pull/18 #132 https://github.com/huggingface/transformers/issues/132 the warning as below:

Some weights of the model checkpoint at bert-base-uncased were not used when initializing BertForMultiLabelSequenceClassification: ['cls.predictions.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.decoder.weight', 'cls.seq_relationship.weight', 'cls.seq_relationship.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.LayerNorm.bias']

• This IS expected if you are initializing BertForMultiLabelSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPretraining model).
• This IS NOT expected if you are initializing BertForMultiLabelSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model). Some weights of BertForMultiLabelSequenceClassification were not initialized from the model checkpoint at bert-base-uncased and are newly initialized: ['classifier.weight', 'classifier.bias'] You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/huggingface/transformers/issues/5421#issuecomment-818586586, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACRE6KEATEJTCCLXBKPFVZDTIQD7ZANCNFSM4OM5S2SQ .

[rkunani]

All of the BertForXXX models consist of a BERT model followed by some head which is task-specific. For sequence classification tasks, the head is just a linear layer which maps the BERT transformer hidden state vector to a vector of length num_labels, where num_labels is the number of classes for your classification task (for example, positive/negative sentiment analysis has 2 labels). If you're familiar with logits, this final vector contains the logits.

In the transformers source code, you can see this linear layer (assigned to self.classifier) initialized in the constructor for BertForSequenceClassification:

class BertForSequenceClassification(BertPreTrainedModel):
    def __init__(self, config):
        super().__init__(config)
        self.num_labels = config.num_labels

        self.bert = BertModel(config)
        self.dropout = nn.Dropout(config.hidden_dropout_prob)
        self.classifier = nn.Linear(config.hidden_size, config.num_labels)

        self.init_weights()

Since self.classifier is not part of the pre-trained BERT model, its parameters must be initialized randomly (done automatically by the nn.Linear constructor).

@s4sarath Anytime you use code like model = BertForSequenceClassification.from_pretrained("bert-base-cased"), the self.classifier linear layer will have to be initialized randomly.

@TingNLP You are getting different predictions each time because each time you instantiate the model using .from_pretrained(), the self.classifier parameters will be different.

[s4sarath]

Absolutely agree .

Task specific heads has to be randomly initialised. Because, it is not a part of official Bert Model.

I agree with that.

On Tue, Apr 13, 2021, 5:49 PM Raguvir Kunani @.***> wrote:

All of the BertForXXX models consist of a BERT model https://huggingface.co/transformers/model_doc/bert.html#bertmodel followed by some head which is task-specific. For sequence classification tasks, the head is just a linear layer which maps the BERT transformer hidden state vector to a vector of length num_labels, where num_labels is the number of classes for your classification task (for example, positive/negative sentiment analysis has 2 labels). If you're familiar with logits, this final vector contains the logits.

In the transformers source code, you can see this linear layer (assigned to self.classifier) initialized in the constructor https://huggingface.co/transformers/_modules/transformers/models/bert/modeling_bert.html#BertForSequenceClassification for BertForSequenceClassification:

class BertForSequenceClassification(BertPreTrainedModel): def init(self, config): super().init(config) self.num_labels = config.num_labels

    self.bert = BertModel(config)
    self.dropout = nn.Dropout(config.hidden_dropout_prob)
    self.classifier = nn.Linear(config.hidden_size, config.num_labels)

    self.init_weights()

Since self.classifier is not part of the pre-trained BERT model, its parameters must be initialized randomly (done automatically by the nn.Linear constructor).

@s4sarath https://github.com/s4sarath Anytime you use code like model = BertForSequenceClassification.from_pretrained("bert-base-cased"), the self.classifier linear layer will have to be initialized randomly.

@TingNLP https://github.com/TingNLP You are getting different predictions each time because each time you instantiate the model using .from_pretrained(), the self.classifier parameters will be different.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/huggingface/transformers/issues/5421#issuecomment-818690286, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACRE6KH7VKR7VKJFIDZC33LTIQZDXANCNFSM4OM5S2SQ .

[TingNLP]

OK... So... the problem is the parameters. Is it possible for us to fix the value?

I think if it can be fixed, the prediction will not be inconsistent every time.

[s4sarath]

there is no point doing that right.

because once the model is trained we will be having fixed set of parameters . :)

On Wed, Apr 14, 2021, 10:45 AM TingNLP @.***> wrote:

OK... So... the problem is the parameters. Is it possible for us to fix the value?

I think if it can be fixed, the prediction will not be inconsistent every time.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/huggingface/transformers/issues/5421#issuecomment-819233571, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACRE6KGVO6PVJZVFHV2TOBTTIUQHNANCNFSM4OM5S2SQ .

[TingNLP]

@s4sarath Thanks for your immediate reply I am still a little confused. If the prediction is different each time, is that still a reasonable result??

[s4sarath]

I will explain bro. Assume classification. Last classification layer is initialised randomly right now. Now, it's okay, because you haven't trained it yet.

But once you train the model and save the checkpoint, at the time of inference you are loading that checkpoint. So the prediction remains consistent.

On Wed, Apr 14, 2021, 12:11 PM TingNLP @.***> wrote:

@s4sarath https://github.com/s4sarath Thanks for your immediate reply I am still a little confused. If the prediction is different each time, is that still a reasonable result??

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/huggingface/transformers/issues/5421#issuecomment-819270472, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACRE6KESSFYPPKW2A5AYEG3TIU2JPANCNFSM4OM5S2SQ .

[Mark-Brass]

It is said that BERT is a pre-trained model. Why then, it is needed to be trained again?

[tarskiandhutch]

It does not need to be trained again to be used for a task that it was trained on: e.g., masked language modeling over a very large, general corpus of books and web text in the case of BERT. However, to perform more specific tasks like classification and question answering, such a model must be re-trained, which is called fine-tuning. Since many popular tasks fall in this latter category, it is assumed that most developers will be fine-tuning the models, and hence the developers of Huggingface included this warning message to ensure developers are aware when the model does not appear to have been fine-tuned.

See Advantages of Fine-Tuning at this tutorial: https://mccormickml.com/2019/07/22/BERT-fine-tuning/#12-installing-the-hugging-face-library

Or check out this page from the documentation: https://huggingface.co/transformers/training.html

[Mark-Brass]

Thank you. Now it is a bit more clear. I am using finBERT for sentiment analysis, and downloaded the model from the official finBERT GIT. Do I need, then, to train the model anew?

[ishandutta0098]

I am facing a similar error while creating an entity extraction model using bert-base-uncased. Here is the code for my model

import config
import torch
import transformers
import torch.nn as nn

def loss_fn(output, target, mask, num_labels):

    lfn = nn.CrossEntropyLoss()
    active_loss = mask.view(-1) == 1
    active_logits = output.view(-1, num_labels)
    active_labels = torch.where(
        active_loss,
        target.view(-1),
        torch.tensor(lfn.ignore_index).type_as(target)
    )
    loss = lfn(active_logits, active_labels)
    return loss

class EntityModel(nn.Module):
    def __init__(self, num_tag, num_pos):
        super(EntityModel, self).__init__()

        self.num_tag = num_tag
        self.num_pos = num_pos
        self.bert = transformers.BertModel.from_pretrained(config.BASE_MODEL_PATH)
        self.bert_drop_1 = nn.Dropout(p = 0.3)
        self.bert_drop_2 = nn.Dropout(p = 0.3)
        self.out_tag = nn.Linear(768, self.num_tag)
        self.out_pos = nn.Linear(768, self.num_pos)

    def forward(self, ids, mask, token_type_ids, target_pos, target_tag):
        o1, _ = self.bert(ids, 
                          attention_mask = mask,
                          token_type_ids = token_type_ids)

        bo_tag = self.bert_drop_1(o1)
        bo_pos = self.bert_drop_2(o1)

        tag = self.out_tag(bo_tag)
        pos = self.out_pos(bo_pos)

        loss_tag = loss_fn(tag, target_tag, mask, self.num_tag)
        loss_pos = loss_fn(pos, target_pos, mask, self.num_pos)

        loss = (loss_tag + loss_pos) / 2

        return tag, pos, loss 

Error Some weights of the model checkpoint at D:\Transformers\bert-entity-extraction\input\bert-base-uncased_L-12_H-768_A-12 were not used when initializing BertModel: ['cls.predictions.transform.dense.bias', 'cls.predictions.decoder.weight', 'cls.seq_relationship.weight', 'cls.predictions.transform.LayerNorm.bias', 'cls.seq_relationship.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.dense.weight', 'cls.predictions.bias']

• This IS expected if you are initializing BertModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
• This IS NOT expected if you are initializing BertModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).

How to reslove this?

[bver]

@veronica320, the pooler layer is not used when doing sequence classification, so there's nothing to be worried about.

Note that this warning is sensitive to a Transformers version used for model training vs. a version used for inference. For instance, the Roberta model finetuned with 4.9.1 expresses this warning when loading the model for RobertaForSequenceClassification inference based on ver. 4.15.0, but the model finetuned with 4.15.0 does not.

[lijiazheng99]

An interesting edge case -- when I created and fine-tuned my custom classification model BertXXXSequenceClassification inherited from BertPreTrainedModel, I found out that I can't name layers called self.beta_layer. Otherwise, I get the warning that says beta_layer is newly initialised and won't be able to load its wights and bias from saved checkpoints. Didn't know what caused this conflict, and refactoring it to self.bate_layer saved me in the end. I used ver 4.15.0.

[Birch-san]

I've been using suppressing the warning with this helper:

from transformers import CLIPTextModel, logging

class log_level:
  orig_log_level: int
  log_level: int
  def __init__(self, log_level: int):
    self.log_level = log_level
    self.orig_log_level = logging.get_verbosity()
  def __enter__(self):
    logging.set_verbosity(self.log_level)
  def __exit__(self):
    logging.set_verbosity(self.orig_log_level)

with log_level(logging.ERROR):
  text_encoder: CLIPTextModel = CLIPTextModel.from_pretrained('openai/clip-vit-large-patch14')

[marcospgp]

Coming here from Google, this was happening when I called AutoModel.from_pretrained("EleutherAI/gpt-neo-125M").

I figured out that you can get the correct model type using the pipeline API instead:

In this case, this means I could also use AutoModelForCausalLM, but not AutoModel as that generated a model of a different type.

[saranpan]

For those who want to suppress the warning for the latest transformers version, try this, hope this helps :D

import logging
logging.getLogger("transformers.modeling_utils").setLevel(logging.ERROR)

[layumi]

I guess the simple solution is to use AutoModelForMaskedLM instead of AutoModel.

from transformers import AutoModelForMaskedLM
model = AutoModelForMaskedLM.from_pretrained('deps/distilbert-base-uncased')

[Muhammad-Asad-Arshed]

This means you are not using Pooler, you need to set add_pooling_layer=True.

[Maximo-Rulli]

I am confused after reading the whole issue. Is it possible to load a custom transformer architecture using HF or not? If not, is it possible to load a custom HF architecture (and weights) from a trainer checkpoint using torch?