behaviour of ZeroShotClassification using facebook/bart-large-mnli is different on online demo vs local machine

[turmeric-blend]

Environment info

• transformers version: 3.4.0
• Platform: Ubuntu 20.04
• Python version: 3.7.7
• PyTorch version (GPU?): 1.6.0 (GPU:Yes)
• Tensorflow version (GPU?): No
• Using GPU in script?: Yes
• Using distributed or parallel set-up in script?: No

Who can help

@sshleifer

Information

Model I am using (Bert, XLNet ...): facebook/bart-large-mnli

The problem arises when using:

• [ ] the official example scripts: (give details below)
• [X] my own modified scripts: (give details below)

The tasks I am working on is:

• [ ] an official GLUE/SQUaD task: (give the name)
• [X] my own task or dataset: (give details below)

To reproduce

Steps to reproduce the behavior:

First I tried the hosted demo online at huggingface, which gives me a very high score of 0.99 for travelling (as expected):

Then I tried to run the code on my local machine, which returns very different scores for all labels (poor scores):

from transformers import pipeline
from transformers import AutoTokenizer, AutoModel
tokenizer = AutoTokenizer.from_pretrained("facebook/bart-large-mnli")
model = AutoModel.from_pretrained("facebook/bart-large-mnli")

zsc = pipeline(task='zero-shot-classification', tokenizer=tokenizer, model=model)

sequences = 'one day I will see the world'
candidate_labels = ['travelling', 'cooking', 'dancing']

results = zsc(sequences=sequences, candidate_labels=candidate_labels, multi_class=False)
print(results)
>>>{'sequence': 'one day I will see the world',
'labels': ['travelling', 'dancing', 'cooking'],
'scores': [0.5285395979881287, 0.2499372661113739, 0.22152313590049744]}

I got this warning message when initializing the model: model = AutoModel.from_pretrained("facebook/bart-large-mnli")

Some weights of the model checkpoint at facebook/bart-large-mnli were not used when initializing BartModel: ['model.encoder.version', 'model.decoder.version']
- This IS expected if you are initializing BartModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPretraining model).
- This IS NOT expected if you are initializing BartModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).

Expected behavior

Code on my local machine's score to be quite similar to the online demo.

[joeddav]

Replace AutoModel with AutoModelForSequenceClassification. The former won't add the sequence classification head, i.e. it will use BartModel instead of BartForSequenceClassification, so the pipeline is trying to use just the outputs of the encoder instead of the NLI predictions in your snippet.

[turmeric-blend]

@joeddav that fixed it thanks !

[gustavengstrom]

Have the same problem:

conda environment: Python 3.7.9

pip3 install torch==1.6
pip3 install transformers

Running

from transformers import AutoModelForSequenceClassification
model = AutoModelForSequenceClassification.from_pretrained("facebook/bart-large-mnli")

Results in message:

Some weights of the model checkpoint at facebook/bart-large-mnli were not used when initializing BartForSequenceClassification: ['model.encoder.version', 'model.decoder.version']

• This IS expected if you are initializing BartForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPretraining model).
• This IS NOT expected if you are initializing BartForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).

@turmeric-blend: How is my setup different from yours?

[turmeric-blend]

actually the error message was still there after the fix, but the scores running on local machine were consistent with the online demo @gustavengstrom

any ideas why is there still the warning message @joeddav ?

[joeddav]

Yeah that warning isn't a concern. It's just letting you know that some of the parameters checkpointed in the pretrained model were not able to be matched with the model class, but in this case it's just a couple of meta-fields (encoder/decoder version), so your weights should be matched up fine.