metric is uninitialized when csv data is supplied to example/pytorch/text-classification/run_glue_no_trainer.py

[daraghhartnett]

Environment info

• transformers version: 4.5.1
• Platform: Darwin-18.7.0-x86_64-i386-64bit
• Python version: 3.7.0
• PyTorch version (GPU?): 1.8.1 (False)
• Tensorflow version (GPU?): not installed (NA)
• Using GPU in script?: No
• Using distributed or parallel set-up in script?: distributed
• OS type and version: Mac OSX 10.14.6

Who can help

• maintained examples (not research project or legacy): @sgugger, @patil-suraj

Information

Model I am using (Bert, XLNet ...): distilbert-base-uncased

The problem arises when using:

• [ x] the official example scripts: (give details below) Running the script: transformers/examples/pytorch/text-classification/run_glue_no_trainer.py With parameters: --model_name_or_path distilbert-base-uncased --per_device_train_batch_size 32 --learning_rate 2e-5 --num_train_epochs 3 --train_file piracy_train.csv --validation_file piracy_validation.csv --output_dir /data/output/distilbert-base-uncased-piracy-no-trainer

Yields the error:

` Some weights of the model checkpoint at distilbert-base-uncased were not used when initializing DistilBertForSequenceClassification: ['vocab_transform.weight', 'vocab_transform.bias', 'vocab_layer_norm.weight', 'vocab_layer_norm.bias', 'vocab_projector.weight', 'vocab_projector.bias']

• This IS expected if you are initializing DistilBertForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
• This IS NOT expected if you are initializing DistilBertForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model). Some weights of DistilBertForSequenceClassification were not initialized from the model checkpoint at distilbert-base-uncased and are newly initialized: ['pre_classifier.weight', 'pre_classifier.bias', 'classifier.weight', 'classifier.bias'] You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference. 100%|██████████| 2/2 [00:00<00:00, 12.75ba/s] 100%|██████████| 1/1 [00:00<00:00, 45.35ba/s] 04/22/2021 14:47:33 - INFO - main - Sample 598 of the training set: {'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], 'input_ids': [101, 6084, 1997, 16298, 1024, 2006, 5641, 2233, 2418, 2012, 5511, 19961, 11396, 1999, 2597, 2410, 1011, 4720, 2078, 1011, 28714, 1011, 2322, 2063, 2019, 19842, 2988, 2048, 10756, 19801, 2018, 7333, 2176, 8301, 21807, 2008, 5411, 1996, 19842, 2000, 2306, 1015, 5830, 1012, 27120, 4273, 3036, 2136, 3662, 4255, 1998, 8301, 21807, 11672, 1012, 6258, 2003, 3647, 1012, 102], 'labels': 1}. 04/22/2021 14:47:33 - INFO - main - Sample 65 of the training set: {'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], 'input_ids': [101, 2105, 2260, 2078, 2213, 22064, 1997, 11937, 9148, 1011, 11937, 9148, 2479, 5137, 1012, 2048, 10027, 2152, 1011, 3177, 6242, 5411, 1037, 9625, 6839, 14128, 1012, 3040, 2992, 1996, 8598, 3626, 21900, 1998, 8878, 2811, 28405, 2543, 21290, 2015, 1012, 5137, 3212, 2038, 2042, 11925, 2011, 27527, 2557, 1012, 1996, 10027, 6242, 5411, 2000, 1037, 3292, 1997, 3156, 5563, 2013, 1996, 2911, 1998, 2333, 2185, 1012, 1996, 2911, 7943, 2014, 6019, 2000, 1996, 2279, 3417, 1997, 7688, 1012, 102], 'labels': 1}. 04/22/2021 14:47:33 - INFO - main - Sample 877 of the training set: {'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], 'input_ids': [101, 7387, 1024, 2006, 2570, 2254, 1037, 9625, 6839, 2988, 2108, 2628, 2379, 2597, 5709, 1011, 2321, 2078, 4002, 2509, 1011, 2410, 2063, 3155, 3963, 13221, 2148, 1997, 16738, 1012, 1996, 2911, 2001, 7283, 2628, 2012, 1037, 3292, 1997, 1021, 2661, 1998, 2439, 1996, 10027, 6258, 2044, 1037, 3177, 3623, 1998, 2607, 2689, 1012, 102], 'labels': 1}. 04/22/2021 14:47:33 - INFO - main - Running training 04/22/2021 14:47:33 - INFO - main - Num examples = 1173 04/22/2021 14:47:33 - INFO - main - Num Epochs = 3 04/22/2021 14:47:33 - INFO - main - Instantaneous batch size per device = 32 04/22/2021 14:47:33 - INFO - main - Total train batch size (w. parallel, distributed & accumulation) = 32 04/22/2021 14:47:33 - INFO - main - Gradient Accumulation steps = 1 04/22/2021 14:47:33 - INFO - main - Total optimization steps = 111 33%|███▎ | 37/111 [07:48<14:24, 11.69s/it]Traceback (most recent call last): File "/Users/daraghhartnett/Projects/D3M/neural_text/code/transformers/examples/pytorch/text-classification/run_glue_no_trainer.py", line 441, in main() File "/Users/daraghhartnett/Projects/D3M/neural_text/code/transformers/examples/pytorch/text-classification/run_glue_no_trainer.py", line 406, in main metric.add_batch( UnboundLocalError: local variable 'metric' referenced before assignment 33%|███▎ | 37/111 [07:49<15:39, 12.69s/it] `

The tasks I am working on is:

• [ ] an official GLUE/SQUaD task: (give the name)
• [x ] my own task or dataset: (give details below)

Simple single sentence text classification

To reproduce

Steps to reproduce the behavior:

1. Pick any csv dataset with a train and validation files and run the transformers/examples/pytorch/text-classification/run_glue_no_trainer.py script using the following parameters:
2. --model_name_or_path distilbert-base-uncased --per_device_train_batch_size 32 --learning_rate 2e-5 --num_train_epochs 3 --train_file piracy_train.csv --validation_file piracy_validation.csv --output_dir /data/output/distilbert-base-uncased-piracy-no-trainer
3. This will yield an error as the metric variable is not initialized when an optional args.task_name is not specified.

` Some weights of the model checkpoint at distilbert-base-uncased were not used when initializing DistilBertForSequenceClassification: ['vocab_transform.weight', 'vocab_transform.bias', 'vocab_layer_norm.weight', 'vocab_layer_norm.bias', 'vocab_projector.weight', 'vocab_projector.bias']

• This IS expected if you are initializing DistilBertForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
• This IS NOT expected if you are initializing DistilBertForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model). Some weights of DistilBertForSequenceClassification were not initialized from the model checkpoint at distilbert-base-uncased and are newly initialized: ['pre_classifier.weight', 'pre_classifier.bias', 'classifier.weight', 'classifier.bias'] You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference. 100%|██████████| 2/2 [00:00<00:00, 12.75ba/s] 100%|██████████| 1/1 [00:00<00:00, 45.35ba/s] 04/22/2021 14:47:33 - INFO - main - Sample 598 of the training set: {'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], 'input_ids': [101, 6084, 1997, 16298, 1024, 2006, 5641, 2233, 2418, 2012, 5511, 19961, 11396, 1999, 2597, 2410, 1011, 4720, 2078, 1011, 28714, 1011, 2322, 2063, 2019, 19842, 2988, 2048, 10756, 19801, 2018, 7333, 2176, 8301, 21807, 2008, 5411, 1996, 19842, 2000, 2306, 1015, 5830, 1012, 27120, 4273, 3036, 2136, 3662, 4255, 1998, 8301, 21807, 11672, 1012, 6258, 2003, 3647, 1012, 102], 'labels': 1}. 04/22/2021 14:47:33 - INFO - main - Sample 65 of the training set: {'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], 'input_ids': [101, 2105, 2260, 2078, 2213, 22064, 1997, 11937, 9148, 1011, 11937, 9148, 2479, 5137, 1012, 2048, 10027, 2152, 1011, 3177, 6242, 5411, 1037, 9625, 6839, 14128, 1012, 3040, 2992, 1996, 8598, 3626, 21900, 1998, 8878, 2811, 28405, 2543, 21290, 2015, 1012, 5137, 3212, 2038, 2042, 11925, 2011, 27527, 2557, 1012, 1996, 10027, 6242, 5411, 2000, 1037, 3292, 1997, 3156, 5563, 2013, 1996, 2911, 1998, 2333, 2185, 1012, 1996, 2911, 7943, 2014, 6019, 2000, 1996, 2279, 3417, 1997, 7688, 1012, 102], 'labels': 1}. 04/22/2021 14:47:33 - INFO - main - Sample 877 of the training set: {'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], 'input_ids': [101, 7387, 1024, 2006, 2570, 2254, 1037, 9625, 6839, 2988, 2108, 2628, 2379, 2597, 5709, 1011, 2321, 2078, 4002, 2509, 1011, 2410, 2063, 3155, 3963, 13221, 2148, 1997, 16738, 1012, 1996, 2911, 2001, 7283, 2628, 2012, 1037, 3292, 1997, 1021, 2661, 1998, 2439, 1996, 10027, 6258, 2044, 1037, 3177, 3623, 1998, 2607, 2689, 1012, 102], 'labels': 1}. 04/22/2021 14:47:33 - INFO - main - Running training 04/22/2021 14:47:33 - INFO - main - Num examples = 1173 04/22/2021 14:47:33 - INFO - main - Num Epochs = 3 04/22/2021 14:47:33 - INFO - main - Instantaneous batch size per device = 32 04/22/2021 14:47:33 - INFO - main - Total train batch size (w. parallel, distributed & accumulation) = 32 04/22/2021 14:47:33 - INFO - main - Gradient Accumulation steps = 1 04/22/2021 14:47:33 - INFO - main - Total optimization steps = 111 33%|███▎ | 37/111 [07:48<14:24, 11.69s/it]Traceback (most recent call last): File "/Users/daraghhartnett/Projects/D3M/neural_text/code/transformers/examples/pytorch/text-classification/run_glue_no_trainer.py", line 441, in main() File "/Users/daraghhartnett/Projects/D3M/neural_text/code/transformers/examples/pytorch/text-classification/run_glue_no_trainer.py", line 406, in main metric.add_batch( UnboundLocalError: local variable 'metric' referenced before assignment 33%|███▎ | 37/111 [07:49<15:39, 12.69s/it] `

Expected behavior

Since providing your own csv files is allowed, the metric object should be initialized when no task_name is provided.

[sgugger]

Thanks for flagging. The PR mentioned above will make it default to accuracy. Of course, you're free to change it to whatever you need!

[daraghhartnett]

Glad to be of help! Ah, I did not see that - I only looked in the Issues to see if it had already been reported. I will check the PR's as well the next time. Thanks very much!

[sgugger]

The PR did not exist before you flagged the issue ;-) I opened it to fix it!

[daraghhartnett]

Ah! Excellent! I am using the PR you proposed locally so I am back in business :)