search

timoschick / pet

This repository contains the code for "Exploiting Cloze Questions for Few-Shot Text Classification and Natural Language Inference"
https://arxiv.org/abs/2001.07676
Apache License 2.0
1.62k stars 283 forks source link

Issue in running on MNLI dataset #32

Closed llStringll closed 3 years ago

llStringll commented 3 years ago

Gives key error in labels, as in the MNLI train dataset, there are no labels.

Line 41, pet/tasks.py function _create_examples in MnliProcessor is where it creates examples, and assigns labels as line[-1], which is an empty string for training data of MNLI dataset, In MNLI training dataset, each line ends with: ....sentence b. ', '31193', '31193n', 'government', 'neutral', '', '', '', ''] those 4 empty strings at the end

2021-05-18 08:52:37,308 - INFO - wrapper - Writing example 0 Traceback (most recent call last): File "cli.py", line 282, in main() File "cli.py", line 263, in main no_distillation=args.no_distillation, seed=args.seed) File "/content/gdrive/My Drive/pet/pet/pet/modeling.py", line 249, in train_pet save_unlabeled_logits=not no_distillation, seed=seed) File "/content/gdrive/My Drive/pet/pet/pet/modeling.py", line 355, in train_pet_ensemble unlabeled_data=unlabeled_data)) File "/content/gdrive/My Drive/pet/pet/pet/modeling.py", line 434, in train_single_model results_dict['train_set_before_training'] = evaluate(model, train_data, eval_config)['scores']['acc'] File "/content/gdrive/My Drive/pet/pet/pet/modeling.py", line 490, in evaluate n_gpu=config.n_gpu, decoding_strategy=config.decoding_strategy, priming=config.priming) File "/content/gdrive/My Drive/pet/pet/pet/wrapper.py", line 354, in eval eval_dataset = self._generate_dataset(eval_data, priming=priming) File "/content/gdrive/My Drive/pet/pet/pet/wrapper.py", line 401, in _generate_dataset features = self._convert_examples_to_features(data, labelled=labelled, priming=priming) File "/content/gdrive/My Drive/pet/pet/pet/wrapper.py", line 426, in _convert_examples_to_features input_features = self.preprocessor.get_input_features(example, labelled=labelled, priming=priming) File "/content/gdrive/My Drive/pet/pet/pet/preprocessor.py", line 83, in get_input_features label = self.label_map[example.label] if example.label is not None else -100 KeyError: ''

If you follow the traceback, you can find that the code is trying to run eval on the train set, which is unlabeled as I mentioned above.

How to train on MNLI dataset.

llStringll commented 3 years ago

Your scripts are very hardcoded for data processing, one needs to change each and every processor for a differently structured data, or force-structure own data to the script

llStringll commented 3 years ago

Can you please specify the format of MNLI dataset that you used, based on which you wrote your preprocessors in tasks.py

timoschick commented 3 years ago

Hi there,

Your scripts are very hardcoded for data processing, one needs to change each and every processor for a differently structured data, or force-structure own data to the script

We did not design the task processors as "general purpose" solutions for different kinds of tasks/formats. Instead, they are supposed to provide a simple interface for implementing your own logic for your own data formats (as shown in this example): All you need to do is to define the get_train_examples, get_dev_examples, get_test_examples, get_unlabeled_examples and get_labels methods, where the first four are supposed to return a list of InputExamples (as defined here) and the last one returns a list of string labels.

The MNLI dataset that we used for our task processor is a tab-separated file with the keys (in that order) index, promptID, pairID, genre, sentence1_binary_parse, sentence2_binary_parse, sentence1_parse, sentence2_parse, sentence1, sentence2, label1, ..., labeln, gold_label. However, the only information that we actually make use of in the task processor (see here) are the columns 0 (index), 8 (sentence1), 9 (sentence2) and -1 (gold_label).

For example, the first two lines of the dev (matched) set are:

index   promptID    pairID  genre   sentence1_binary_parse  sentence2_binary_parse  sentence1_parse sentence2_parse sentence1   sentence2   label1  label2  label3  label4  label5  gold_label
0   63735   63735n  slate   ( ( The ( new rights ) ) ( are ( nice enough ) ) )  ( Everyone ( really ( likes ( the ( newest benefits ) ) ) ) )   (ROOT (S (NP (DT The) (JJ new) (NNS rights)) (VP (VBP are) (ADJP (JJ nice) (RB enough)))))  (ROOT (S (NP (NN Everyone)) (VP (ADVP (RB really)) (VBZ likes) (NP (DT the) (JJS newest) (NNS benefits))))) The new rights are nice enough  Everyone really likes the newest benefits   neutral entailment  neutral neutral neutral neutral

You can either modify your own files to match that format or simply implement your own task processor that works with your input format. Let me know if you need any further help for that!

llStringll commented 3 years ago

Yes, thank you for the detailed answer, Cheers.