Closed llStringll closed 3 years ago
Your scripts are very hardcoded for data processing, one needs to change each and every processor for a differently structured data, or force-structure own data to the script
Can you please specify the format of MNLI dataset that you used, based on which you wrote your preprocessors in tasks.py
Hi there,
Your scripts are very hardcoded for data processing, one needs to change each and every processor for a differently structured data, or force-structure own data to the script
We did not design the task processors as "general purpose" solutions for different kinds of tasks/formats. Instead, they are supposed to provide a simple interface for implementing your own logic for your own data formats (as shown in this example): All you need to do is to define the get_train_examples
, get_dev_examples
, get_test_examples
, get_unlabeled_examples
and get_labels
methods, where the first four are supposed to return a list of InputExample
s (as defined here) and the last one returns a list of string labels.
The MNLI dataset that we used for our task processor is a tab-separated file with the keys (in that order) index
, promptID
, pairID
, genre
, sentence1_binary_parse
, sentence2_binary_parse
, sentence1_parse
, sentence2_parse
, sentence1
, sentence2
, label1
, ..., labeln
, gold_label
. However, the only information that we actually make use of in the task processor (see here) are the columns 0 (index
), 8 (sentence1
), 9 (sentence2
) and -1 (gold_label
).
For example, the first two lines of the dev (matched) set are:
index promptID pairID genre sentence1_binary_parse sentence2_binary_parse sentence1_parse sentence2_parse sentence1 sentence2 label1 label2 label3 label4 label5 gold_label
0 63735 63735n slate ( ( The ( new rights ) ) ( are ( nice enough ) ) ) ( Everyone ( really ( likes ( the ( newest benefits ) ) ) ) ) (ROOT (S (NP (DT The) (JJ new) (NNS rights)) (VP (VBP are) (ADJP (JJ nice) (RB enough))))) (ROOT (S (NP (NN Everyone)) (VP (ADVP (RB really)) (VBZ likes) (NP (DT the) (JJS newest) (NNS benefits))))) The new rights are nice enough Everyone really likes the newest benefits neutral entailment neutral neutral neutral neutral
You can either modify your own files to match that format or simply implement your own task processor that works with your input format. Let me know if you need any further help for that!
Yes, thank you for the detailed answer, Cheers.
Gives key error in labels, as in the MNLI train dataset, there are no labels.
Line 41, pet/tasks.py function _create_examples in MnliProcessor is where it creates examples, and assigns labels as line[-1], which is an empty string for training data of MNLI dataset, In MNLI training dataset, each line ends with: ....sentence b. ', '31193', '31193n', 'government', 'neutral', '', '', '', ''] those 4 empty strings at the end
2021-05-18 08:52:37,308 - INFO - wrapper - Writing example 0 Traceback (most recent call last): File "cli.py", line 282, in
main()
File "cli.py", line 263, in main
no_distillation=args.no_distillation, seed=args.seed)
File "/content/gdrive/My Drive/pet/pet/pet/modeling.py", line 249, in train_pet
save_unlabeled_logits=not no_distillation, seed=seed)
File "/content/gdrive/My Drive/pet/pet/pet/modeling.py", line 355, in train_pet_ensemble
unlabeled_data=unlabeled_data))
File "/content/gdrive/My Drive/pet/pet/pet/modeling.py", line 434, in train_single_model
results_dict['train_set_before_training'] = evaluate(model, train_data, eval_config)['scores']['acc']
File "/content/gdrive/My Drive/pet/pet/pet/modeling.py", line 490, in evaluate
n_gpu=config.n_gpu, decoding_strategy=config.decoding_strategy, priming=config.priming)
File "/content/gdrive/My Drive/pet/pet/pet/wrapper.py", line 354, in eval
eval_dataset = self._generate_dataset(eval_data, priming=priming)
File "/content/gdrive/My Drive/pet/pet/pet/wrapper.py", line 401, in _generate_dataset
features = self._convert_examples_to_features(data, labelled=labelled, priming=priming)
File "/content/gdrive/My Drive/pet/pet/pet/wrapper.py", line 426, in _convert_examples_to_features
input_features = self.preprocessor.get_input_features(example, labelled=labelled, priming=priming)
File "/content/gdrive/My Drive/pet/pet/pet/preprocessor.py", line 83, in get_input_features
label = self.label_map[example.label] if example.label is not None else -100
KeyError: ''
If you follow the traceback, you can find that the code is trying to run eval on the train set, which is unlabeled as I mentioned above.
How to train on MNLI dataset.