search

timoschick / pet

This repository contains the code for "Exploiting Cloze Questions for Few-Shot Text Classification and Natural Language Inference"
https://arxiv.org/abs/2001.07676
Apache License 2.0
1.62k stars 283 forks source link

CPU Machine and Predictions of Unlabeled Data #11

Closed Mahhos closed 4 years ago

Mahhos commented 4 years ago

Thanks for the great repo. I have got 2 questions.

  1. How many days you expect a customized task with sizes of the train: 610, dev: 500, test: 1000, unlabeled data: 2000 would take on a CPU machine? It's been four days and the code is still going :D this is the current step "learning_rate": 3.1000000000000004e-06, "loss": 0.007363705812799708, "step": 3450}

  2. To get the predictions/final_labels for unlabeled data which folder/file should we look at? I have 7 folders in my output_directory so far. p0-i0, p0-i1, p0-i2, p1-i0, p1-i0, p1-i1, p1-i2, final I designed 2 patterns (0,1) for my customized_PVP.
    There are two files in my output_directory as well: result_test.txt and unlabeled_logits.txt. The first three lines in my unlabeled_logits.txt: -1 8.052350375527427 5.843138097046414 14.612752109704642 2.8329479116033762

timoschick commented 4 years ago

Hi @Mahhos,

  1. I have never tried using PET on CPU so I have no experience with regards to the expected training time, sorry. If you can somehow get access to a single GPU with >8GB of memory, you should be able to perform training for your data in less than a day.

  2. Each folder pX-iY corresponds to the results obtained in the Yth iteration of training using pattern X. You'll probably only be interested in the results of the final, distilled model, which you will find in final/p0-i0 once training is complete. Note that by default, the final model does not compute predictions for the unlabeled data, but only for the validation (or test) data. You can simply define your get_dev_examples() method to return the same data as your get_unlabeled_examples() method if you want the final model to annotate the unlabeled data.

Mahhos commented 4 years ago

Thank you so much.

timoschick commented 4 years ago

You're welcome!

Mahhos commented 4 years ago

Having get_dev_examples() method to return the same data as the get_unlabeled_examples() method raised this error. It seems that it expects labels for dev.csv. However, the unlabeled.csv does not have labels and has , instead.

File "/pet-master/pet/preprocessor.py", line 83, in get_input_features label = self.label_map[example.label] if example.label is not None else -100 KeyError: ','

timoschick commented 4 years ago

@Mahhos: Please see my answer here: https://github.com/timoschick/pet/issues/12#issuecomment-710049248