timoschick / pet

This repository contains the code for "Exploiting Cloze Questions for Few-Shot Text Classification and Natural Language Inference"
https://arxiv.org/abs/2001.07676
Apache License 2.0
1.62k stars 285 forks source link

Random seed parameter for iterations #90

Open MaviccPRP opened 2 years ago

MaviccPRP commented 2 years ago

Hi @timoschick ,

thank you very much for pushing your amazing PET tool to this repo. I am currently trying to investigate, how the random seed works, during different iteration per pattern.

If I understand correctly, PET trains and evaluates for each pattern n=repetitions independent models (stored in the folders final/p0-iY). This is done, to calculate later std deviation for the results.

You are describing that in your paper, too: "each model is trained three times using different seeds and average results are reported" https://arxiv.org/pdf/2001.07676v3.pdf

This would happend in pet/modeling.py in line: 326 and 327

    set_seed(seed)

    for pattern_id in pattern_ids:
        for iteration in range(repetitions):

Do I interpret this correctly? I am currently struggeling to understand how the different seeds for the model initialization per iteration is handled. We are setting a single seed in line 324. Are the models initialized with diferent weights per iteration? Are there other seeds set?

Thank youfso much in advance for your help!