Performed minor adoptions of the LAMA templates and evaluation procedure to work with Causal LM instead of a masked LM as discussed on the Slack.
The quality in comparison to masked LM is much worse.
Model: GPT 2
Runtime: 75min on CPU
Results: 6.98% Precision
Performed minor adoptions of the LAMA templates and evaluation procedure to work with Causal LM instead of a masked LM as discussed on the Slack. The quality in comparison to masked LM is much worse.
Model: GPT 2 Runtime: 75min on CPU Results: 6.98% Precision