Multi-mask PET evaluation step is very slow

[LiweiPeng]

First of all, thanks for supporting multiple token verbalizers. This feature is critical to my scenario. My verbalizers typically have 4-6 tokens.

As mentioned in https://github.com/timoschick/pet#pet-with-multiple-masks, using multi-mask PET is very very slow at evaluation step because 1) max eval batch size has to be 1; 2) Multiple forward passes needs to be run for each eval step.

My questions are: 1) Is there any plan to address the evaluation step speed issue with some new design? 2) Is there any alternative way to speed up the evaluation step?

Thanks.

[timoschick]

Hi @LiweiPeng, regarding your questions:

1. Unfortunately, we currently do not have any plans to address the evaluation step speed issue. However, as the last step of PET is to train a regular sequence classifier on data annotated by the individual models, the final model should be fast (it's just the intermediate evaluations and the generation of soft-labeled data that takes very long).
2. As an alternative, you may try using a generative language model with GenPET, but our current implementation of GenPET only supports PEGASUS, so you'd have to modify some of the code to make it work for other generative models like T5.

[LiweiPeng]

Thanks for the suggestions. I will see whether I can use less mask tokens first before I try generative models.