evaluation takes a huge amount of time

[Punchwes]

Hello,

Thanks so much for your excellent work. I am doing purely evaluation on the model. The batch size I uses is 32 and other settings remain default. By doing pure evaluation, each query(with 1000 documents) would take nearly 50 seconds to go. The evaluation process seems too slow, is there anything wrong or it is the case that it would take that long time?

Thanks in advance.

[rodrigonogueira4]

When using a BERT Large model, a TPU v2 processes ~150-250 query-doc pairs per second (or 5 seconds per query with 1000 documents), which is ~5-10x faster than a single GPU. You can use TPUs for free in this Colab: https://drive.google.com/open?id=1vaON2QlidC0rwZ8JFrdciWW68PYKb9Iu

Alternatively, you can train and evaluate a BERT Base model, which can be done ~4 times faster than the large version while being only 1-2 MAP points lower.

[rodrigonogueira4]

Closing due to inactivity. Please let me know if you have any questions.

[guotong1988]

Thank you