Should we also report FLOPs (or similar)? I believe one can get this from the Trainer pretty easily (if not by default): https://huggingface.co/docs/transformers/v4.22.2/en/main_classes/trainer#transformers.Trainer.floating_point_ops
This would allow people to plot performance vs compute, which is arguably more meaningful (modulo caveats around hardware differences)
Per @lewtun: