NolanoOrg / cformers

SoTA Transformers with C-backend for fast inference on your CPU.
MIT License
311 stars 29 forks source link

Add benchmark tasks like BIG Bench #21

Open kamalojasv181 opened 1 year ago

kamalojasv181 commented 1 year ago

Whenever a new model is added, there need to be sufficient metrics that help the user estimate the drop in performance of the optimized model compared to the original model.

I suggest Big Bench. Other suggestions are welcomed.