search

facebookresearch / ParlAI

A framework for training and evaluating AI models on a variety of openly available dialogue datasets.
https://parl.ai
MIT License
10.5k stars 2.09k forks source link

BlenderBot2 is too slow #4073

Closed sheraleks closed 3 years ago

sheraleks commented 3 years ago

Hi! I am stuck with BB2 400M (default parameters) huge response time. Even when internet search is off. Is there any way to speed up response generation? Maybe there is some parameters that I could change to speed up model? Or maybe I should perform knowledge distillation? I’ve seen examples for BB1, but not sure if it is possible for BB2.

mailong25 commented 3 years ago

I'm interested as well. Btw, can you share your cmd to turn off the internet search @sheraleks ?

klshuster commented 3 years ago

I am working on adding incremental decoding, which should significantly reduce response time

sheraleks commented 3 years ago

I'm interested as well. Btw, can you share your cmd to turn off the internet search @sheraleks ?

@mailong25, yep. Just use this parameters: knowledge_access_method: memory_only, search_server: None

github-actions[bot] commented 3 years ago

This issue has not had activity in 30 days. Please feel free to reopen if you have more issues. You may apply the "never-stale" tag to prevent this from happening.