Speculative Decoding Part 4: Lookahead scheduling - Githubissues

PygmalionAI / aphrodite-engine

PygmalionAI's large-scale inference engine

https://pygmalion.chat

GNU Affero General Public License v3.0

606 stars 78 forks source link

Speculative Decoding Part 4: Lookahead scheduling #402

Closed AlpinDale closed 1 month ago

AlpinDale commented 1 month ago

The dummy arg num_lookahead_slots has been added.