If I want to change REST code to support multi-batch inference, what needs to be changed? - Githubissues

FasterDecoding / REST

REST: Retrieval-Based Speculative Decoding, NAACL 2024

Apache License 2.0

177 stars 11 forks source link

If I want to change REST code to support multi-batch inference, what needs to be changed? #16

Open yangbohust opened 6 months ago

yangbohust commented 6 months ago

If I want to change REST code to support multi-batch inference, what needs to be changed?