search

bigcode-project / starcoder2

Home of StarCoder2!
Apache License 2.0
1.76k stars 158 forks source link

support SPM mode for FIM prompts #14

Closed erfanium closed 6 months ago

erfanium commented 7 months ago

from fim paper (https://arxiv.org/pdf/2207.14255.pdf) section 3.1: SPM mode can be used to reuse kv cache across completion requests.

SPM modes can enable further latency optimization (which is very important in case of code completion tools). is there any reason that startcoder models are using normal PSM mode?

loubnabnl commented 6 months ago

We train with both modes (50% PSM and 50% SPM), similarily to StarCoder (cf paper). So you can also try SPM mode for inference.

image
erfanium commented 6 months ago

Got it. thanks!