Closed schmidt-sebastian closed 1 day ago
Sessions lead to crashes in append-only chat apps. We use should the non-Session API as this creates a new implicit session for every message, thereby reducing the chance that we run out of tokens in the KV cache.
Sessions lead to crashes in append-only chat apps. We use should the non-Session API as this creates a new implicit session for every message, thereby reducing the chance that we run out of tokens in the KV cache.