search

rendezqueue / rendezllama

CLI for llama.cpp with various commands to guide, edit, and regenerate tokens on the fly.
ISC License
11 stars 1 forks source link

bug: rollforget with 4096 context_token_limit and 2048 batch_count violates assertion in ggml #32

Closed grencez closed 1 year ago

grencez commented 1 year ago

We seem to be counting tokens incorrectly, missing some.

I don't know what's triggering this yet.

The larger-than-default batch_count seems to be exceeding memory.

grencez commented 1 year ago

I don't think there's anything to fix here.