Open THEWHITEBOY503 opened 1 year ago
I can get it to work if I turn the context below 2048. I also updated KoboldAI to the latest version. But, it works super slow (much slower than I used to be getting). When I run a prompt, I get this error before my prompt. I think it has something to do with my DSA error, since it talks about the index being out of bounds.
Token indices sequence length is longer than the specified maximum sequence length for this model (1575 > 1024). Running this sequence through the model will result in indexing errors
The last error is normal for some architectures such as GPT-Neo and GPT-J based models. They use the gpt2 tokenizer which only supports up to 1024 tokens while the model supports higher context. So it warns you that its applying a workaround but its fine.
Most models do not support more than 2048 tokens.
The last error is normal for some architectures such as GPT-Neo and GPT-J based models. They use the gpt2 tokenizer which only supports up to 1024 tokens while the model supports higher context. So it warns you that its applying a workaround but its fine.
Huh, interesting.
Most models do not support more than 2048 tokens.
Previously I was running like 2300 tokens context and IIRC generating around 20 tokens/second. Now I have to keep it below 2048 and I'm getting 2 tokens/second or less, otherwise it throws an error around 75% in.
I just got an RTX 3060 today and have been playing with KoboldAI all day. At some point, I attempted to overclock my GPU using MSI Afterburner with reasonable settings, and now every time I try and generate, I get this error:
I reset my Afterburner to default settings and disabled it on Windows startup, updated my GPU drivers, and even reinstalled KoboldAI dependancies, and rebooted twice, but have had no luck.