search

PromtEngineer / Langchain-Tutorilas

This repo contains code for Langchain tutorials on my youtube channel.
38 stars 27 forks source link

tokens exceeded for chatbot_localGPT.ipynb #1

Open rodrigofarias-MECH opened 1 year ago

rodrigofarias-MECH commented 1 year ago

Hello my friend!

I tested Chatbots-with-memory/chatbot_localGPT.ipynb with orca-mini and it worked!

Now Im testing different models, they get loaded by the jupyter, and a common error is this: ValueError: Requested tokens exceed context window of 2048

I changed all values related to tokens in load_model, but the error persists, just the number changes.

I'm using CPU only, and all the models tested are supposedly to run in only CPU configurations. For example, I tested TheBloke/stable-vicuna-13B-GGML/ stable-vicuna-13B.ggmlv3.q4_0.bin.

I can give more information if requested.

What could be the problem? Any hints? Great work!

PromtEngineer commented 1 year ago

Okay so seems like you are running into the context window limitation issue. Most of these models have (open-source) a context window 0f 2048 tokens. That includes the chunks provided by your embedding based search, your original prompt as well as the information in the memory. My recommendation will be to reduce the number of chunks that you are returning form the semantic search as a starting point. Hope this helps.