search

amaiya / onprem

A tool for running on-premises large language models with non-public data
https://amaiya.github.io/onprem
Apache License 2.0
684 stars 32 forks source link

Number of tokens #38

Closed lysa324 closed 10 months ago

lysa324 commented 10 months ago

what is the maximum number of tokens the framework can accept as model inputs? does this number depend on the WizardLM model or OnPrem.LM?

amaiya commented 10 months ago

This depends on the model you use with OnPrem.LLM. The default context window size is set to n_ctx=2048 in OnPrem.LLM, but you can increase it if you use models that accept larger context sizes.

lysa324 commented 10 months ago

good morning, thank you for answering! i would like to understand how the model has the ability to read multiple pdf's from the sample_data folder when the maximum accepted token is 2048?

thanks

amaiya commented 10 months ago

See the Retrieval Augmented Generation (RAG) paper for details.