search

coleam00 / bolt.new-any-llm

Prompt, run, edit, and deploy full-stack web applications using any LLM you want!
https://bolt.new
MIT License
3.85k stars 1.58k forks source link

Created DEFAULT_NUM_CTX VAR with a default of 32768 #328

Closed aaronbolton closed 5 hours ago

aaronbolton commented 3 days ago

Adding DEFAULT_NUM_CTX to enable easier adjust on context sizes

DEFAULT_NUM_CTX VAR set to a default of 32768 but can be adjusted in .env.local, also adding some example values to .env.example

Example Context Values for qwen2.5-coder:32b

32768 # Consumes 36GB of VRAM 24576 # Consumes 32GB of VRAM 12288 # Consumes 26GB of VRAM 6144 # Consumes 24GB of VRAM

chrismahoney commented 3 days ago

Keeping an eye on the community thread for this, as long as it keeps working well for y'all I'll merge tomorrow. Doesn't have any negative impact if this new env var is missing, just continues defaulting to 32768 num_ctx. Cheers!