Gracefully handle unknown context size for local llms - Githubissues

paul-gauthier / aider

aider is AI pair programming in your terminal

https://aider.chat/

Apache License 2.0

14.49k stars 1.36k forks source link

Gracefully handle unknown context size for local llms #441

Closed paul-gauthier closed 2 months ago

paul-gauthier commented 6 months ago

https://discord.com/channels/1131200896827654144/1133060780649087048/1194811273595068436

when I try to set openai compatible endpoint using cmdline args: openai-api-key: XXXX openai-api-base: https://api.deepinfra.com/v1/openai model: cognitivecomputations/dolphin-2.6-mixtral-8x7b

I get ValueError: Unknown context window size for model: cognitivecomputations/dolphin-2.6-mixtral-8x7b

We should probably not throw an error and just accept an unknown context size.

paul-gauthier commented 2 months ago

Aider handles this gracefully now.

I'm going to close this issue for now, but feel free to add a comment here and I will re-open or file a new issue any time.