search

Codium-ai / AlphaCodium

Official implementation for the paper: "Code Generation with AlphaCodium: From Prompt Engineering to Flow Engineering""
https://www.codium.ai
GNU Affero General Public License v3.0
3.41k stars 252 forks source link

Recommended approach for local models? i.e. Swappable model support. #37

Open illuminatedjavascript opened 7 months ago

illuminatedjavascript commented 7 months ago

Looks like litellm does a (too) good job of encapsulating the calls to openai, making calls to local openai-api-based models require a proxy to intercept and re-route.

Is this the recommended approach for the time being? Any plans to drop the litellm dependency, use one that's a little more open, or write your own layer?

It would be nice to use this with swappable models especially since AC seems to generalize across general instruct models and not require function-calling models.

pslusarz commented 1 week ago

I added AWS Bedrock support on this MR. Not sure how good results would be on the local models, since they are kind of weak at development, but the code seems to support them, ie you can try something like model="huggingface/deepseek-ai/deepseek-coder-33b-instruct" in configuration.toml.