updates to support open-source private self-hosting

[vercel[bot]]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Updated (UTC)
otto-mvp	✅ Ready (Inspect)	Visit Preview	Mar 19, 2024 11:13pm

[kleneway]

WIP to support a fully open-source, private, locally-hosted version of JACoB

• [x] Updated README to provide instructions on how to set up the private local version
• [x] Removed dependency on AWS (gpt-4-vision will safely fall back to the default model)
• [x] Removed dependency on hosted Portkey (portkey has an open-source local proxy - https://github.com/portkey-ai/gateway)
• [ ] Introduce DEFAULT_LLM_MODEL env var to switch between gpt-4-turbo and ollama (or other models)
• [ ] Support running local models via Ollama on local portkey (should just be able to set the model name to ollama and portkey should work)
• [ ] Add portkey to docker compose (https://github.com/Portkey-AI/gateway/blob/main/docs/installation-deployments.md#deploy-using-docker)

@cpirich FYI here's the approach I was thinking of for supporting the scenario of a developer who wants to avoid using any hosted services like AWS or OpenAI, and run everything locally. Since we already have Portkey set up for the hosted version, my thought was to use their new open-source local proxy which has support for running 100+ models including any local models running on Ollama. I got pretty close with this but ran into a few issues with the last few items on the list. I likely won't be able to finish this before I head out for Indy tomorrow so I wanted to set a checkpoint here.

Let me know if you think this approach could work or if you have other ideas.

[cpirich]

@jacob-ai-bot code review

[jacob-ai-bot[bot]]

JACoB here...

I'm starting a code review on this PR.

I'll continue to comment on this pull request with status as I make progress.