Feature Request: Allow to connect to other APIs

[mkellerman]

Here is the docker compose file i'm using to deploy chatbot-ui and llama-cli api:

version: '3.6'

services:
  chatgpt:
    build: .
    # image: ghcr.io/mckaywrigley/chatbot-ui:main
    ports:
      - 9080:3000
    environment:
      - 'OPENAI_API_KEY='
      - 'OPENAI_API_HOST=http://api:8080'

  api:
    image: quay.io/go-skynet/llama-cli:v0.4
    volumes:
      - /Users/Shared/Models:/models
    ports:
      - 9000:8080
    environment:
      - MODEL_PATH=/models/7B/gpt4all-lora-quantized.bin
      - CONTEXT_SIZE=700
      - THREADS=4
    command: api

Unfortunatly, it doesn't work out of the box. And not sure if the front end should connect to 'any' API, or the api should mimic the api endpoints of openapi so it works seamlessly.

But will cross post in both repos.

[mkellerman]

Cross post in: https://github.com/go-skynet/llama-cli/issues/10

[abetlen]

Hi @mkellerman if you just want to get chatbot-ui working with a local version of Llama (or anything that llama.cpp supports) check out llama-cpp-python. I have an example FastAPI server that matches the OpenAI API that you can run. I'm working on moving it to a command but for now you should just be able to do the following to set it up:

pip install llama-cpp-python
git clone https://github.com/abetlen/llama-cpp-python.git
cd llama-cpp-python/examples/high_level_api
export MODEL=/path/to/ggml/model
pip install fastapi uvicorn sse_starlette typing_extensions
uvicorn --reload --host 0.0.0.0 fastapi_server:app
# go to http://localhost:8000/docs to ensure it's working

Then just start chatbot-ui with

docker run -e OPENAI_API_HOST=http://<host-ip-address>:8000 -e OPENAI_API_KEY="" -p 3000:3000 ghcr.io/mckaywrigley/chatbot-ui:main

You may need to tweak the docker network settings and / or the ip address but I was able to get this working.

[mkellerman]

That is BRILLIANT! Will try it out now and close the ticket if I don't have any more needs/concerns.

Cheers!

[mkellerman]

@abetlen I packaged it up into a nice docker-compose: https://github.com/mkellerman/gpt4all-ui

Let me know what you think! ;)

[dantrauner]

~Using python3 -m llama_cpp.server for llama-cpp-python did you have to modify chatbot-ui in any way? I thought Python's server listened on all interfaces but chatbot-ui is giving me:~

 [TypeError: fetch failed] {
  cause:  [Error: connect ECONNREFUSED 127.0.0.1:8000] {
  errno: -111,
  code: 'ECONNREFUSED',
  syscall: 'connect',
  address: '127.0.0.1',
  port: 8000
}

~and shows this:~

EDIT: I'm a 🤡 – needed more coffee to realize I wasn't running the API server in docker too...

[mkellerman]

Created a PR to add it directly here: #481

[N0THSA]

Any updates?