Offline Chat does not work if khoj starts offline

[meonkeys]

I install and start khoj in a Docker container with full network access. I persist the container's /root folder on the Docker host using a volume. I enable Offline Chat and it downloads llama, creating /root/.cache/gpt4all/llama-2-7b-chat.ggmlv3.q4_K_S.bin. The Chat feature works, and guessing by the CPU usage, it is running locally.

To confirm I'm offline, I turn off my network. Chat still works. Yay!

Now I try destroying the Docker container and creating a new one, again mounting the data preserved in /root. Now the Chat page says I need to set my OpenAI key or enable a local LLM. The Settings page indicates Offline Chat is still enabled, though. Why did Chat stop working? Is data persisted somewhere other than /root? Is there some check that happens when khoj starts up that prevents it from trying offline Chat?

Does this happen for non-Docker installs, too?

If I then destroy the container, turn on my network, and start a new container (same image, same /root data mounted) Chat works again. No changes to config, I don't have to re-download llama. The only change is whether or not network is available on startup.

Here's my (original, work-in-progress, non-optimized) Dockerfile:

FROM python:3

WORKDIR /usr/src/app

COPY requirements.txt ./

RUN pip install --upgrade pip
RUN apt-get update
RUN apt-get --quiet --assume-yes --no-install-recommends install build-essential
RUN pip install --no-cache-dir --requirement requirements.txt

EXPOSE 42110

CMD ["khoj", "--host", "0.0.0.0"]

Here's requirements.txt:

khoj-assistant

I'm building my Docker image like so:

docker build -t offline-ai-chat .

I'm starting my container like so:

docker run --rm -it --name khoj -v $(pwd)/volumes/root:/root -p 42110:42110 offline-ai-chat

I'm destroying the container with Ctrl-c.

Here's /root/.khoj/khoj.yml:

app:
  should-log-telemetry: false
content-type:
  github: null
  image: null
  markdown: null
  notion: null
  org: null
  pdf: null
  plaintext: null
  plugins: null
processor:
  conversation:
    conversation-logfile: /root/.khoj/processor/conversation/conversation_logs.json
    enable-offline-chat: true
    openai: null
search-type:
  asymmetric:
    cross-encoder: cross-encoder/ms-marco-MiniLM-L-6-v2
    encoder: sentence-transformers/multi-qa-MiniLM-L6-cos-v1
    encoder-type: null
    model-directory: /root/.khoj/search/asymmetric
  image:
    encoder: sentence-transformers/clip-ViT-B-32
    encoder-type: null
    model-directory: /root/.khoj/search/image
  symmetric:
    cross-encoder: cross-encoder/ms-marco-MiniLM-L-6-v2
    encoder: sentence-transformers/all-MiniLM-L6-v2
    encoder-type: null
    model-directory: ~/.khoj/search/symmetric
version: 0.10.1

[sabaimran]

Thanks for the detailed issue, @meonkeys, and for testing this out! I was actually able to reproduce the error in my local machine without docker as well. This has to do with the way GPT4All loads models. Here it goes out to see supported models and downloads one that it finds matching the spec. In our case, we don't actually need this; we should be able to configure it using the allow_download parameter. I'll push a fix for the issue.

[sabaimran]

This should be addressed with https://github.com/khoj-ai/khoj/commit/f9e09ba49006cdfabbdec9b24459931da78ffc67

[sabaimran]

If you want to try out the fix, get the latest image with docker image pull ghcr.io/khoj-ai/khoj:latest. Let me know if there are any other issues!

[meonkeys]

Awesome! Works for me.