search

open-flux-ai / open-flux-ai

Apache License 2.0
40 stars 7 forks source link

Problem running `open-flux-ai` #6

Open Z80coder opened 9 months ago

Z80coder commented 9 months ago

Hi,

I might be missing something obvious, but after I successfully execute:

ollama run chanwit/flux-7b:v0.2

And then type:

>>> tell me about SOPS.

I get the error:

Error: Post "http://127.0.0.1:11434/api/generate": EOF

Any help appreciated!

chanwit commented 9 months ago

Hi @Z80coder

Thank you for trying it. It seems you are running into an Ollama problem, probably the ollama serve or ollama-runner process is not running for some reasons.

Could you ps aux | grep ollama to double check please?

If you still have a problem with Ollama, I can prepare the model in other formats like Llamafile or maybe Docker image for you to try.

Z80coder commented 9 months ago

I haven't used ollama before. So I might be mis-using it. When I type

ps aux | grep ollama

I see:

xxxx@xxxx:~/repos/open-flux-ai$ ps aux | grep ollama
ollama    782213  0.0  0.3 2373524 404904 ?      Ssl  15:14   0:00 /usr/local/bin/ollama serve
xxxx    783855  0.0  0.0   9120  2304 pts/0    S+   15:29   0:00 grep --color=auto ollama
chanwit commented 9 months ago

So the ollama serve is running. Could you run ollama pull chanwit/flux-7b:v0.2 to explicitly pull the model before running please? If the command worked, you would be able list it with the list command:

❯ ollama list
NAME                    ID              SIZE    MODIFIED
chanwit/flux-7b:latest  7b103ada79f4    5.1 GB  3 days ago
chanwit/flux-7b:v0.2    108f27a65749    4.4 GB  2 minutes ago
chanwit/kube-7b:latest  bf8635f2eb40    5.1 GB  9 days ago
chanwit/kube-7b:v0.1    bf8635f2eb40    5.1 GB  9 days ago
Modelfile:latest        1e0105e46b85    5.1 GB  9 days ago
tinyllama:latest        2644915ede35    637 MB  2 weeks ago
zephyr:latest           bbe38b81adec    4.1 GB  9 days ago

Then please try running the model again with ollama run chanwit/flux-7b:v0.2.

Z80coder commented 9 months ago

I explicitly pulled the model. ollama list gives

NAME                    ID              SIZE    MODIFIED       
chanwit/flux-7b:latest  a7887c593dfb    5.1 GB  37 minutes ago  
chanwit/flux-7b:v0.2    108f27a65749    4.4 GB  10 seconds ago

But I get the same error.

Do I need a GPU of sufficient size?

chanwit commented 9 months ago

Nop, my tests were solely on CPU. The model size is 5GB so it requires ~6GB of RAM to fully load.

BTW, which OS you're using?

chanwit commented 9 months ago

Running a smaller model might help test your Ollama setup. TinyLlama is 1.1B parameters which would requires only 1.2-1.3GB of RAM to run. (It's going to say some non-sense things anyway)

ollama run tinyllama:chat
Z80coder commented 9 months ago

ollama run tinyllama:chat works!

I have plenty of RAM on my machine. My OS is Ubuntu 23.10.

chanwit commented 9 months ago

Now I'm guessing that the model file is corrupted. Will dig more into Ollama to find a debug flag.

Not sure if you would be able to run other versions of the image? Could you please try chanwit/flux-7b:v0.1 or chanwit/flux-7b:v0.3 (WIP) to see if any of these would work for you.

Z80coder commented 9 months ago

Both these models generate the same error. Thanks for investigating!

chanwit commented 9 months ago

Thank you for trying @Z80coder I'll also prepare other ways to run the model in the mean time too.

chanwit commented 8 months ago

I'm planning to drop Ollama support very soon because it's broken after I updated it to the newest version.

A new inference engine will be compiled directly from llama.cpp or maybe using llama-cpp-python.