search

hyperonym / basaran

Basaran is an open-source alternative to the OpenAI text completion API. It provides a compatible streaming API for your Hugging Face Transformers-based text generation models.
MIT License
1.29k stars 81 forks source link

Docker run runs and then exits, does not set up server #183

Open handrew opened 1 year ago

handrew commented 1 year ago

I'm running

docker run -p 80:80 -e MODEL=user/repo hyperonym/basaran:0.14.1

where user/repo is a Hugging Face repo. It then appears to download the model, but then once it finishes it just exits the process.

I'm using 0.14.1 because it looks like that's the only one that supports arm64 chips.

peakji commented 1 year ago

Hi @handrew ! This is a bit awkward. 😅

In https://github.com/hyperonym/basaran/pull/111, we added support for ARM architecture, but due to the resource limitations of GitHub Action runners, we had to use some hacks (https://github.com/hyperonym/basaran/pull/114, https://github.com/hyperonym/basaran/pull/131) to squeeze out more build space.

Unfortunately, these hacks soon became invalid (https://github.com/jlumbroso/free-disk-space/issues/4), so support for ARM architecture was cancelled...

We are also looking for a new solution.

bigtiger commented 1 year ago

Understandable. I'm reading this while Downloading model.safetensors fails to reach more than 65%. 😅 Thanks for your efforts.