ollama / ollama

Get up and running with Llama 3.2, Mistral, Gemma 2, and other large language models.
https://ollama.com
MIT License
95.91k stars 7.61k forks source link

`pulling manifest Error: EOF` when pulling after disk is full #1731

Open jmorganca opened 10 months ago

jmorganca commented 10 months ago

To reproduce, pull with little disk space lefT:

$ ollama run deepseek-coder:33b
pulling manifest 
Error: write /usr/share/ollama/.ollama/models/blobs/sha256:065b9a7416ba28634cd4efc2cd3024d4755731c1275dc0286b81b01793185fbb-partial-0: no space left on device

Even with more space, future ollama pull commands fail until Ollama restarted:

$ ollama run deepseek-coder:33b
pulling manifest 
Error: EOF
mxyng commented 9 months ago

I haven't been able to reproduce this. Ollama behaves as expected. Here is the test I ran:

  1. Create a VM with multiple, small physical volumes
  2. Create a logical volume by attaching one of the physical volumes
  3. Format, mount, and configure as OLLAMA_MODELS
  4. Pull a large model, e.g. llama2:70b
  5. Step 4 should fail once the logical volume is full
  6. Expand the disk by attaching another physical volumes to the logical volume
  7. Repeat step 4 which should resume the download where it left off

@jmorganca do you have any more details on how to reproduce this?

Pytness commented 9 months ago

I just managed to unintentionally reproduce this bug. The steps I followed:

Restarting ollama fixes the issue.

ictus4u commented 5 months ago
> ollama --version
ollama version is 0.1.34

Restarting didn't work for me.

After having a look at https://github.com/ollama/ollama/pull/344, I ended with the below command:

sudo rm /usr/share/ollama/.ollama/models/blobs/*-partial*
candera commented 3 months ago

Deleting only the ones of zero size in ~/.ollama/models/blobs/ worked for me, and I didn't have to restart the entire download.