Loading checkpoint shards gets killed at 0%

[B0-B]

Describe the bug

Description

I installed with one click installer on Ubuntu 20.04 and I called the conda environment with

source "./installer_files/conda/etc/profile.d/conda.sh" &&
conda activate "./installer_files/env"

After I start the server with

cd ./text-generation-webui &&
python server.py

checkpoints loading gets killed at 0%

bin /home/b1/oobabooga_linux/installer_files/env/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cpu.so
/home/b1/oobabooga_linux/installer_files/env/lib/python3.10/site-packages/bitsandbytes/cextension.py:34: UserWarning: The installed version of bitsandbytes was compiled without GPU support. 8-bit optimizers, 8-bit multiplication, and GPU quantization are unavailable.
  warn("The installed version of bitsandbytes was compiled without GPU support. "
/home/b1/oobabooga_linux/installer_files/env/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cpu.so: undefined symbol: cadam32bit_grad_fp32
INFO:Loading lmsys_vicuna-7b-delta-v1.1...
Loading checkpoint shards:   0%|                                                            | 0/2 [00:00<?, ?it/s]
Killed

I have tested with GPU and CPU-only install, in both cases the process gets killed. Not sure if it is a memory problem, as there is no output.

Is there an existing issue for this?

• [X] I have searched the existing issues

Reproduction

Download the model (should take ~12GB of space)

cd ~/oobabooga_linux/text-generation-webui && 
python3 download-model.py lmsys/vicuna-7b-delta-v1.1

source "./installer_files/conda/etc/profile.d/conda.sh" &&
conda activate "./installer_files/env" &&
python server.py

bin /home/b1/oobabooga_linux/installer_files/env/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cpu.so
/home/b1/oobabooga_linux/installer_files/env/lib/python3.10/site-packages/bitsandbytes/cextension.py:34: UserWarning: The installed version of bitsandbytes was compiled without GPU support. 8-bit optimizers, 8-bit multiplication, and GPU quantization are unavailable.
  warn("The installed version of bitsandbytes was compiled without GPU support. "
/home/b1/oobabooga_linux/installer_files/env/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cpu.so: undefined symbol: cadam32bit_grad_fp32
INFO:Loading lmsys_vicuna-7b-delta-v1.1...
Loading checkpoint shards:   0%|                                                            | 0/2 [00:00<?, ?it/s]
Killed

Screenshot

No response

Logs

None outputted ...

System Info

## Specs
OS: Ubuntu 20.04
RAM: 16GB
vRAM: 8GB
CPU: Ryzen 1700x

[elbowdonkey]

I'm seeing the same issue.

I can't say for certain, but I have a feeling this happens when CPU ram is low.

I see the "Loading checkpoint shards" message hang when I try to load anything beyond facebook_opt-2.7b.

I also only have 16GB of CPU Ram.

[github-actions[bot]]

This issue has been closed due to inactivity for 30 days. If you believe it is still relevant, please leave a comment below.

[dilvan1]

I'm having the same problem.

[jccbbb]

I also have the same problem.

[bbirdxr]

I also have the same problem.

[fazlicodes]

+1

[YugwonWon]

Me too

[Redster1]

I also have this problem. When I look at my CPU, RAM, and Memory usage, it is maxxed out until it gives up and then the command prompt prompts me to press any key, at which point, it closes.

[ebaker-github]

I hope this helps someone. I think elbowdonkey is onto something with the RAM thoughts, I run Oobabooga in a Hyper-V VM. In the VM memory setting if I have Dynamic Memory enabled loading checkpoint shards fails at 0%. When I have Dynamic Memory disabled the models load fine.

[fazlicodes]

@ebaker-github How to disable dynamic memory when loading the model?

[ebaker-github]

@ebaker-github How to disable dynamic memory when loading the model?

If you are asking how to disable dynamic memory in Hyper-V:

1, Select and right-click the VM in Hyper-V Manager Virtual Machines. 2, Navigate to the Memory section. 3, Uncheck the box labeled "Enable Dynamic Memory."

That disables and greys out the Dynamic Memory section. Also note that I am running completely in CPU RAM: I do not have a GPU.

[bitshop]

Adding a swap file worked for my error, although the memory usage between RAM and swap file was larger than I had anticipated. In my case I REALLY didn't have sufficient RAM + swap when we got this failure.

[raghavared]

I also have same problem

(env) ubuntu@raghava:~/$ python app.py Loading checkpoint shards: 0%| | 0/2 [00:00<?, ?it/s] Killed

[ebaker-github]

I also have same problem

(env) ubuntu@raghava:~/$ python app.py Loading checkpoint shards: 0%| | 0/2 [00:00<?, ?it/s] Killed

After a bit of time it definitely seems to be caused by not enough RAM.

[NeevrajKB]

Same issue. How can i fix it?