Process Killed with Deep Floyd

[AmanKishore]

Description

[INFO] loading DeepFloyd IF-I-XL...

A mixture of fp16 and non-fp16 filenames will be loaded.
Loaded fp16 filenames:
[safety_checker/pytorch_model.fp16.bin, text_encoder/pytorch_model.fp16-00001-of-00002.bin, text_encoder/pytorch_model.fp16-00002-of-00002.bin, unet/diffusion_pytorch_model.fp16.bin]
Loaded non-fp16 filenames:
[watermarker/diffusion_pytorch_model.bin
If this behavior is not expected, please check your folder structure.
The config attributes {'lambda_min_clipped': -5.1} were passed to DDPMScheduler, but are not expected and will be ignored. Please verify your scheduler_config.json configuration file.
Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████████████████| 2/2 [01:27<00:00, 43.56s/it]
Killed

Steps to Reproduce

python3 main.py --text "a hamburger" --workspace trial -O --IF --vram_O

Expected Behavior

Start training

Environment

Ubuntu 20.04, Cuda 11.7 GPU A10, 24GB VRAm

[tydia]

It is likely you ran out of RAM when loading shards. You can verify this by running htop and observing Mem and Swp.

If this is the case, you can temporarily try to increase the swap size to trade your disk space as memory.

Here is a guide on how to do that: https://askubuntu.com/questions/178712/how-to-increase-swap-space

[AmanKishore]

Thanks for the quick reply, tried doing this with docker but still running into issues:

docker run -itd --gpus all --privileged --shm-size=8gb --memory=8gb --memory-swap=16gb stable-dreamfusion:latest bash

any tips?