Should this work, even if slowly, on just a 2080Ti?

[Utopiah]

Hi, just tried to run it via cog using r8.im/kuprel/min-dalle and even though it setups correctly I'm getting a PyTorch OOM error :

RuntimeError: CUDA out of memory. Tried to allocate 20.00 MiB (GPU 0; 10.73 GiB total capacity; 9.87 GiB already allocated; 20.06 MiB free; 9.88 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

This is "just" a gaming card but I'm wondering still if that's a bug, a hard limit on minimum hardware, or rather a default parameter based on T4 starting at 16GB of VRAM being the bare minimum replicate expects that could eventually be changed. Or even if starting from this repository instead could solve it.

PS: nvidia-smi shows no other process using the GPU, 11264MiB to use and CUDA 11.7

[kuprel]

Did you try float16 and grid size of 1?

[kuprel]

Oh also, the docker image on replicate is using float32

[Utopiah]

I did try grid_size 1 but not float16 as I didn't see it as a parameter. Should I try instead from the repo?

[kuprel]

Yeah float16 from the repo might work. You should be able to just run the notebook.

[Utopiah]

Thanks, any recommendation on the Docker base image to do so? Anaconda?

[kuprel]

Are you able to modify the code in the docker image? You would need to change this line to float16 https://github.com/kuprel/min-dalle/blob/9b04ff6987840722804e969de9dd8b7631f04c17/replicate/predictor.py#L29

[Utopiah]

Super, thanks in indeed it works now relying on this Dockerfile :

FROM continuumio/anaconda3 
# Assuming it runs on a directory resulting from git clone https://github.com/kuprel/min-dalle
COPY . /min-dalle/
WORKDIR /min-dalle
RUN pip install -r requirements.txt
RUN pip install min-dalle
RUN mkdir output # should be volume instead, see example below
ENTRYPOINT bash
# docker run --gpus=all -v $(pwd):/test -it ae7705b67ea2 python image_from_text.py --text='penguin on a large pirate boat' --no-mega --image-path=/test/generated.png