search

leaderj1001 / Stand-Alone-Self-Attention

Implementing Stand-Alone Self-Attention in Vision Models using Pytorch
MIT License
456 stars 83 forks source link

Excessive Memory Usage #21

Open digantamisra98 opened 4 years ago

digantamisra98 commented 4 years ago

ResNet-26 on CIFAR-10 (8.3M params) using your implementation without any changes fails to fit in memory in HAL and even on Colab - P100, T4 and P4. What was your system configuration for doing the benchmarks?

acthelen commented 4 years ago

I have the same problem ResNet-26 on CIFAR-10. Running an RTX 2070 with 8gb graphics card RAM, I get the message:

Tried to allocate 308.00 MiB (GPU 0; 8.00 GiB total capacity; 5.08 GiB already allocated; 266.22 MiB free; 24.83 MiB cached)

This seems really odd. I would also like to know what hardware this was created on.

Lishey commented 4 years ago

I have met the same issue, the memory seems never enough.

sammens commented 4 years ago

It uses about 13.00 GiB so if your GPU memory is less than that, you may get excessive memory usage problem. Maybe you should reduce the batch size to resolve this.

digantamisra98 commented 4 years ago

13 GB for a ResNet-26?

sammens commented 4 years ago

I don't know why it is so but this is how it looks like at the moment. Check the GPU Process 1 Screenshot from 2020-07-08 10-39-24

digantamisra98 commented 4 years ago

Holy cow. How did this pass muster? I don't have access to my cluster now but 13.6 gigs for a ResNet-26 is absolutely ridiculous. What's the use of such costly models?

sammens commented 4 years ago

I agree with you. This is very costly.

ksouvik52 commented 4 years ago

yes, this is too costly and slow to replace Convs.

CCYChongyanChen commented 3 years ago

This is ridiculous...I am using p2.16xlarge with 16 GPU and the CUDA memory is still not enough?

image

iyaja commented 3 years ago

Have you all tried running the experiments with different batch sizes? I was able to run the default CIFAR10 experiments with a batch size of 8 using an RTX 2060. Total memory consumption was under 5 GB. On a more powerful machine with 32 GB V100 cards, I was able to run the default experiment configuration for CIFAR10 while using ~15 GB.

CCYChongyanChen commented 3 years ago

Have you all tried running the experiments with different batch sizes? I was able to run the default CIFAR10 experiments with a batch size of 8 using an RTX 2060. Total memory consumption was under 5 GB. On a more powerful machine with 32 GB V100 cards, I was able to run the default experiment configuration for CIFAR10 while using ~15 GB.

@iyaja Hi Ajay, thank you a lot for your reply! I am running it on the ChestXray dataset. It doesn't work with even a batch size of 1. Could I ask how did you change the batch size? I do it via --batch-size=1.
What is weird is that the RuntimeError always stays the same no matter how small the batch size I set. Setting batch size to 32, it says "CUDA out of memory. tried to allocate 25 MB" Setting batch size to 1, it still says "CUDA out of memory. tried to allocate 25 MB".

image

CCYChongyanChen commented 3 years ago

Have you all tried running the experiments with different batch sizes? I was able to run the default CIFAR10 experiments with a batch size of 8 using an RTX 2060. Total memory consumption was under 5 GB. On a more powerful machine with 32 GB V100 cards, I was able to run the default experiment configuration for CIFAR10 while using ~15 GB.

@iyaja Hi Ajay, thank you a lot for your reply! I am running it on the ChestXray dataset. It doesn't work with even a batch size of 1. Could I ask how did you change the batch size? I do it via --batch-size=1. What is weird is that the RuntimeError always stays the same no matter how small the batch size I set. Setting batch size to 32, it says "CUDA out of memory. tried to allocate 25 MB" Setting batch size to 1, it still says "CUDA out of memory. tried to allocate 25 MB".

image

I also have no idea why it shows gpu_devices: None and gpu: None....Shown in my last image, it seems that all 16 GPUs are occupied.

Another weird thing is... When I was using 8 GPUs, it says CUDA tried to allocate 47MB. When I changed it to 16 GPUs, it says CUDA tried to allocate 25MB....Why increasing GPU just help that little...

SangHoon95 commented 3 years ago

I am trying to use AttentionConv on another model and I got a small Params size but got a super large Forward/backward pass size at the same time. Why is Forward/backward pass size so large?

chenslcool commented 1 year ago

I am trying to use AttentionConv on another model and I got a small Params size but got a super large Forward/backward pass size at the same time. Why is Forward/backward pass size so large?

because it use unfold, it will bring huge gpu memory cost. The param is small... but intermediate featuremaps is very big.