I'm wondering about how memory is utilized in a multi-gpu setup. At the moment I run on a single RTX 4090 and often find that the 24GB of VRAM is insufficient. At which point it will start using shared system memory and become terribly slow. I'm trying to figure out what would happen if I added another 4090 to my system.
I'm aware that when running on multiple GPU's a batch size of 8 would effectively become 2x a batch size of 4. Lowering the memory required per GPU for higher batch sizes. I'm also aware that you need at least a batch size of 2 for multi GPU. In my current use case I'm using a batch size of 1 and this already uses more than 24GB. If I increase the batch size to 2 and add a 2nd GPU, then what happens?
My understanding is that by adding a 2nd GPU I would effectively run the same code but on 2 GPU's simultaneously. Each processing a single view. If this is true then my assumption would be that both will overflow into shared system memory. Is this correct?
Hi,
I'm wondering about how memory is utilized in a multi-gpu setup. At the moment I run on a single RTX 4090 and often find that the 24GB of VRAM is insufficient. At which point it will start using shared system memory and become terribly slow. I'm trying to figure out what would happen if I added another 4090 to my system.
I'm aware that when running on multiple GPU's a batch size of 8 would effectively become 2x a batch size of 4. Lowering the memory required per GPU for higher batch sizes. I'm also aware that you need at least a batch size of 2 for multi GPU. In my current use case I'm using a batch size of 1 and this already uses more than 24GB. If I increase the batch size to 2 and add a 2nd GPU, then what happens?
My understanding is that by adding a 2nd GPU I would effectively run the same code but on 2 GPU's simultaneously. Each processing a single view. If this is true then my assumption would be that both will overflow into shared system memory. Is this correct?
Thank you in advance!🙂