Issues with DDP with FFCV dataloader

libffcv / ffcv

FFCV: Fast Forward Computer Vision (and other ML workloads!)

https://ffcv.io

Apache License 2.0

2.84k stars 178 forks source link

Issues with DDP with FFCV dataloader #170

Closed Jasonlee1995 closed 2 years ago

Jasonlee1995 commented 2 years ago

I used the ffcv imagenet codes and use it to my dataset.

What I experienced DDP with pytorch dataset and dataloader, they use mostly the same vram (balanced)

However when I use with FFCV dataset and dataloader, they are not balanced (0 gpu is getting way more vram than others)

Is this normal when I use DDP with FFCV dataset and dataloader or it's bug?

GuillaumeLeclerc commented 2 years ago

Hello,

This is a well known bug and has been fixed, we haven't made a formal release yet but you can install directly from source and it should be fixed (branch 0.0.4 or 0.0.1 should work)