Closed jeannotes closed 2 years ago
FFCV replaces the DistributedSampler of pytorch. Otherwise the usage is the same. Please refer to Pyotrch documentation for more information.
PS: We rarely find it useful to train ImageNet on more than one machine, you get significant slowdown if you don't have a high speed networking interface and you pay more at each gradient synchronization.
@GuillaumeLeclerc hopes that you give a hint.
like this
python -m torch.distributed.launch \ --nproc_per_node=2 \ --nnodes=2 \ --node_rank=1 \ train_imagenet.py --config-file rn50_configs/rn50_88_epochs.yaml \ --data.train_dataset=/home/star/Documents/jh/data/imagenet/imagenet_generate/train_500_0.50_90.ffcv \ --data.val_dataset=/home/star/Documents/jh/data/imagenet/imagenet_generate/val_500_0.50_90.ffcv \ --data.num_workers=12 --data.in_memory=1 \ --logging.folder=/home/star/Documents/jh/logs/imagenet \ --address = 30.1.100.134 --port = 1234 \ --launch pytorch
?
thanks!
To train in a distributed fashion you need to spawn multiple processes. You can either
torch.distributed.launch
(which acts as the main script).Because ti's easier to use for most people in our example we use the first option. What I suspect is happening in your example is that you are doing both 1. and 2. To make it work with option 2. you have to remove the section of the code in our example that spawn the sub-processes or you are just going to have too many and they are going to fight for the GPUs. Moreover a machine won't even be aware that it is supposed to collaborate with another one.
I suggest you start with a piece of code that works on your cluster without FFCV and then integrate FFCV inside it. We can't have it work at of the box as there are many cluster configuration and different way of spawning the jobs on the individual machines.
when train imagenet, if we have 4 machines with same number of gous, how to train imagenet, any instructions?
thanks!