search

facebookresearch / moco

PyTorch implementation of MoCo: https://arxiv.org/abs/1911.05722
MIT License
4.83k stars 794 forks source link

Can't use one GPU? #47

Closed HymEric closed 4 years ago

HymEric commented 4 years ago

If I want to use one or two GPUs in one server? What should I do?

KaimingHe commented 4 years ago

We exploit the fact that by default BN is split into multiple GPUs in which mean/std is independently computed. To run on one GPU, you may implement BN split along the N (batch) dimension to mimic this effect (see NaiveBatchNorm in Detectron2). You also need to change the lr (e.g., linearly) if you change the batchsize to fit memory. To run on 2 GPUs, try --lr 0.0075 --batch-size 64.

rbgirshick commented 4 years ago

implement BN split along the N (batch) dimension to mimic this effect

There is an implementation of split (aka sub-batch) BN for video (3d) models that might be helpful to you: https://github.com/facebookresearch/SlowFast/blob/master/slowfast/models/batchnorm_helper.py#L37-L46

This can easily be adapted to the the 2d BN case.

JaejinCho commented 3 years ago

@rbgirshick Thanks for the link. I actually needed this and am writing a code myself starting from there. BTW, if you know, It would be great if you could answer this question, which is regarding the code snippet. https://github.com/facebookresearch/SlowFast/issues/357#issue-784645472

JingLiJJ commented 2 years ago

@rbgirshick Thanks for the link. I actually needed this and am writing a code myself starting from there. BTW, if you know, It would be great if you could answer this question, which is regarding the code snippet. facebookresearch/SlowFast#357 (comment)

@rbgirshick Thanks for the link. I actually needed this and am writing a code myself starting from there. BTW, if you know, It would be great if you could answer this question, which is regarding the code snippet. facebookresearch/SlowFast#357 (comment)

Hello, have you realized this? I also want to implement MOCO on a single GPU.

DiaosWang commented 1 year ago

@JingLiJJ maybe you can see https://github.com/leftthomas/MoCo

solauky commented 1 year ago

来信收到，谢谢！

athenacon commented 9 months ago

@JingLiJJ maybe you can see https://github.com/leftthomas/MoCo

Hey! I would really want to run it on a single GPU, but I am not sure if I understand how this implementation works for a single GPU, could you please explain?

solauky commented 9 months ago

来信收到，谢谢！