Closed matthew-jurewicz closed 4 years ago
Can't find batch normalization in the code, so presumably this works just fine with batch size 1?
oh hey! so transformers don't use batch norm, they use layer norm, so any batch size is ok! I would do batch size x gradient accumulation = 32 at least
batch size x gradient accumulation = 32
Oh wow, I never knew that!
Can't find batch normalization in the code, so presumably this works just fine with batch size 1?