FartyPants
commented
1 year ago If you load previous project, It will load the previous data and change the GA = batch size / micro batch size
it is exactly as you say 4 x 32 = 128 There is no such thing as micro batch size and batch size in LLM - it was ooba's way of trying to make it logical for newbies but IMHO it only muddies things up. (also nobody else uses such convention)
There is only one batch size. that's how many blocks will be processed on the same time. That's the IMPORTANT number.
The GA is a fake batch size (a hack) if you can't have enough real batch sizes in GPU. It accumulates gradients over so many steps then updates weights all at once.
The readme has more about this. In short - move True Batch size as high as it gets for your GPU before blowing up - if you have 3090 that's easily 10 or more on 13b @ 4-bit training. Using GA on that will likely make the training worse. (Think of it as averaging) Use GA only if you can't have BS because of low GPU. For example BS = 1 then you should move GA to make up some "fake" batch steps at least which will make up for the lack of real batch sizes.
I really want to be consistent with everyone else and how the LLM transformers are set up - there is ONE batch size and that's it.
Hello! I have a question about a change in the interface. How to understand it now? For example, Batch Size=128 and Micro Batch Size=4 now correspond to True Batch Size=4 and Gradient Accumulation Steps=32? Or is it wrong?