search

richarddwang / electra_pytorch

Pretrain and finetune ELECTRA with fastai and huggingface. (Results of the paper replicated !)
324 stars 41 forks source link

duration of training #4

Closed amy-hyunji closed 4 years ago

amy-hyunji commented 4 years ago

Hi, I was wondering if we have to train for 10000 epochs as the default setting of your code to get the result. In official electra implementation, it trained for 1000000 steps so 10000 epochs seemed too long. (by the printed value it seems like there are 273193 steps for each epoch) Also, when training electra-small the save points seemed to be 0.0625*(10**6). Is this number related to the step number? Thanks :)

richarddwang commented 4 years ago

Hi @amy-hyunji , thanks for your interest! Because fastai didn't provide user a way to run steps instead of epochs (tell me plz if there is any), so I write a callback RunSteps which will stop training when 10**6 steps are completed. So the number of epochs is just any large enough integer.