search

dhlee347 / pytorchic-bert

Pytorch Implementation of Google BERT
Apache License 2.0
589 stars 181 forks source link

Nice work! #4

Closed thomwolf closed 4 years ago

thomwolf commented 5 years ago

I like it! You may want to check the work NVIDIA did to incorporate FP16 training in our repo. It really speeds the model on recent GPUs (4x speed up on a V100!). You basically just have to change the Layer Norm module in the model and tweak a bit the training to use NVIDIA's apex.

dhlee347 commented 5 years ago

Thanks, wolf ! Your team's works (Open AI GPT + Google BERT in Pytorch) always helped my research a lot ! I also consider to add the FP16 feature with NVIDIA's apex.

abhishekvermasg commented 5 years ago

have you implemented the FP16 tweak in your code?

dhlee347 commented 4 years ago

Sorry, I don't have enough time to do it recently. I'll do it when I have a time maybe next year. Thanks !