search

AGI-Edgerunners / LLM-Adapters

Code for our EMNLP 2023 Paper: "LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Models"
https://arxiv.org/abs/2304.01933
Apache License 2.0
1.02k stars 92 forks source link

About the fp16 parameter setting #17

Open noob-ctrl opened 1 year ago

noob-ctrl commented 1 year ago

I I try to set the fp16 parameter to True and False respectively, why does the training time become longer when it is set to True?

HZQ950419 commented 1 year ago

Hi @noob-ctrl ,

I don't know the exact reason. Two possible reasons are：

  1. the batch size is too small, try to increase the batch size to 128 or larger.
  2. FP16 is not always faster than FP32 in all devices.

Please let me know if you have further questions.