What is the exactly count of parameters in DeBERTaV3-base

QingruZhang / AdaLoRA

AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning (ICLR 2023).

MIT License

231 stars 23 forks source link

What is the exactly count of parameters in DeBERTaV3-base #5

Closed Lucius-lsr closed 11 months ago

Lucius-lsr commented 11 months ago

In your Implementation Details, it was previously mentioned that DeBERTaV3-base consists of 183 million parameters. However, the correct information, based on official sources (https://huggingface.co/microsoft/deberta-v3-base) (https://github.com/microsoft/DeBERTa), is that DeBERTaV3-base actually has 86 million parameters.

Lucius-lsr commented 11 months ago

OK, I found that the embedding of DeBERTaV3-base accounts for 97 million parameters. The official data does not count them. I've taken care of my confusion.