Number of parameters in Transformer

[SoojungHong]

Hi, All

I want to count the number of trainable parameters in Transformer. In PyTorch, there is a function (e.g. encoder.parameters()) to get the number of parameter. But it seems that d2l.EncoderDecoder(TransformerEncoder, TransformerDecoder) doesn't provide this kind of function. The reference model is following Transformer. https://d2l.ai/chapter_attention-mechanisms/transformer.html

Anyone know how to count the parameter number in Transformer?

[AnirudhDagar]

You can try using collect_params() for any network and loop over the params like this:

num_params = 0
for p in net.collect_params().values():
    num_params += np.prod(np.array(p.data().shape))

Not sure if a more concise/easier method exists. But this should work.

[SoojungHong]

Thank you so much. I could count parameters with your advice.

[goldmermaid]

Hey @SoojungHong , could you close this issue if everything is clear?

[SoojungHong]

ok, thanks a lot for your help!