Question about the hyper-parameters of Llama?

[mama0512]

In workload/Workload_spec_v1.1.csv, I found that ffn_hidden_sizes are different from those the Llama paper mentions as below shows:

However,in your file, I found:

I was confused, please give me some response.

[zhouheyang-alibaba]

AICB workload suite is based on the relevant descriptions of the llama3 405B model structure found in the publicly available paper from llama. The paper can be accessed at the following link: https://arxiv.org/pdf/2407.21783