search

IDEA-CCNL / Fengshenbang-doc

16 stars 9 forks source link

Hi, there are some questions about training hardware #16

Closed AnShengqiang closed 2 years ago

AnShengqiang commented 2 years ago

I noticed that you wrote "32张A100训练14天" in the document. Erlangshen-MegatronBert-1.3B.md

I have some questions:

  1. Does 32 A100 mean (8 cards in one machine) 4 or (32 cards in one machine) 1? (In other words, multiple cards on one machine or multiple machines)
  2. If multiple machines, how do these machines communicate at the hardware level? (NVSwitch+NVlink or netword card?)

We look forward to your answer, thank you ^_^