Investigate E75 support

tenstorrent / tt-metal

:metal: TT-NN operator library, and TT-Metalium low level kernel programming model.

Apache License 2.0

473 stars 75 forks source link

Investigate E75 support #3246

Closed jvasilje closed 1 year ago

mbahnasTT commented 1 year ago

tt-rkim commented 1 year ago

Hammad on cloud team confirmed no E75 available in Colo. therefore not in cloud

mbahnasTT commented 1 year ago

Please access the e75 card, using ssh ttmetal@10.230.36.111 password= tte75 Please coordinate with me when you are using the workstation, it's located in SC lab and its nickname is GS-11

abhullar-tt commented 1 year ago

Is this machine 2x row harvested?

tt-rkim commented 1 year ago

Yes, product definition for E75 is two rows harvested.

abhullar-tt commented 1 year ago

@davorchap did we settle on the harvested GS dispatch/storage core config?

abhullar-tt commented 1 year ago

Ops that need to be uplifted:

abhullar-tt commented 1 year ago

Ops that need to be uplifted:

[ ] E75 Support: Uplift maxpool op for resnet config #3369

[ ] E75 Support: Sharded op expects 98 num cores #3370

[ ] E75 Support: Resnet50 first conv fails on reduced grid size #3371

[ ] E75 Support: Bert MMs configs have hardcoded grid sizes #3373

[ ] E75 Support: CB OOM for Falcon LM Head with seq_len = 512 #3374

Merged https://github.com/tenstorrent-metal/tt-metal/pull/3308 which enables E75 support. There are some ops/op configs that are not functional on E75 (issues for relevant ops listed above) . Unit tests and model tests using these ops are disabled.

FYI @chekangliang

jvasilje commented 1 year ago

great progress @abhullar-tt!

chekangliang commented 1 year ago

As discussed, this issue tracked the bring up of E75 - with the progress so far we can close this. The remaining bring up work will be tracked in #3484. There is more investigation on the ttnn team that needs to discussed but we can de-couple