search

tenstorrent / tt-umd

User-Mode Driver for Tenstorrent hardware
Apache License 2.0
9 stars 5 forks source link

Cannot detect multiple devices on 2x p150 machine #229

Closed abhullar-tt closed 1 week ago

abhullar-tt commented 1 week ago

yyzo-bh-01 is 2x p150 but running tt_SiliconDriver::detect_available_device_ids() only returns one device. Not sure if this is a UMD issue or something wrong with the machine.

broskoTT commented 1 week ago

I'm detecting 2 devices. ls /dev/tenstorrent also returns 2 devices This was mu reservation command for ird, did you indeed take 2 cards? ird reserve --docker-image ghcr.io/tenstorrent/tt-metal/tt-metalium/ubuntu-20.04-amd64:v0.52.0-rc4-dev --timeout max blackhole --machine yyzo-bh-01 --num-pci 2

Note that create-ethernet-map still doesn't work for blackhole, so somewhere in the code we might just assume 1 card in the cluster. But enumerate_device, which just lists /dev/tenstorrent seems to be working.

Can you double check, and if it still doesn't work please provide steps for reproducing the issue

abhullar-tt commented 1 week ago

Ah user error, I didn't add --num-pci 2 last night when I tried this. But I confirmed that create-eth-map does not work for BH

broskoTT commented 1 week ago

Will be supported after #235