search

tenstorrent / tt-umd

User-Mode Driver for Tenstorrent hardware
Apache License 2.0
9 stars 6 forks source link

[open-umd] Reserve the last 128MB of outbound NOC address unmapped if iATU is used #124

Open pjanevskiTT opened 1 month ago

pjanevskiTT commented 1 month ago

Copy of the issue from old open-umd repo - https://yyz-gitlab.local.tenstorrent.com/tenstorrent/open-umd/-/issues/8

In order to allow systems team to use a slice of the device->host range as a channel of device->host communication, we need to agree upon a partitioning scheme for the outbound NOC address. The behaviour should not be subject to iATU programming performed by UMD and by syseng side driver/tools. So we need to purposely ensure that we do not program this translation range for the iATU. init_pcie_iatus needs to be modified such that outbound NOC address 0xF8000000-0xFFFFFFFF is not mapped.

joelsmithTT commented 1 month ago

UMD currently has some code for this, but the intent is difficult to discern and nothing takes advantage of it.

This problem goes away if we abandon the hugepage/iATU technique currently used to implement sysmem (host memory that is accessible by hardware) and use the IOMMU instead. With KMD's IOMMU support, different applications can allocate different buffers, map them for DMA, and receive the NOC address (in PCIe) that data mover kernels can use to access the buffer. The constraint here is WH's limited address space for NOC->host access.