tenstorrent / tt-metal

:metal: TT-NN operator library, and TT-Metalium low level kernel programming model.
Apache License 2.0
307 stars 26 forks source link

[HELP] How to resolve WAR dependency in Circular Buffer #9364

Open woundedleaf opened 1 month ago

woundedleaf commented 1 month ago

In a multicast example, NoC0 Thread in a sender is responsible for reading data and multicast to receiver, and Compute Thread is responsible for compute and pop the same CB. Then a WAR issue happens. When NoC0 Thread uses push_back to notify the Compute Thread that the data is ready, it also starts to read the data in CB and multicast to receiver. How to ensure the pop_up operation in Compute Thread always happens after the read operation in NoC0 thread is done? Is there any hardware dependency check mechanism that I missed in the programming guide?

// NoC0
reserve_back
noc_read
read_barrier
noc_write_multicast
// no write barrier
push_back
// Compute
wait_front
"compute"
pop_up

Can someone please help answer this question?

Thank you!

TT-BrianLiu commented 1 month ago

We don't push_back in the NoC0 thread until after the mcast is finished. So the compute will never happen for the sender core until after the mcast is done.

Actually, you're right that mcast is non-blocking and that there can be potential issues if compute uses the data at the same time. This isn't an issue so far because:

woundedleaf commented 1 month ago

Thanks for your replay!

Is there a possibility like this could happen?

The Compute and NoC Threads are moving forward a lot, and the async mcast operation is blocking for some reason, causing the Compute Thread to free up space in the CB. Then the NoC Thread can reserve and write to it again, but this space is actually still being read by the mcast operation.