Closed mtatsumiTT closed 2 weeks ago
hanging on N150 in main post-commit
https://github.com/tenstorrent/tt-metal/actions/runs/8728371259/job/23948244160 https://github.com/tenstorrent/tt-metal/actions/runs/8739120961/job/23980015228
seems to be specifically this test
tests/ttnn/unit_tests/test_sd_e2e.py::test_unet_2d_condition_model_512x512[batch_size=2-in_channels=4-input_height=64-input_width=64]
some discussions are happening over here - https://tenstorrent.slack.com/archives/C055REZR6Q3/p1713992657339109
quick update: some one the unit tests were raising OOM and allocator errors, but e2e test was passing. I'll skip the tests with OOM errors, and launch a pipeline after rebasing to the latest main to double-check all SD unit tests pass
Next step: Please try to repro it on the lastest FD2/main branch.
Next step:
I have been able to successfully reproduce the hang running watcher without NOC sanitization three times. Twice on the same op, once on a different one. I also ran without hangs ~5 times, so still ND. The different one is in the same submodule (the one @mtatsumiTT identified as problematic), so probably related.
To repro, run WH_ARCH_YAML=wormhole_b0_80_arch_eth_dispatch.yaml TT_METAL_WATCHER=1 TT_METAL_WATCHER_DISABLE_NOC_SANITIZE=1 pytest --count=100 -svv tests/ttnn/integration_tests/stable_diffusion/test_unet_2d_condition_model.py -k 512
on aknezevic/hang_debug
Sounds like we should try to repro on the submodule as next step. Then we will have a smaller test to debug in detail. Less ops.
I have found a way to reliably repro on the submodule, trying to further isolate.
I've narrowed the hang to matmul. If I reduce the subblock to 1, the hang is gone. As @TT-BrianLiu pointed out, this does change timing, so it's still unsure whether this is an MM issue or runtime. This happens in several locations in Stable Diffusion.
I have a repro with two matmuls and an eltwise:
On aknezevic/repro_MM_hang
first clear built with rm -rf built
then run: WH_ARCH_YAML=wormhole_b0_80_arch_eth_dispatch.yaml TT_METAL_WATCHER=1 TT_METAL_WATCHER_DISABLE_NOC_SANITIZE=1 pytest -svv tests/ttnn/integration_tests/stable_diffusion/test_geglu.py::test_geglu_512x512[N=1-C=2-H=256-W=1280-index=1-model_name=CompVis/stable-diffusion-v1-4-device_l1_small_size=32768]
This will loop over the test 3000 times. For me, the hang usually occurs around iter 1500.
The watcher log of the previous test is different than the one we saw previously. Perhaps a different style of hang? That one can be reproed using a 5-6 op unit test on the same branch: WH_ARCH_YAML=wormhole_b0_80_arch_eth_dispatch.yaml TT_METAL_WATCHER=1 TT_METAL_WATCHER_DISABLE_NOC_SANITIZE=1 pytest tests/ttnn/integration_tests/stable_diffusion/test_sharded_attention.py::test_time_sharded_attnention -k 4096
@jliangTT, can you please coordinate?
this issue is being discussed in the daily hang standup. The collective decision is to allow the FD2 merge / stabilization to take its course today and assign resource right afterward.
the FD2 merged over this weekend - can we rebase and re-test against the main to get the latest baseline result?
Hang is still present on both tests. I rebased and pushed aknezevic/repro_MM_hang
@AleksKnezevic have you even seen a seg fault on WH_ARCH_YAML=wormhole_b0_80_arch_eth_dispatch.yaml TT_METAL_WATCHER=1 TT_METAL_WATCHER_DISABLE_NOC_SANITIZE=1 pytest -svv tests/ttnn/integration_tests/stable_diffusion/test_geglu.py::test_geglu_512x512[N=1-C=2-H=256-W=1280-index=1-model_name=CompVis/stable-diffusion-v1-4-device_l1_small_size=32768]
?
Seem to be getting it around ~150 iterations (note this was on a t3k system, will try to repro on a vm next)
I have on occasion seen a seg fault on VM but not on BM. I've been using an N300. @tt-aho, are you clearing the built directory before running the test?
High bw debugging happening in slack thread (internal) - https://tenstorrent.slack.com/archives/C055REZR6Q3/p1715102452304479
catching up on debug log, @jliangTT can you add me to the slack thread you linked?
hey @AleksKnezevic,
Could you please summarize the tests used to reproduce the problem: both a small one and the complex one? Also, is it reproducible on main?
The small test is two matmuls and an elementwise multiply, the larger test is self attention (matmul, softmax, matmul). The tests are currently on branch aknezevic/repro_MM_hang
Fyi: https://github.com/tenstorrent/tt-metal/issues/8644 In falcon7b prefill, we've also had a problem with matmul->softmax->matmul. We needed to set subblock_h/w to 1 on both of these matmuls and furthermore to reduce number of cores allocated for matmuls to 57 to avoid di/dt issues.
Fyi: #8644 In falcon7b prefill, we've also had a problem with matmul->softmax->matmul. We needed to set subblock_h/w to 1 on both of these matmuls and furthermore to reduce number of cores allocated for matmuls to 57 to avoid di/dt issues.
Subblok specs shouldn't affect how many cores are used for matmul. It increases the number of loops in compute which essentially slows it down.
Fyi: #8644 In falcon7b prefill, we've also had a problem with matmul->softmax->matmul. We needed to set subblock_h/w to 1 on both of these matmuls and furthermore to reduce number of cores allocated for matmuls to 57 to avoid di/dt issues.
Subblok specs shouldn't affect how many cores are used for matmul. It increases the number of loops in compute which essentially slows it down.
Yup, those are 2 changes we needed to apply to get rid of di/dt (subblocks and grid size reduction) (Just realised that ‘furthermore’ is a bad choice of words for what I was trying to say :D)
Hello, was there any resolution/fix as workaround for di/dt here? We have SD hang on CI n300. Also, I see that this was downgraded to P2 a couple months ago, is that still applicable with the WH launch readiness?
By the way, doing some bisecting, looks like https://github.com/tenstorrent/tt-metal/commit/db25a356488a5e4b10752cdb87189c0e33aa238d is what causes the recent additional hang (on top of the di/dt hang)
evidence: https://github.com/tenstorrent/tt-metal/actions/runs/9967178513/job/27540588230
@tt-rkim how sure are you that this commit is the issue? Is it deterministically reproducible and how often have you seen it on this commit vs previous commits?
So one thing is I had to extend the timeout on branch because after Vincent's pytest changes, 300s was too short so initially SD was just timing out because it normally takes 10min+
However, after extending it, I bisected near the region of commits where it started failing on N300, and it always hung on yours or after.
To be fair, what I said is heuristic evidence. I've told @mywoodstock that he could try lowering FMAX on a WH X2 VM to 500MHz and try the test in that region of commits with an extended timeout (I recommend 900s to be sure).
I was trying to reproduce a hang running SD demo yesterday on N300 machine - on main it hangs regardless of the workarounds (SLOW_MATMUL env var), so then I checked out the commit before the one @tt-rkim mentioned and I wasn't able to get a hang in 16 iterations of a full demo.. I also wasn't able to reproduce a hang running the integration tests without the workarounds.
Do you plan to revert/fix the changes from the problematic commit, so that we can try and reproduce di/dt hang with more recent code?
To clarify, are you saying commit 3e7273c45b196881d39c9fe5693bc2c70706bcd2 passes but db25a35 does not? Or did you also revert 3e7273c45b196881d39c9fe5693bc2c70706bcd2?
I am saying https://github.com/tenstorrent/tt-metal/commit/e7a43ccf0852eedfda983441abe6853892148143 passes (commit before https://github.com/tenstorrent/tt-metal/commit/3e7273c45b196881d39c9fe5693bc2c70706bcd2), I assumed the two commits (https://github.com/tenstorrent/tt-metal/commit/3e7273c45b196881d39c9fe5693bc2c70706bcd2 and https://github.com/tenstorrent/tt-metal/commit/db25a356488a5e4b10752cdb87189c0e33aa238d) were a part of the same change, I wasn't clear.
Yes, so even after lowering the FMAX to 500MHz, the runs still consistently hang on main
. I was unable to get the commit before https://github.com/tenstorrent/tt-metal/commit/3e7273c45b196881d39c9fe5693bc2c70706bcd2 working -- i get some weird python errors :(.
But yes, so the conclusion is that the didt issue is fixed with the new FW (no hangs), and the commit https://github.com/tenstorrent/tt-metal/commit/3e7273c45b196881d39c9fe5693bc2c70706bcd2 looks like the issue. Correct?
I am taking a look into why this commit would cause issues specifically for SD, unless it is causing timing issues for a different bug/problem or SD is hitting some kind of very specific corner case. Currently it is not ideal to try and revert these changes.
Yeah, it would most likely be timing issue -- the thing is I see it hang in between iterations .. .either between 3-4 or 4-5 usually.
I was unable to get the commit before https://github.com/tenstorrent/tt-metal/commit/3e7273c45b196881d39c9fe5693bc2c70706bcd2 working -- i get some weird python errors :(
There was an assert that was commented out later on: https://github.com/tenstorrent/tt-metal/commit/d46d435e9a3cbe5d802fc5d6bb02fb261c52838cm, I just commented it out, and it worked.
so the conclusion is that the didt issue is fixed with the new FW (no hangs)
Overall, the new fw helps a lot, most of the repro examples we had are fixed, but we still have a couple of remaining issues still hanging less frequently.
Regarding SD specifically, the small repro examples we had from this thread are fixed, however I was unable to reproduce a hang running a full demo even with the old firmware (80.8.12) on the https://github.com/tenstorrent/tt-metal/commit/e7a43ccf0852eedfda983441abe6853892148143 commit. So I can't really say nd hangs from demo are resolved with new firmware, when I couldn't catch the nd hang in the old setup either.
Is the previous di dt issue exposed more by host constraints? For example, maybe the machine @s-jovic was using is too powerful? We have less powerful CI VMs we could try running this on.
And how many iterations did you try? @s-jovic
Is the previous di dt issue exposed more by host constraints?
Shouldn't be, but it does depend on the chips - N300 or N150 chips aren't identical, some expose hangs more often than the others. I was using a chip that was proved to expose a lot of hang repros we already have, but it doesn't have to mean it can expose all di/dt hangs that can happen.
And how many iterations did you try?
I ran 15 iterations, the demo takes a bit longer.
Might be good to test the full demo with 50 iters?
I've isolated the specific change that is causing the hang, though still unknown why it's causing it.
This commit changed which FD core is used for what, specifically eth core (0, 4) was used for dispatcher, and (0, 5) was used for prefetcher. Was changed so that (0, 4) is now prefetcher and (0, 5) is now dispatcher. This should have no real change in functionality and other tests/models are functional, so something weird is happening as a result of this (potentially some timing/race issue).
I have a fix for this in this pr #10911. Didn't enable the test in CI though. Will you enable it after doing di/dt testing on latest main @s-jovic ?
After the fix is merged, I guess it should be checked if the ND hangs still exist or not with the latest fw. However, we won't be doing these checks and removing workarounds for all models, model owners will need to check if they can remove workarounds from the models. We will let everybody know once the software fix and fw testing is done, so that model owners can addess this.
The fix is on main. So retesting/re-enabling of SD tests can be done
Should we re-enable the unstable tests?
Assigning myself to this since I'm taking ownership of SD for now. I'll test out @tt-aho's fix to see if we can re-enable these tests on CI.
Tests were re-enabled in 9492740 and no longer hang on N300 due to @tt-aho's fix. From my testing, we still require SLOW_MATMULS=1
to avoid hanging.
Should we close this or keep it open until the di/dt issues are completely resolved?
There is another spurious failure (see here) but I will track that in a separate issue.
Unless others have objections, I think we can close this specific issue.
Should we make a follow up to eventually get rid of SLOW_MATMULS
from the stack once we no longer support WH?
Unless others have objections, I think we can close this specific issue.
Should we make a follow up to eventually get rid of
SLOW_MATMULS
from the stack once we no longer support WH?
Yes I'll create an issue to track it 👍 Unless some says otherwise, I will close this once I create the 2 follow on issues.
Running SD unit tests with
WH_ARCH_YAML
on N300 devices non-deterministically hangs.To repro the issue, switch to
main
branch and run the following on N300 device:EDIT: Running the same test with enabling watcher in the fast-dispatch CI raises the
std::runtime_error
below ontests/ttnn/integration_tests/stable_diffusion/test_cross_attn_up_block_2d.py
(full log):fyi @AleksKnezevic @vtangTT @TT-billteng