TTNN sweep infrastructure revamp planning

[sjameelTT]

Some of the features that are desired include (thanks Brian for the suggestions):

Major features

• [x] Handling hangs (need to add timeouts and a tt-smi reset when it occurs)
• [x] Handling out of L1 errors (can be during tensor creation, CB allocation)
• [x] #10855
• [ ] #10856
• [x] Streaming out results of tests into DB as they're generated rather than only once all are finished
• [x] Different tests for weekly/nightly/etc (how this is done is up in the air rn - parameterize with tags above the rest like "skip_for_nightly" or by putting them in different folders.
• [ ] Separate actions for matmul/TMs/etc since sweeps are already quite long
• [x] Try-Catch around every test for uncaught errors

Minor features:

• [ ] Regex for tests in folders
• [x] --collect-only or dry run option that tells you which files are going to be run (Done)
• [x] Separate tables for the same test with different runs
• [x] Hash the timestamp so we can get the history of a test, have different columns and add an option to print only the most recent results
• [x] Remove the expected fail string matching expectation for exceptions

[ntarafdar]

I was thinking a bit more about hangs. A lot of hangs aren't hangs if we have watcher running with it. Do you think we could have our framework interact with watcher and record the watcher crash (e.g watcher says this kernel on this core read/wrote the wrong memory at this address).

I personally think this would be a nice thing to track. Each test is either (possibly not exhaustive list):

• Pass
• Fail due to PCC
• Fail due to TT ASSERT/THROW
• Watcher Exception
• Host Unknown Crash (Seg fault or some other error)
• Lastly hang

I kind of want a finer granularity instead of just a hang as this will make it a lot easier to debug.

[sjameelTT]

@jvasilje the issue tracking the sweep infra changes

[sjameelTT]

Also want to add:

• Converting sweeps to unit tests when sweeps start failing
• Running individual tests via hash or index
• Better tracking of tests (new failures, existing failures, etc)
• Using Postgres DB to enable all of the above (need a server for this) (high priority)
• Running everything with Watcher on (high priority)
• Breaking up sweeps into different tests

[jdesousa-TT]

Most of the items in this issue were covered in this PR: #10706 Remaining items are broken into issues #10855 and #10856

We are working with @kyma-tt and team to setup a production database for our test vectors and results.

All the details surrounding what was included in the framework so far is available in the tests/sweep_framework/README.md file and on these slides: https://docs.google.com/presentation/d/1zo1pS40IZAcVFivAUSJIYljjVerGWJccpVJsP5KL00E/edit?usp=sharing