t3k runner management

[ttmchiou]

Currently there are a very limited number (4 total) of t3k runners to run all the following pipelines.

• t3k frequent
• t3k demo
• t3k unit tests
• t3k model perf
• t3k profiler

These machines are currently and some aspects of t3k pipelines are not functioning properly right now.

• [ ] https://github.com/tenstorrent/tt-metal/issues/9345 f10cs08 runner exhibits ND behavior for falcon tests
• [x] https://github.com/tenstorrent/tt-metal/issues/9791 f12cs08 runner shows reduced performance when running trace stress tests for some reason.
• [ ] t3k demo branch does not pass when running on some specific runners.
• [ ] slack notifications for models does not seem to be pinging the correct user
• [x] weka issues on f10cs07 + f12cs02 - not mounting properly
• [x] Bring up more t3k/llmbox Cloud VMs to manage load on t3k pipelines
• [x] tt-metal-ci-vm-t3k-01 showing "Device 0 is not connected to Device 3"
• [ ] t3k provisioning pipeline inconsistent, sometimes weka or ethernet connections are not setup properly for a newly provisioned device.
• [ ]
• [ ] t3k pipelines could be in non-passing state, need to review again after fixing

[ttmchiou]

https://github.com/tenstorrent/tt-metal/pull/9856 fixed f12cs02 and perf diff for stress tests

f12cs02 and f10cs08 seem to have issues running falcon40b tests? now