[Master issue] Harden data pipeline infrastructure with extra goodies and cleanup like retries, unit testing, gtest junit xml support, and documentation

[tt-rkim]

Transferring cleanup list from https://github.com/tenstorrent/tt-metal/issues/9319

Cleanup tasks

• [ ] More defensive programming / re-tries for missing logs / copied jobs for workflow run retries and artifact downloading since we can hang there / error out there because a job doesn't exist anymore
• [x] #12885
• [ ] gtest information support
• [x] Fixing errors in the data:
  • [x] #12840
  • [x] re-runs can create new submission time stamps, but we still look for the earliest job in the earliest run, causing negative queue times
• [ ] Mypy / pydantic / strict type checking because we have so many random fields
• [x] https://github.com/tenstorrent/tt-metal/issues/12271
• [x] https://github.com/tenstorrent/tt-metal/issues/11711
• [x] https://github.com/tenstorrent/tt-metal/issues/10691
• [x] https://github.com/tenstorrent/tt-metal/issues/11469
• [x] https://github.com/tenstorrent/tt-metal/issues/11614
• [x] https://github.com/tenstorrent/tt-metal/issues/11704
• [ ] https://github.com/tenstorrent/tt-metal/issues/10533
• [x] https://github.com/tenstorrent/tt-metal/issues/10680
• [ ] https://github.com/tenstorrent/tt-metal/issues/11134
• [ ] https://github.com/tenstorrent/tt-metal/issues/11137
• [x] https://github.com/tenstorrent/tt-metal/issues/11238
• [x] #12877
• [x] #12878
• [x] #12879
• [x] #12891
• [ ] https://github.com/tenstorrent/tt-metal/issues/12966
• [x] #13419
• [ ] bug: https://github.com/tenstorrent/tt-metal/actions/runs/11111440141 w/ https://github.com/tenstorrent/tt-metal/actions/runs/11110261767/job/30868548829 (timeout in middle, so maybe no junit)
• [ ] bug: https://github.com/tenstorrent/tt-metal/actions/runs/11158826752 w/ https://github.com/tenstorrent/tt-metal/actions/runs/11159570104/job/31018206893
• [ ] bug: https://github.com/tenstorrent/tt-metal/actions/runs/11482896288
• [ ] #13116
• [x] #13285
• [ ] #13363
• [ ] #13380

[tt-rkim]

~Looks like timeouts during pytests can cause no data to be loaded into the junit xml~

Nahhhhh this is a different, check https://github.com/tenstorrent/tt-metal/issues/12799 from now on

[tt-rkim]

Looks like re-runs don't repeat artifacts: https://github.com/tenstorrent/tt-metal/actions/runs/10908473854/

tried by: https://github.com/tenstorrent/tt-metal/actions/runs/10910636802/job/30281509160