Longhaul improvements

[tanvigour]

Describe the proposal

Longhaul infrastructure and monitoring needs to be improved in order for the community to restore trust in the Longhaul tracking

Following are the proposed improvements in this space, ordered as per priority

1. Need to be able to reproduce scenarios
2. Better Documentation - Longhauls, dashboards, metrics etc
3. Make longhaul dashboards publicly accessible: Running Longhaul using different subscription that doesn't have MSIT restrictions might help with this
4. Define what is the measurement goal using Longhaul
5. Change metric to be more towards semantic correctness in longhaul dashboards
6. Introduce chaos to Longhaul tracking to monitor the behavior for tracking longterm stability
7. Have a terraform script for longhaul for people to deploy their own version of of tests.
8. Decide if performance metric needs to be a part of Longhaul dashboards, if not, how else do we track them well.
9. Introduce a centralized data store for aggregating use cases

cc: @halspang @johnewart @artursouza

[tanvigour]

https://github.com/dapr/proposals/issues/17

[tmacam]

Some additional enhancements in light of #211 and #201.

• [ ] Use Azure-hosts prometheus and grafana instead of helm-installed ones and retire/remove prometheus and grafana in this cluster
• [ ] Configure Group-based auth to hosted grafana so members with access to the OSS sub can authenticate using Microsoft Entra.
• [ ] Configure DNS aliases for the hosted grafana instances.
• [ ] Configure logging for AKS instances of longhaul clusters
• [ ] Use AAD Authentication instead of passwords

Whenever possible, those should be done by means of bicep updates so we can treat those clusters more like cattle and less like pets.