Use AppInsigths to capture Cosmos Benchmark metrics

[M-Lipin]

Subtasks:

• [x] Illustrate in the diagram how metrics are collected (detailed building blocks)
• [x] Capture Realtime P90, P99, P999, P9999
• [x] Explore other ways to capture metrics (AppMetrics and OpenTelemetry)
• [x] Investigate integration with AppInsights
• [x] Collect metrics with OpenTelemetry and send to AppInsights
• [x] Collect metrics from multiple VMs and aggregate these in AppInsights
• [x] Add dashboard using percentiles query
• [ ] Availability visualization and 10 seconds granularity for latencies

sequenceDiagram
    participant A as Azure VM
    participant B as Benchmark Tool
    participant C as CosmosDB
    participant D as OpenTelemetry
    participant E as Application Insights

    A->>B: Launch Benchmark Tool

        B->>C: Request
        activate C

        C->>B: Response 
        deactivate C
        B->>D: Put metrics on each operation completion
        activate D

        par Every 5 sec (flush)
            D->>E: Send successes and failures, latencies and RPS
            Note right of C: collect and aggregate raw data
        end
        deactivate D
        B->>B: Benchmark completed

[M-Lipin]

• Right way to capture.
• Need to fix the issue with point percentiles to Application Insights.
• Distinguish latencies by read latencies, write latencies, queries and so on.
• Fix the issue with window latencies (10 seconds or configured).

[M-Lipin]

• Need to collect metrics using OpenTelemetry
• Need to aggreagate metrics from multiple VMs
• Need to illustrate the process in details how metrics are collected and sent to AppInsights

[M-Lipin]

https://github.com/Azure/azure-cosmos-dotnet-v3/pull/3932 - Dashboard template

[philipthomas-MSFT]

Please add project type for associated PRs so that it appears on board.