[FEATURE REQ]Perf investigation/improvements for workloads other than just point read and write (with retries, with failure injection, and especially with monitoring enabled)

[FabianMeiswinkel]

Run perf tests for above scenarios, capture profiles and decide where to make improvements.

• In general worth looking at places where we use flatMap (or other reactor operators forcing thread switches)
• Investigate whether we should always switch off of I/O thread in the SDK
• Investigate data structures used in CosmosDiagnostics and see where efficiency improvements can be made
• Investigate whether moving diagnostics handling off of hot path is a good idea
• See whether it is worth misusing FailureInjection to suppress any actual I/O calls for perf test
• DiagnosticsProvider uses PropagatingMono/Flux to push tracing off o hot path or i/o threads - my preference would be to make COSMOS.SWITCH_OFF_IO_THREAD_FOR_RESPONSE the default instead - but needs testing

[ghost]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @kushagraThapar, @TheovanKraay

[joshfree]

/cc @g2vinay @mikeharder