[Epic] Network monitoring

[chuckbelisle]

Related to #839 Depends on #845 & #908

In order to better get a hold of our solution, we require the testing and implementation of Jaeger network tracing solution on our clusters. https://www.jaegertracing.io/

Tasks

• [x] #915
• [x] #916

[vexingly]

Good example of distributed tracing of an application using jaeger and istio: https://piotrminkowski.com/2022/01/31/distributed-tracing-with-istio-quarkus-and-jaeger/

[Jose-Matsuda]

Given the result of the task Determine where we are with network monitoring in the cluster. Here are some steps we can take to advance further.

• [x] https://github.com/StatCan/aaw/issues/1797
• [x] https://github.com/StatCan/aaw/issues/1802
• [x] https://github.com/StatCan/aaw/issues/1805
• [x] https://github.com/StatCan/aaw/issues/1843
• [x] https://github.com/StatCan/aaw/issues/1804
• [ ] https://github.com/StatCan/aaw/issues/1858
• [ ] https://github.com/StatCan/aaw/issues/1842
• [ ] The above is just for a MVP. The OTEL team does suggest using a collector instead of just an agent to send data to the backend (and of course elastic also supports getting data from a collector. Setting up a collector does not seem too bad either, just need to determine which mode we want. Should be managed via argocd (Work for this already being done in https://github.com/StatCan/aaw-argocd-manifests/tree/1805-otel-instruments)
• [ ] Then after we've confirmed some working, we can go on and instrument our other applications with OTel and just continue to configure them to feed into Elastic / Jaeger
• [ ] Will need to ensure that each instrument'ed application has the label on it.