DESIGN: OMS agent for App Metrics

[bartr]

Description

What:

Why:

When:

Where:

Tasks

• [ ] Review the spike work in /spikes/oms-log-analytics to see the work that has been done
• [ ] Schedule Design Review
• [ ] Implement feature #607
• [ ] Publish #606

Acceptance Criteria

• [ ]
• [ ] Resolve linter errors
• [ ] Remove to-do, dead code, & unnecessary comments
• [ ] Existing documentation is updated (readme, .md's)
• [ ] New documentation needed to support the change is created
• [ ] Validation test run if applicable

Constraints

References:

• References #
• OMS Agent sends Prometheus data to Log Analytics (without running Prometheus server).
• Internally OMS is using Telegraf and FluentBit.
• Explore the possibility to use FluentBit's TCP Input plugin(JSON) along with Prometheus query API
• Prometheus Query API: https://prometheus.io/docs/prometheus/latest/querying/api/
• FluentBit TCP Plugin: https://docs.fluentbit.io/manual/v/0.14/input/tcp

[jomalsan]

@bartr I'm a bit confused on the vision for this task. Is it to build a custom pod which queries Prometheus and passes those metrics to FluentBit? By default, FluentBit is only listening for JSON messages over TCP, but has no way of reaching out and querying Prometheus

[jomalsan]

Based on a discussion in standup, this story is regarding an Azure specific monitoring stack for the NGSA deployment.

Need to:

• Review the spike work in /spikes/oms-log-analytics to see the work that has been done
• Design the end goal of the development effort
  • Diagram of what is being built
  • How does this scale when we have multiple applications?
  • What metrics need to be collected?
  • What logs need to be collected?
  • What visualizations do we want? - Including which tools to display them
  • Alerting?
• Add tasks for implementing the design
• Execute

The end goal should be three fold:

• Documentation
  • Why we made the decisions we did
  • Architecture diagram
  • Installation instructions (or link to other pages for it)
• Reusable assets - configMaps, queries, dashboard definitions (if applicable)
• Demo solution running on a cluster
  • This one we need to have a larger discussion on if this should be a personal cluster, dev, or pre-prod
  • Should it be left in place as the monitoring solution for one of the higher environments?

cc: @jkeane, @atxryan, @gled4er

[jomalsan]

End goal:

• Design review on Thursday 3/25
• Productionalization of the spike code
• Deployment of OMS Agent into Dev
• Deployment of OMS Agent into Pre-Prod
• Use the IaC deployment mechanisms

[jomalsan]

OMS Agent is the backing technology for Azure Monitor for Containers, which is the recommended monitoring system in the AKS Secure Baseline documentation: https://docs.microsoft.com/en-us/azure/architecture/reference-architectures/containers/aks/secure-baseline-aks#monitor-and-collect-metrics

Azure Monitor for Containers can be turned on through the portal, az cli, ARM templates or Terraform. Used in this manor it comes pre-configured with a host of out of the box dashboards, workbooks and alerts for infrastructure and cluster health monitoring.

So, we will be taking a dependency on using AKS, and having Azure Monitor for Containers installed through this mechanism. In order to enable the collection of application logs and metrics, we will need to apply a container-azm-ms-agentconfig.yaml file to the cluster. For examples please see spikes/oms-log-analytics/3-container-azm-ms-agentconfig.yaml and the example in the CAF Terraform deployment of the secure baseline: https://github.com/Azure/caf-terraform-landingzones-starter/blob/starter/enterprise_scale/construction_sets/aks/online/aks_secure_baseline/cluster-baseline-settings/container-azm-ms-agentconfig.yaml

We will use Flux to deploy the configuration files to the cluster's, so we will have two yaml files to enable an easy way to track and update the configuration. We will track this through two files: the first in gitops/deployments/dev and the other in gitops/deployments/preprod/common.

[jomalsan]

Implementation based on these learnings is in story #628