Create observability artifacts based on YAML configurations

[minherz]

Description

Today the properties of all observability artifacts are hardcoded in the Terraform files (see provisioning/terraform/monitoring). It means that every time we want to create another artifact (e.g. a SLO), we need to copy/paste the resource configuration. An attempt to avoid this kind of repetition was made with the monitoring dashboard artifacts. The configuration of multiple dashboards are stored in provisioning/configurations/online-boutique/dashboards.

It would be useful to generalize this approach so it can be used for the all artifact types that Sandbox provisions. The current list of the artifacts includes:

• Monitoring dashboards
• Alerts
• Uptime checks
• Notification channels
• Log-based metrics
• SLOs

Additionally, the support for custom monitored microservices can be added as well.

Technical Details

The configurations should be stored in YAML because it is a much more human-friendly format. The configuration schema should be defined using JSON schema. The configurations should not support all possible options to provision the artifacts. There should be a reference configuration which will be used CI workflow for testing modifications of the terraform code that is responsible to provision configurations.

[minherz]

YAML is favored to JSON because it is a more human friendly format. The configurations that have to be in JSON (e.g. Monitoring dashboard for google_monitoring_dashboard can be either converted from YAML or incorporated into the YAML. YAML natively supports comments making it easier to document without additional challenges to schema validation.

[bourgeoisor]

Is this issue still p2 or should it be bumped to p3? Note that p2 has a resolution SLO of 6 months