bump grafana-agent container image to 0.40.4-22.04_stable

[ca-scribner]

Issue

Contributes to #296. That issue is affected by a hard to reproduce upstream bug that may have been fixed in 0.36.

Solution

Bumps the grafana-agent container image

Context

grafana-agent may include breaking changes in minor version bumps. I've scanned through the release notes between 0.35 (current version) and 0.40.5 (latest), but I don't have enough background context to really catch anything that affects us. When reviewing, please also read through this summary of breaking changes and highlight anything that matters:

• v0.40.5

  • some pg_stat_* metrics were changed because prometheus.exporter.postgres was updated.

• v0.40.0

  • "Prohibit the configuration of services within modules"?

• v0.39.0

  • (I dont think these are relevant, but don't know for sure)
    • otelcol.receiver.prometheus will drop all otel_scope_info metrics when converting them to OTLP.
    • The target block in prometheus.exporter.blackbox requires a mandatory name argument instead of a block label

• v0.37.1

  • seeveral breaking changes - see link

• v0.36.0

  • loki.source.file component will no longer automatically detect and decompress logs from compressed files. A new configuration block is available to enable decompression explicitly. See the [upgrade guide][] for migration instructions. (@thampiotr)
  • otelcol.exporter.prometheus: Set include_scope_info to false by default. You can set it to true to preserve previous behavior. (@gouthamve)
  • prometheus.remote_write: Set retry_on_http_429 to true by default in the queue_config block.You can set it to false to preserve previous behavior. (@wildum)

[ca-scribner]

Note that atm we only have rocks for <=0.40.2. 0.40.3 and 0.40.4 exist in grafana-agent-rock but I don't see them published, and 0.40.5 is still as a PR. For now I've bumped things to 0.40.2 to spot any issues, but leaving this as a draft until we decide if we want one of the later versions

[ca-scribner]

ah, 0.40.3 and 0.40.4 are stuck behind some oci-factory CLA issues