[agent-operator] statsd_exporter push integration not working

[VLZZZ]

What's wrong?

Hi!

I can't understand if agent-operator (aka static mode agent) is capable of running statsd_exporter integration in a push mode (TCP/UDP listener) as described in Grafana agent static mode docs.

The issue is that you can't have custom ports for the Kind: GrafanaAgent unless custom containers are used.
statsd_exporter listener also can't be configured to match the single server listen port 8080 as it will be a conflict (as expected).

Is it even possible to implement a push model for statsd_exporter integration while using Kind: Integrations CRD? No documentation was found addressing this kind of do/don't approach.

Steps to reproduce

1. Deploy agent-operator
2. Deploy custom GrafanaAgent with integrations CRD support
3. Deploy statsd_exporter CRD integration - no way to send TCP/UDP metrics to statsd_exporter integrations

System information

No response

Software version

Grafana agent v0.33.1

Configuration

apiVersion: monitoring.grafana.com/v1alpha1
kind: Integration
metadata:
  labels:
    agent: grafana-agent-metrics # will be collected by the other MetricsIntance
    type: grafana-statsd-integration
  name: agent-statsd-exporter
  namespace: grafana
spec:
  name: statsd
  type:
    unique: true
  config:
    listen_udp: :9125
    listen_tcp: :9125
    mapping_config:
      defaults:
        observer_type: histogram
        buckets: [0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 0.75, 1, 1.25, 1.5, 1.75, 2, 2.25, 2.5, 2.75, 3, 5, 10, 60, 240, 600, 1200, 2400, 5000, 20000, 50000, 100000, 500000, 1000000, 10000000]

Logs

Grafana agent logs:
---------
ts=2023-07-12T17:16:01.060956424Z caller=server.go:191 level=info msg="server listening on addresses" http=[::]:8080 grpc=127.0.0.1:12346 http_tls_enabled=false grpc_tls_enabled=false
ts=2023-07-12T17:16:01.061351097Z caller=node.go:85 level=info agent=prometheus component=cluster msg="applying config"
ts=2023-07-12T17:16:01.061501793Z caller=remote.go:180 level=info agent=prometheus component=cluster msg="not watching the KV, none set"
ts=2023-07-12T17:16:01.062104241Z caller=zapadapter.go:78 level=info component=traces msg="Traces Logger Initialized"
ts=2023-07-12T17:16:01.062137833Z caller=integrations.go:166 level=warn msg="integrations-next is enabled. integrations-next is subject to change"
ts=2023-07-12T17:16:01.065371883Z caller=reporter.go:103 level=info msg="running usage stats reporter"
ts=2023-07-12T17:16:05.921989832Z caller=entrypoint.go:337 level=info msg="reload of config file requested"
ts=2023-07-12T17:26:10.061142161Z caller=entrypoint.go:337 level=info msg="reload of config file requested"
---------------

If you try set `listen_udp/listen_tcl` to `:8080`:

---
ts=2023-07-12T17:38:36.906100758Z caller=workers.go:119 level=error component=integrations msg="integration exited with error" id=statsd/ip-10-200-150-149.us-west-2.compute.internal:8080 err="failed to start TCP listener: listen tcp 0.0.0.0:8080: bind: address already in use"
ts=2023-07-12T17:38:36.906963616Z caller=workers.go:119 level=error component=integrations msg="integration exited with error" id=statsd/ip-10-200-150-149.us-west-2.compute.internal:8080 err="failed to start TCP listener: listen tcp 0.0.0.0:8080: bind: address already in use"
ts=2023-07-12T17:38:36.909158031Z caller=reporter.go:103 level=info msg="running usage stats reporter"
ts=2023-07-12T17:38:36.909260776Z caller=workers.go:119 level=error component=integrations msg="integration exited with error" id=statsd/ip-10-200-150-149.us-west-2.compute.internal:8080 err="failed to start TCP listener: listen tcp 0.0.0.0:8080: bind: address already in use"
panic: send on closed channel

goroutine 198 [running]:
github.com/prometheus/statsd_exporter/pkg/event.(*EventQueue).FlushUnlocked(...)
    /go/pkg/mod/github.com/prometheus/statsd_exporter@v0.22.8/pkg/event/event.go:120
github.com/prometheus/statsd_exporter/pkg/event.(*EventQueue).Flush(0xc0007fbef0)
    /go/pkg/mod/github.com/prometheus/statsd_exporter@v0.22.8/pkg/event/event.go:116 +0x94
github.com/prometheus/statsd_exporter/pkg/event.NewEventQueue.func1()
    /go/pkg/mod/github.com/prometheus/statsd_exporter@v0.22.8/pkg/event/event.go:95 +0x3a
created by github.com/prometheus/statsd_exporter/pkg/event.NewEventQueue
    /go/pkg/mod/github.com/prometheus/statsd_exporter@v0.22.8/pkg/event/event.go:92 +0x1b9

[Upanshu11]

@VLZZZ Although the operator doesn't creates the service on it's own, I created a kbernetes service like this:

apiVersion: v1
kind: Service
metadata:
  labels:
    operator.agent.grafana.com/name: grafana-agent
    operator.agent.grafana.com/shard: "0"
    operator.agent.grafana.com/type: integrations
  name: statsd
spec:
  ports:
  - port: 9125
    protocol: UDP
    targetPort: 9125
  selector:
    operator.agent.grafana.com/name: grafana-agent
    operator.agent.grafana.com/shard: "0"
    operator.agent.grafana.com/type: integrations

In the service's spec->selector I am matching lables for the "grafana-agent-integrations-deploy" deployment which gets created by the GrafanaAgent operator.

My Integration is a little different though:

apiVersion: monitoring.grafana.com/v1alpha1
kind: Integration
metadata:
  labels:
    agent: grafana-agent-metrics # will be collected by the other MetricsIntance
    type: grafana-statsd-integration
  name: agent-statsd-exporter
  namespace: grafana
spec:
  name: statsd
  type:
    unique: true
  config:
    listen_udp: :9125
    listen_tcp: :9125
    autoscrape:
      enable: true
      metrics_instance: grafana/metrics-instance

Where the metrics_instance helps me point to the correct remoteWrite

Once the above two are set, you can use the host and port of the service to send metrics to statsD exporter.