Airflow integration doesn't output any metric

[IvanovOleg]

Hello, I am trying to configue an integration between airflow and kibana using Airflow | Documentation (elastic.co). I deployed elasticsearch/kibana/agent/fleet-server using ECK operator. here is kibana yaml:

apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: kibana
  namespace: elasticapm
spec:
  version: 8.13.2
  count: 1
  elasticsearchRef:
    name: elasticsearch
  config:
    xpack.fleet.agents.elasticsearch.hosts: ["https://elasticsearch-es-http.elasticapm.svc:9200"]
    xpack.fleet.agents.fleet_server.hosts: ["https://fleet-server-agent-http.elasticapm.svc:8220"]
    xpack.fleet.packages:
      - name: system
        version: latest
      - name: elastic_agent
        version: latest
      - name: fleet_server
        version: latest
      - name: airflow
        version: 0.6.0
    xpack.fleet.agentPolicies:
      - name: Fleet Server on ECK policy
        id: eck-fleet-server
        namespace: default
        monitoring_enabled:
          - logs
          - metrics
        unenroll_timeout: 900
        package_policies:
        - name: fleet_server-1
          id: fleet_server-1
          package:
            name: fleet_server
      - name: Elastic Agent on ECK policy
        id: eck-agent
        namespace: default
        monitoring_enabled:
          - logs
          - metrics
        unenroll_timeout: 900
        package_policies:
          - name: system-1
            id: system-1
            package:
              name: system
          - name: airflow-1
            id: airflow1-1
            package:
              name: airflow

fleet-server.yaml:

apiVersion: agent.k8s.elastic.co/v1alpha1
kind: Agent
metadata:
  name: fleet-server
  namespace: elasticapm
spec:
  version: 8.13.2
  kibanaRef:
    name: kibana
  elasticsearchRefs:
    - name: elasticsearch
  mode: fleet
  fleetServerEnabled: true
  policyID: eck-fleet-server
  deployment:
    replicas: 1
    podTemplate:
      spec:
        serviceAccountName: elastic-agent
        automountServiceAccountToken: true
        containers:
          - name: agent
            env:
              - name: FLEET_SERVER_ELASTICSEARCH_CA
                value: /etc/pki/tls/certs/ca.crt
            volumeMounts:
              - name: elasticsearch-certs
                readOnly: true
                mountPath: /etc/pki/tls/certs/ca.crt
                subPath: ca.crt
        volumes:
          - name: agent-data
            emptyDir: {}

agent.yaml

---
apiVersion: agent.k8s.elastic.co/v1alpha1
kind: Agent
metadata:
  name: elastic
  namespace: elasticapm
spec:
  version: 8.13.2
  kibanaRef:
    name: kibana
  fleetServerRef:
    name: fleet-server
  mode: fleet
  policyID: eck-agent
  daemonSet:
    podTemplate:
      spec:
        serviceAccountName: elastic-agent
        automountServiceAccountToken: true
        containers:
          - name: agent
            env:
              - name: FLEET_SERVER_ELASTICSEARCH_CA
                value: /etc/pki/tls/certs/ca.crt
            volumeMounts:
              - name: elasticsearch-certs
                readOnly: true
                mountPath: /etc/pki/tls/certs/ca.crt
                subPath: ca.crt
            ports:
              - containerPort: 8125
                protocol: UDP
        volumes:
          - name: agent-data
            emptyDir: {}

Agent reports to kibana, everything looks fine. In the agent log I see:

{"log.level":"info","@timestamp":"2024-04-19T19:22:56.117Z","message":"Started listening for UDP on: 0.0.0.0:8125","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"statsd/metrics-default","type":"statsd/metrics"},"log":{"source":"statsd/metrics-default"},"log.origin":{"file.line":82,"file.name":"udp/udp.go","function":"github.com/elastic/beats/v7/metricbeat/helper/server/udp.(*UdpServer).Start"},"service.name":"metricbeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}

Airflow metrrics dashboard is empty and in the metrics explorer I don't see any airflow related metrics. I also tried to send a fake metric using netcat like this:

echo "foo:1|c" | nc -u -w0 <agent_ip> 8125

and didn't get the foo metric stored. I don't see error messages in the agent's log. Can anyone help me to figure out why it doesn't work?

https://discuss.elastic.co/t/airflow-integration-doesnt-work-properly/357823

[ritalwar]

In the issue details, it appears they are trying to send test data using the command echo "foo:1|c" | nc -u -w0 <agent_ip> 8125. However, instead of using <agent_ip>, the data should be sent to the StatsD server.

In the airflow.cfg file also, the statsd_host = %HOST% parameter should be set to the StatsD server IP where the data is being sent. Additionally, the same IP address should be configured in the Host field for the Airflow integration.

Could you also confirm whether the metrics sent by Airflow to the StatsD server are visible in the StatsD server logs or monitoring interface? Accessing any available logs related to this would be helpful in further analysis.

[IvanovOleg]

@ritalwar when agent connects to the fleet server, it get's a configuration from kibana via policy. Then it installs metricbeat with statsd plugin on itself and binds udp socket to the one of it's interfaces (you can't listen an interface which is not available for this particular agent, you will get an error on the agent startup that it is unable to bind to such ip). Not sure why you are separating agent and statsd server since agent doesn't pull anything, it listens for incoming metrics on it's own port. I will post again a piece of the agent's log where it starts listening a udp port:

{"log.level":"info","@timestamp":"2024-04-19T19:22:56.117Z","message":"Started listening for UDP on: 0.0.0.0:8125",

[IvanovOleg]

Here is another example using docker compose that works and proves that metricbeat and statsd are running together: metricbeat.yaml

metricbeat.config:
  modules:
    path: ${path.config}/modules.d/*.yml
    # Reload module configs as they change:
    reload.enabled: false

metricbeat.modules:
  - module: statsd
    host: "0.0.0.0"
    port: "8125"
    enabled: true
    #ttl: "30s"

processors:
  - add_cloud_metadata: ~

output.elasticsearch:
  enabled: false

output.console:
  pretty: false

docker-compose.yaml

services:
  metricbeat:
    image: docker.elastic.co/beats/metricbeat:8.13.2
    ports:
      - "8125:8125"
    volumes: 
      - ./metricbeat.yml:/usr/share/metricbeat/metricbeat.yml
  netcat:
    image: subfuzion/netcat
    entrypoint:
      - sleep
      - "3600"

[IvanovOleg]

got it working

[ritalwar]

Great! Can you share what was the issue, as we were not able to reproduce and for us it was working all fine.