search

airflow-helm / charts

The User-Community Airflow Helm Chart is the standard way to deploy Apache Airflow on Kubernetes with Helm. Originally created in 2017, it has since helped thousands of companies create production-ready deployments of Airflow on Kubernetes.
https://github.com/airflow-helm/charts/tree/main/charts/airflow
Apache License 2.0
663 stars 475 forks source link

Airflow 1.10.14 with statsd-exporter #132

Closed ngreen-ny closed 3 years ago

ngreen-ny commented 3 years ago

I am using the helm chart for airflow version 1.10.14 in k8s.

I have the following metrics defined:

AIRFLOW__METRICS__STATSD_ON: "True"
AIRFLOW__METRICS__STATSD_HOST: "test-svc.airflow" (This is a service in k8s in the same airflow namespace where airflow is deployed)
AIRFLOW__METRICS__STATSD_PORT: "9125" (9125 is correct)
AIRFLOW__METRICS__STATSD_PREFIX: "testing-airflow"

I have statsd-exporter setup as an extracontainer in the values.yaml:

  - name: statsd-exporter
  image: prom/statsd-exporter
  imagePullPolicy: IfNotPresent
  ports:
  - containerPort: 9102
    protocol: TCP
  - containerPort: 8125
    protocol: TCP
  args:
    - --log.level=debug

The web and scheduler pod is split and I am using the LocalExecutor:

airflow-test-scheduler-779bcf686-4xjpp 4/4 Running 0 58m airflow-test-web-598fbc9f87-hr26r 4/4 Running 0 58m

I am scraping test-svc.airflow:9102 as a target on prometheus side.

I can send a test from the scheduler and web pod by doing the following:

echo "testnew09042021.test.myservice:1|c" | nc -w 1 -u test-svc.airflow 9125

The following shows up in promtheus:

testnew09042021_test_myservice{instance="test-svc.airflow:9102",job="statsd-exporter"}

I have version of statsd 3.3.0 installed by default with the chart.

However there are no other metrics from airflow showing up from the instance. This works on v2.0.0 which I have setup previously.

There is no statsd errors in the airflow logs. Any idea what else I can check here? Thanks.

thesuperzapper commented 3 years ago

@ngreen-ny what is the issue>

ngreen-ny commented 3 years ago

I don't see metrics getting exported on the instance or via Prometheus, however I have multiple dags running and working as expected in airflow.

ngreen-ny commented 3 years ago

Please note I am using chart version 7.16.0

thesuperzapper commented 3 years ago

@ngreen-ny when using the 8.X.X versions of the chart (preferably with airflow 2.0.0) do you still have this issue?

stale[bot] commented 3 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.