If StatsD metrics are enabled, any action that outputs metrics will fail if StatsD endpoint is not available

[fiodarsazanavets]

Apache Airflow version: 1.10.9

Environment: Docker Swarm

What happened: We are running Airflow in a separate Docker container where we have StatsD metrics enabled. The StatsD server is running in a separate container on the same network and Airflow is configured to send metrics to it.

If, for whatever reason, the StatsD server container is not available, absolutely any Airflow action that outputs metrics will fail. You won't even be able to execute basic commands, such as "airflow version".

The example error message is as follows:

File "/usr/local/bin/airflow", line 25, in <module>
    from airflow.configuration import conf
  File "/usr/local/lib/python3.7/site-packages/airflow/__init__.py", line 31, in <module>
    from airflow.utils.log.logging_mixin import LoggingMixin
  File "/usr/local/lib/python3.7/site-packages/airflow/utils/__init__.py", line 24, in <module>
    from .decorators import apply_defaults as _apply_defaults
  File "/usr/local/lib/python3.7/site-packages/airflow/utils/decorators.py", line 34, in <module>
    from airflow import settings
  File "/usr/local/lib/python3.7/site-packages/airflow/settings.py", line 121, in <module>
    prefix=conf.get('scheduler', 'statsd_prefix'))
  File "/usr/local/airflow/.local/lib/python3.7/site-packages/statsd/client/udp.py", line 35, in __init__
    host, port, fam, socket.SOCK_DGRAM)[0]
  File "/usr/local/lib/python3.7/socket.py", line 748, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -2] Name or service not known

What you expected to happen: This error shouldn't occur, as the metrics are sent via UDP in a fire-and-forget fashion. Perhaps, it would make sense to log an error if the metrics endpoint doesn't exist, but it doesn't make sense to prevent the action from executing.

How to reproduce it:

1. Build and run a Docker container with any type of StatsD receiver (e.g. Prometheus StatsD Exporter).
2. Build and run Airflow Docker container with StatsD requirement installer that is configured to send metrics to the StatsD receiver container (as per this instruction).
3. Stop the StatsD receiver container.
4. Enter bash on the Airflow container and execute "airflow version" command.

[boring-cyborg[bot]]

Thanks for opening your first issue here! Be sure to follow the issue template!

[ravigillella]

+1

[github-actions[bot]]

This issue has been automatically marked as stale because it has been open for 365 days without any activity. There has been several Airflow releases since last activity on this issue. Kindly asking to recheck the report against latest Airflow version and let us know if the issue is reproducible. The issue will be closed in next 30 days if no further activity occurs from the issue author.

[github-actions[bot]]

This issue has been closed because it has not received response from the issue author.