inconsistent gunicorn workers statsd metric

[arlendarcy]

When connected to datadog, gunicorn is providing inconsistent worker metrics. Stats are visible for a couple minutes after the instance is brought up but then the worker stat drops to 0. This behavior appears to be consistent.

Added context is this is running in ECS Fargate and gunicorn and datadog are running in separate containers.

Datadog support indicated this is an issue with gunicorn.

[xkobal]

I have the same problem as @arlendarcy. It works only if I set proc_name in gunicorn.d config file on datadog agent.

[arlendarcy]

I have the same problem as @arlendarcy. It works only if I set proc_name in gunicorn.d config file on datadog agent.

@xkobal what do you set proc_name to or does any value other than the default work?

[benoitc]

can you share your config and command line?

[xkobal]

I am running an Helm Datadog Agent on a K8s cluster @arlendarcy Here is my /etc/datadog-agent/conf.d/gunicorn.d/auto_conf.yaml

init_config:

instances:
  - proc_name: internal-api

It generate some error on agent logs because it couldn't find a gunicorn process on its container, but it's the only way I made it work. If I remove this config, I lost every worker stats, but other gunicorn stats are still here.

In my gunicorn config file, I have this:

proc_name = "internal-api"
statsd_host = "datadog-agent.svc:8125"

IMHO it seems to be a Datadog bug.

[arlendarcy]

@xkobal in our case there were no stats until we fixed the config to contain the proc_name and the statsd host. Now we are just facing intermittent worker stats. Datadog support indicated it was gunicorn bug 👉 👈

@benoitc gunicorn --print-config below. No command line arguments passed besides our config file.

accesslog                         = -
backlog                           = 2048
bind                              = ['0.0.0.0:5003']
ca_certs                          = None
capture_output                    = True
cert_reqs                         = 0
certfile                          = None
chdir                             = /opt/service
check_config                      = False
child_exit                        = <ChildExit.child_exit()>
ciphers                           = None
config                            = ./gunicorn.conf.py
daemon                            = False
default_proc_name                 = gunicorn
disable_redirect_access_to_syslog = False
do_handshake_on_connect           = False
dogstatsd_tags                    =
enable_stdio_inheritance          = True
errorlog                          = -
forwarded_allow_ips               = ['127.0.0.1']
graceful_timeout                  = 65
group                             = 0
initgroups                        = False
keepalive                         = 65
keyfile                           = None
limit_request_field_size          = 8190
limit_request_fields              = 100
limit_request_line                = 4094
logconfig                         = None
logconfig_dict                    = {}
logger_class                      = gunicorn.glogging.Logger
loglevel                          = warning
max_requests                      = 0
max_requests_jitter               = 0
nworkers_changed                  = <NumWorkersChanged.nworkers_changed()>
on_exit                           = <OnExit.on_exit()>
on_reload                         = <OnReload.on_reload()>
on_starting                       = <OnStarting.on_starting()>
paste                             = None
pidfile                           = None
post_fork                         = <Postfork.post_fork()>
post_request                      = <PostRequest.post_request()>
post_worker_init                  = <PostWorkerInit.post_worker_init()>
pre_exec                          = <PreExec.pre_exec()>
pre_fork                          = <Prefork.pre_fork()>
pre_request                       = <PreRequest.pre_request()>
preload_app                       = False
print_config                      = True
proc_name                         = gunicorn
proxy_allow_ips                   = ['127.0.0.1']
proxy_protocol                    = False
pythonpath                        = /opt/service
raw_env                           = []
raw_paste_global_conf             = []
reload                            = False
reload_engine                     = auto
reload_extra_files                = []
reuse_port                        = False
secure_scheme_headers             = {'X-FORWARDED-PROTOCOL': 'ssl', 'X-FORWARDED-PROTO': 'https', 'X-FORWARDED-SSL': 'on'}
sendfile                          = None
spew                              = False
ssl_version                       = 2
statsd_host                       = ('localhost', 8125)
statsd_prefix                     =
strip_header_spaces               = False
suppress_ragged_eofs              = True
syslog                            = False
syslog_addr                       = udp://localhost:514
syslog_facility                   = user
syslog_prefix                     = None
threads                           = 4
timeout                           = 65
tmp_upload_dir                    = None
umask                             = 0
user                              = 0
when_ready                        = <WhenReady.when_ready()>
worker_abort                      = <WorkerAbort.worker_abort()>
worker_class                      = gthread
worker_connections                = 1000
worker_exit                       = <WorkerExit.worker_exit()>
worker_int                        = <WorkerInt.worker_int()>
worker_tmp_dir                    = None
workers                           = 5
wsgi_app                          = run:app