Can seqlog be used for gunicorn web server logging?

[vitekzach]

• PySeq version: 0.3.16
• Python version: 3.7.5
• Operating System: linux

Description

Describe what you were trying to get done. I have a python app running in a Docker container using flask and gunicorn to deploy it. I would like to have not only in-app logs to be logged in seq, but also the web-server logs (coming from gunicorn).

Tell us what happened, what went wrong, and what you expected to happen. I tried to use the logconfig_dict parameter to configure the logging for the gunicorn server, but it fails.

What I Did

Here is the gunicorn config file I was using:

import logging
import sys
import json
import seqlog

# Gunicorn configuration file

# which socket and IP to bind
bind = '0.0.0.0:5000'

# store heartbeat file on a memory-only part of the system
worker_tmp_dir = '/dev/shm'

# at least 2 workers, because if only 1 heartbeat can time out if response takes a long time
workers = 2

# 
threads = 2

#
worker_class = 'gthread'

# level of logs to put out
loglevel = 'debug'

# specify location for error log - choose '-' if  want stderr
# errorlog = '/app/errors'
errorlog = '-'

# specify location for acess log - choose '-' if  want stdout
# accesslog = '/app/accesses'
accesslog = '-'

logconfig_dict = {
    'version': 1,
    'disable_existing_loggers': True,
    'formatters': {
        'formatter_seq': {
            'style': '{'
        },

        # 'formatter_generic': {
        #     'format': '%(asctime)s [%(process)d] [%(levelname)s] %(message)s',
        #     'datefmt': '%Y-%m-%d %H:%M:%S',
        #     'class': logging.Formatter
        # },
        #
        # 'formatter_access': {
        #     'format': '%(message)s',
        #     'class': logging.Formatter
        # }

    },
    'handlers': {
        # 'handler_console': {
        #     'class': 'StreamHandler,
        #     'formatter': 'generic',
        #     'args': (sys.stdout, )
        # },

        'handler_seq_toseq': {
            'server_url': '*****',
            'class': 'seqlog.structured_logging.SeqLogHandler',
            'formatter': 'formatter_seq',
            'api_key': '*****',
            'level': 'DEBUG',
            'batch_size': 100,
            'auto_flush_timeout': 10,
            'json_encoder_class': 'json.encoder.JSONEncoder',
        },

        # 'handler_seq_console': {
        #     'class': seqlog.structured_logging.ConsoleStructuredLogHandler,
        #     'formatter': 'formatter_seq'
        # }
    },
    'loggers': {
        'root': {
            'level': 'DEBUG',
            'handlers': ['handler_seq_toseq']#, 'handler_seq_console']
        },

        'logger_gunicorn.error': {
            'level': 'DEBUG',
            'handlers': ['handler_seq_toseq'], #, 'handler_seq_console'],
            'propagate': 1,
            'qualname': 'gunicorn.error'
        },

        'logger_gunicorn.access': {
            'level': 'DEBUG',
            'handlers': ['handler_seq_toseq'], #, 'handler_seq_console'],
            'propagate': 0,
            'qualname': 'gunicorn.access'
        }
    }
}

I get following error:

Error: Unable to configure root logger

Before it was failing on the console handler.

Is it possible to take advantage of the seq logger to set up logging this way? Is there another way I can achieve this? I would like to keep the console logging as well, except it was failing too for to me an unknown reason.

Thanks!

[tintoy]

Hi - I’ve never tried to do this, so I’ll have to look into it and get back to you :)

[grosren]

Hi,

I stumpled over this issue, because I faced the same error during my project, where I use Gunicorn + Flask inside a Docker container. I use a gunicorn-config.py which loads

with open('gunicorn_logging.yaml', 'rt') as file:
    log_config = yaml.safe_load(file.read())
logconfig_dict = log_config

After upgrading to Gunicorn 20.0.4 I always got the error: Error: Unable to configure root logger

Solution: I added

root:
  level: INFO
  handlers: [default]
  propogate: yes

to my YAML and now it works again.

I assume the root logger needs to be on the top level of your logging dictionary. https://docs.python.org/3/library/logging.config.html#logging.config.dictConfig

The dictionary passed to dictConfig() must contain the following keys:

• version
• loggers
• root

[tintoy]

Hi - thank you, this completely fell off my task list and I’m sorry for not getting back to you both sooner!

[tintoy]

I’ll update the documentation to mention this 😃

[tintoy]

I just realised that the example in the docs already includes the root element. I'm closing this issue for now, but feel free to reopen if you need to :)