Closed fchastanet closed 1 year ago
I fixed the issue using this sub class
class JsonFormatter(jsonlogger.JsonFormatter):
def format(self, record):
"""
fix issue https://github.com/madzak/python-json-logger/issues/138
"""
# check if record.getMessage() would succeed
if not isinstance(record.msg, dict):
try:
record.msg = {
"message": record.getMessage()
}
except Exception as ex:
# ignore format exception
if record.exc_info:
ex.__cause__ = record.exc_info
outer_exception = Exception(
"Internal logging error : Unable to convert error message to string was raised " +
"during inner exception management",
)
outer_exception.__cause__ = ex
try:
raise outer_exception
except Exception as ex2:
# generate the tuple needed to format the exception
#record.exc_info = sys.exc_info()
record.exc_info = None
record.exc_text = traceback.format_exc()
record.message = record.msg = {
"message": str(record.msg)
}
record.args = {}
return super().format(record)
I'm not a python expert, I guess we can do better here
You've thrown an exception while trying to call logger.error()
itself, which bypasses the logging infrastructure and instead prints to stderr.
Specifically, the problem is on the last line. Logger.error doesn't take an exception as the first argument (msg) and a string as the second (*args):
try:
1 / 0
except ZeroDivisionError as e:
# TypeError: not all arguments converted during string formatting
logger.error(e, "test exception but logger called with invalid arguments")
If you drop the e, it is fine:
... logger.error("test exception but logger called with invalid arguments")
{"asctime": "2022-10-25 06:31:00,387", "levelname": "ERROR", "message": "test exception but logger called with invalid arguments"}
Also possible to use the exception as an arg:
... logger.error("test exception but logger called with invalid arguments: %s", e)
{"asctime": "2022-10-25 06:33:44,858", "levelname": "ERROR", "message": "test exception but logger called with invalid arguments: division by zero"}
Or probably best case, capture exc_info:
... logger.error("test exception but logger called with invalid arguments: %s", e, exc_info=True)
{"asctime": "2022-10-25 06:41:44,785", "levelname": "ERROR", "message": "test exception but logger called with invalid arguments: division by zero", "exc_info": "Traceback (most recent call last):\n File \"<stdin>\", line 2, in <module>\nZeroDivisionError: division by zero"}
Sorry this was too late to be helpful :|
I ran into this same issue and it took me a while to track down what entry was causing it because it was an infrequently logged thing happening at the same time as 10k other entries. By the logging error not being formatted, it didn't get properly parsed and each line became a separate entry in aggregation, which meant it was very difficult to track it down. The issue is certainly caused by the user (in this case, us) but I think the expectation that the json logger would handle it and emit the error in a json structure would still be valid.
There are 3 solutions I've found:
logging.raiseExceptions
to False
(ref)[https://docs.python.org/3/howto/logging.html#exceptions-raised-during-logging]. This is the recommended in production, although I think silently dropping them can be harmful, so I opted not to do this.getMessage
in a try and updates the record if it fails. I found this limited and annoying, as you can't configure it with dictConfig (or I couldn't find a way) and it really only handles some issues, like providing the wrong number of arguments to a formatted log message (logger.info("hi %s", "thing1", "thing2")
) - this happened to be my issue, but it's not exactly uniform.
import logging
from pythonjsonlogger import jsonlogger
class ErrorHandlingJSONFormatter(jsonlogger.JsonFormatter): def format(self, record: logging.LogRecord) -> str: try: return super().format(record) except Exception as exc: record.msg = "Logging error" record.args = () record.levelno = logging.ERROR record.levelname = logging.getLevelName(logging.ERROR) record.exc_info = (type(exc), exc, exc.traceback)
return super().format(record)
N.B. the above is still not uniformly handling every issue, it could still cause the original logging issue and it'd be probably nicer to create a new record and set only the arguments which we know for sure we can (relatively) safely set.
A possibly simpler option is to override the handleError()
method of a handler, per https://docs.python.org/3/library/logging.html#logging.Handler.handleError, but it would be required for every handler in your system, if you wanted to capture all of those log errors. It might be practical if you have only a single handler for aggregation. I've included that override, plus stdout/stderr capturing, and a global excepthook in the example below. You can experiment with them by commenting out some of the handlers. Not sure any one is the best solution (e.g. some multi-line output on stdout/stderr), but they cooperate with aggregation, and there seems to be a dearth of clear examples online.
import logging
import sys
from pythonjsonlogger import jsonlogger
# logging config setup (json output)
logger = logging.getLogger()
logHandler = logging.StreamHandler()
formatter = jsonlogger.JsonFormatter('%(asctime)s %(name)s %(message)s')
logHandler.setFormatter(formatter)
logger.addHandler(logHandler)
# set up redirect handlers
class StandardOut(object):
def __init__(self, logname, loglevel):
self.logger = logging.getLogger(logname)
self.loglevel = loglevel
def write(self, msg):
"""Strip to prevent extra spacing"""
if (msg_clean := msg.strip()) not in ['', '\n']:
self.logger.log(self.loglevel, msg_clean)
def flush(self):
pass
def handle_log_error(*args):
logger.error('uncaught logging error', exc_info=True)
def handle_uncaught_exception(*args):
logger.error('uncaught exception', exc_info=args)
# capture failures and std streams
sys.stdout = StandardOut('projname.daemon.stdout', logging.WARNING)
sys.stderr = StandardOut('projname.daemon.stderr', logging.ERROR)
logHandler.handleError = handle_log_error
sys.excepthook = handle_uncaught_exception
### tests
# actual logging
logger.warning('test regular logging')
# stdout, stderr
print('test stdout')
print('test stderr', file=sys.stderr)
# make a mistake with logging
try:
1 / 0
except ZeroDivisionError as e:
# TypeError: not all arguments converted during string formatting
logger.error(e, "test exception but logger called with invalid arguments")
# other unhandled error
raise RuntimeError("Test unhandled exception")
Great suggestions above!
Hello, I have this issue
Here my configuration
Here the problematic use case
The result is not json encoded