search

eht16 / python-logstash-async

Python logging handler for sending log events asynchronously to Logstash.
MIT License
186 stars 51 forks source link

Connection refused backtrace suppression / exception handling #55

Closed wmacomber closed 4 years ago

wmacomber commented 4 years ago

What is the recommended way to catch when the connection to the Logstash endpoint is refused? I'd really like to be able to catch this and just locally log "CONNECTION REFUSED" rather than display 17 lines of backtrace that aren't going to help me. The connection was refused in this case because I deliberately pointed it at the wrong endpoint to trigger the errors, but the same principal applies to whenever it cannot make a connection to the Logstash endpoint.

<27>Sep 23 19:10:04 python3[2805]: An error occurred while sending events: [Errno 111] Connection refused
<27>Sep 23 19:10:04 python3[2805]: Traceback (most recent call last):
<27>Sep 23 19:10:04 python3[2805]:   File "/usr/lib/python3.8/site-packages/logstash_async/worker.py", line 118, in _fetch_events
<27>Sep 23 19:10:04 python3[2805]:   File "/usr/lib/python3.8/site-packages/logstash_async/worker.py", line 140, in _fetch_event
<27>Sep 23 19:10:04 python3[2805]:   File "/usr/lib/python3.8/queue.py", line 167, in get
<27>Sep 23 19:10:04 python3[2805]: _queue.Empty
<27>Sep 23 19:10:04 python3[2805]:
<27>Sep 23 19:10:04 python3[2805]: During handling of the above exception, another exception occurred:
<27>Sep 23 19:10:04 python3[2805]:
<27>Sep 23 19:10:04 python3[2805]: Traceback (most recent call last):
<27>Sep 23 19:10:04 python3[2805]:   File "/usr/lib/python3.8/site-packages/logstash_async/worker.py", line 214, in _flush_queued_events
<27>Sep 23 19:10:04 python3[2805]:   File "/usr/lib/python3.8/site-packages/logstash_async/worker.py", line 260, in _send_events
<27>Sep 23 19:10:04 python3[2805]:   File "/usr/lib/python3.8/site-packages/logstash_async/transport.py", line 176, in send
<27>Sep 23 19:10:04 python3[2805]:   File "/usr/lib/python3.8/site-packages/pylogbeat.py", line 85, in __enter__
<27>Sep 23 19:10:04 python3[2805]:   File "/usr/lib/python3.8/site-packages/pylogbeat.py", line 95, in connect
<27>Sep 23 19:10:04 python3[2805]:   File "/usr/lib/python3.8/site-packages/pylogbeat.py", line 104, in _create_and_connect_socket
<27>Sep 23 19:10:04 python3[2805]: ConnectionRefusedError: [Errno 111] Connection refused
eht16 commented 4 years ago

There is not a really a way to ignore the stacktrace. You could set the log level for the appropriate logger to CRITICAL but obviously this would remove all log messages not just the stack trace (logging.getLogger('LogProcessingWorker').setLevel(logging.CRITICAL)).

A better solution could be to try catch known network exceptions (like connection refused and timeout) and log those without stacktrace. If you are interested, feel free.