Open benelot opened 5 years ago
Alternatively, is there a default interface to implement for eliot destinations?
Hi,
Handler
expect logging.Record
objects.So the pseudo-code would be something like:
def destination_for_stdlib_handler(handler):
def destination(m):
record = makeLoggingRecordForMessage(m) # choose INFO vs ERROR level, figure out what message would be if any... maybe message is JSON serialized dict?
handler.emit(record)
return destination
And example usage:
from logging.handlers import SocketHandler
import eliot
eliot.add_destinations(destination_for_stdlib_handler(SocketHandler('yourlogserver', 8080)))
There's interesting policy questions of how you turn Eliot message into a logging.Record
, which depend in part on how you consume logs (do you expect them to be human readable? Will you use eliot-tree? etc.).
Logging levels would plausibly be INFO by default, ERROR for a failed action message or a traceback.
As next steps I would suggest:
I am using Eliot to send messages to an ELK stack (logz.io in our case), so we have a nice handler for rest-calls optimized for higher throughput. I played a bit with destinations and found out the same problem you identified: logging.Records are stupid when it comes to representing eliot messages. Yes, we would love to use eliot-tree, so my current solution is just to use the rest-call sender backend that sends the json.dumps(eliotmessage) to the stack. Works well so far. If you are interested in a contribution regarding this, I can post the backend code and we can turn it into proper code for a pull request. For now I just pack it into a function called to_http(message): which calls the sender inside with the json.dump.
I'd be interested in seeing the code, yes, and understanding how general purpose it is. E.g. would it work for any ElasticSearch backend?
I would recommend BTW reusing/wrapping the JSON code already in Eliot (and if that means some refactoring, that can be done), both because it allows registering custom serializers, and because I will at some point be switching it to a faster JSON library (#372).
(By the way, you can probably convince Kibana to show messages in the right order: find specific task_uuid, sort by task_level (you just need to make sure it's interpreting it as a list of integers, not a string). Not quite as pretty as eliot-tree, but it does show messages in the right order.)
The handlers of the default logging library do a good job at sending information to different destinations. https://docs.python.org/3/library/logging.handlers.html
Your library does an excellent job at creating log messages. Why not combine both? At our company, we already have several custom handlers, so that would be perfect to just plug them into eliot. If you are interested, how can I help you with the implementation of it?