Feature Request: Add time to logs/output

[danielsaporo]

If I run Cartography from the terminal, I can see output for each step, but it's missing a timestamp. I can't quite see how long individual steps or indeed the overall run took without setting up a stats server. This can be solved by simply prepending the current timestamp when writing log output.

[achantavy]

Cartography doesn't prepend a timestamp to the log msg itself because when the msg reaches a log aggregator like Kibana then Kibana will include the log msg as a field called msg and the timestamp in a separate field. Having the timestamp in both the msg and time fields is redundant in this case.

It'd be possible to allow a user to configure the log msg format - maybe by subclassing the python logger - but it'd still be a decent sized refactor.

Anyone have other ideas?

[danielsaporo]

I might be a little pendantic here, but the time when the log was generated (let's call it event time) is not the same as the time that e.g. Elasticsearch auto-allocates (ingestion time).

To give you an idea where this concept would fail, if you're using Filebeat to ship logs to Elasticsearch, and Elasticsearch is down for a bit, then Filebeat would, upon reconnecting, send all the logs that were written since the last successful write, and they would all have the same timestamp.

Usually logging libraries support easy customisation of their output format, although I'd have to check about the Python one.

[achantavy]

I'm open to customization of the logger output, hopefully it's a small refactor.