Add support for reading structured logs (aka logv2) from MongoDB 4.4+

[stennie]

Starting in the MongoDB 4.4 release, mongod and mongos output all log messages in a new structured JSON format which is designed to be parseable using standard JSON libraries. JSON tools such as jq and mongoimport can also be used.

[nanangarsyad-mt]

Hi,

Thanks for this amazing tools. Unfortunately, I'm really in need of mplotqueries, but my log already in a structured format (i'm using mongo 4.4). Is there any temporary workaround to solve this issue ? Like for example, a way to convert a structured log into the old log, so mplotqueries could read the log again.

Best regards,

[stennie]

@nanangarsyad-mt I'm not aware of a trivial conversion of JSON logs to the legacy log format, but that's the general scope of this improvement suggestion. Since MongoDB 4.4+ logs are in JSON format you can likely work out conversion using a JSON library or tool like jq, but some trial and error will be required to match the previous log output expected by mplotqueries. The MongoDB 4.4 documentation on log messages has some helpful examples of working with jq.

If you do work out an approach, please comment here with more info.

I also suggest having a look at the Keyhole project, which aims to provide more actionable insights than mtools including performance analytics, tuning, and monitoring. Keyhole supports legacy MongoDB log formats, 4.4+ structured logs, and also the compressed FTDC (Full Time Diagnostic data Capture) diagnostic data from MongoDB 3.2+.

There's a good series of blog posts with more info from Keyhole's creator: Peek at your MongoDB Clusters like a Pro with Keyhole Part 1, Part 2, and Part 3.

[derwaldgeist]

I am using MongoDB on an ATLAS cluster and was pretty surprised mtools cannot handle its default log format. I guess the issue is about addressing this?

[NareshBavisetty]

@stennie,

I am using Mongodb atlas, tried to convert the log files to old format it did but m tools unable to read it!

sed -E 's/^{\"t\"\:{\"\$date\"\:\"|\"}\,\"s\"\:\"|\"\,[ ]{0,9}\"c\"\:\"|\"\,[ ]{0,9}\"id\"\:|\,[ ]{0,9}\"ctx\"\:\"|\"\,\"msg\"\:|}$/ /g' yourMongod4_4.log > mongodb.log

[itstin]

Did we get any fix on this ?

[SteveH-US]

Having used keyhole, I can say that it attempts to be much more than what the log analysis part of mtools did, but without the features many like about mtools, including plotting queries and getting list of query shapes. The point is, that keyhole's target consumer appears to be someone like Ken, a MongoDB consultant, and not directly to someone like us, customers using MongoDB.

Additionally, having read through the three blog post, keyhole's features does not include log analysis.

[simagix]

@SteveH-US, the keyhole log analytics blog is first available in the Peek at your MongoDB Clusters like a Pro with Keyhole Part 1. Additional blogs about the feature can be found at Feel The Pulse of Mongo and Streamline Keyhole and Maobi Reports Creation. Keyhole prints out a summary, but to get a better analytics report, use it with Maobi. The latest Moabi also plots ops in a chart.

[SteveH-US]

Hi Ken,

Having read through the three blog post, keyhole's features does not include log analysis.

[inetufo]

hope to support

[simagix]

@SteveH-US , I created Hatchet for JSON log processing, and it includes an audit page to report exceptions and usages as well as other reports and charts. A blog is also available. Ideas and suggestions are welcome.

[jhyearsley]

Any updates on this? Why not just hack it and use convert-json-logs-legacy as a dependency to handle the conversion

[jacobemcken]

Any updates on this? Why not just hack it and use convert-json-logs-legacy as a dependency to handle the conversion

convert-json-logs-to-legacy seems like a very nice workaround :+1:

It took me a little while to figure out why I got the error:

bash: ./generate_mplot_logs.py: cannot execute: required file not found

The first line in the file generate_mplot_logs.py points to /usr/local/bin/python3 which I don't have. I changed the line to point to where I have Python: /usr/bin/python3

I have now successfully parsed the converted content of a Mongo JSON file, with mloginfo.

[ghost]

I've tried to use convert-json-logs-to-legacy but when I attempt to plot queries from a resulting log it shows no data for r or w, other mplotqueries modes like numYields, nScanned work fine:

>mplotqueries legacy.log --logscale --yaxis r

no data to plot.