search

mrsarm / mongotail

Command line tool to log all MongoDB queries in a "tail"able way
https://pypi.org/project/mongotail/
GNU General Public License v3.0
192 stars 17 forks source link

Crash for queries containing `MaxKey` / `MinKey` #35

Closed pp0rtal closed 2 years ago

pp0rtal commented 2 years ago 
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/mongotail/out.py", line 57, in print_obj
    query = json_encoder.encode(cmd['filter']) if 'filter' in cmd else "{}"
  File "/usr/local/lib/python3.8/site-packages/mongotail/jsondec.py", line 63, in encode
    result = super(JSONEncoder, self).encode(o)
  File "/usr/local/lib/python3.8/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/local/lib/python3.8/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "/usr/local/lib/python3.8/site-packages/mongotail/jsondec.py", line 60, in default
    return json.JSONEncoder.default(self, o)
  File "/usr/local/lib/python3.8/json/encoder.py", line 179, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type MaxKey is not JSON serializable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/mongotail", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.8/site-packages/mongotail/mongotail.py", line 239, in main
    tail(client, db, args.n, args.follow, args.verbose, args.metadata)
  File "/usr/local/lib/python3.8/site-packages/mongotail/mongotail.py", line 83, in tail
    print_obj(result, verbose, metadata, server_version)
  File "/usr/local/lib/python3.8/site-packages/mongotail/out.py", line 200, in print_obj
    warn('Unknown registry\nDump: %s' % json_encoder.encode(obj))
  File "/usr/local/lib/python3.8/site-packages/mongotail/jsondec.py", line 63, in encode
    result = super(JSONEncoder, self).encode(o)
  File "/usr/local/lib/python3.8/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/local/lib/python3.8/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "/usr/local/lib/python3.8/site-packages/mongotail/jsondec.py", line 60, in default
    return json.JSONEncoder.default(self, o)
  File "/usr/local/lib/python3.8/json/encoder.py", line 179, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type MaxKey is not JSON serializable

Note I did not tried with all 20+ Mongodb types https://docs.mongodb.com/manual/reference/operator/query/type/

mrsarm commented 2 years ago

@pp0rtal sorry for the delay to respond, and thanks for your report. I just confirmed the error reproducing it locally.

I'll provide a fix soon.

mrsarm commented 2 years ago

Hey @pp0rtal , I'm going to release a new major version of Mongotail, this is the pre-release: 3.0b1.

It adds support to these types and add supports to the new MongoDB driver, if you want to give it a try, update (-U) your version installed with the --pre flag:

$ pip3 install -U --pre mongotail
pp0rtal commented 2 years ago

!Hola @mrsarm! Thank you so much for handling this issue :pray: and sorry for this late answer

➜  ~ mongotail --version               
mongotail 3.0b1 <https://github.com/mrsarm/mongotail> (python 2.7.18)
➜  ~ mongotail localhost/mydb -f
2021-12-03 13:54:25.621 DISTINCT  [groups] : "_id", {"$or": [{"users": ObjectId("5fa80a75b401a755a321c8b5")}], "sys": {"$ne": "hello"}}
...
2021-12-03 13:54:25.630 AGGREGATE [projects] : [{"$match": {"learningNeed": ObjectId("61795e026b8b4b6f553e1863")}}, {"$limit": 1}, {"$group": {"_id": 1, "n": {"$sum": 1}}}]
Mongotail EXCEPTION - Unknown command operation
Dump: {"ns": "360Learning.courses", "command": {"$db": "360Learning", "killCursors": "courses", "cursors": [4530188732135917703]}, "ts": ISODate("2021-12-03T14:34:19.517Z"), "op": "command"}

I must be doing something wrong, I'll give a new try later

mrsarm commented 2 years ago

Thanks @pp0rtal for your feedback !

About Docker, I've published the latest beta on Docker Hub, but because it's not a final version I didn't make the "latest" tag points to it, so in order to run the beta with Docker you have to run it with the version tag: docker run -it --rm mrsarm/mongotail:3.0b1 192.168.0.39/mydb -f, and to continue using the last stable version, just don't use the tab or use the tag latest.

Anyway, there is good stuff in your last feedback, I'm seeing this:

Mongotail EXCEPTION - Unknown command operation
Dump: {"ns": "360Learning.courses", "command": {"$db": "360Learning", "killCursors": "courses", "cursors": [4530188732135917703]}, "ts": ISODate("2021-12-03T14:34:19.517Z"), "op": "command"}

Which looks like Mongo started to log the killCursors commands with the C capitalized, which cause my query that tries to omit these internal operations to be skipped. Note aside, big part of Mongotail work is skip these kind of noise from the Mongo logs that vary from version to version. Could you tell me what exact version of MongoDB are you working with? and on which platform?

If your are not sure, just execute Mongotail with the -i argument to get the info, e.g. docker run -it --rm mrsarm/mongotail:3.0b1 192.168.0.39/mydb -i

mrsarm commented 2 years ago

Hey, just pushed into PyPI and Docker Hub the new beta release 3.0b2.

So changing the version either with pip or docker you will be able to test it. It should fix the error mentioned above, although I can't reproduce it, so if you can check it would be great !

pp0rtal commented 2 years ago

@mrsarm Sorry for the delay (again)

My MongoDb server is 4.2.1 on Ubuntu (my database is running in a container)

docker run -it --rm mrsarm/mongotail:3.0b2 192.168.0.39/mydb -i
Unable to find image 'mrsarm/mongotail:3.0b2' locally
3.0b2: Pulling from mrsarm/mongotail
5eb5b503b376: Pull complete 
5c69ac0246d0: Pull complete 
2bd4622f6a75: Pull complete 
3afb63acb482: Pull complete 
9836c3fb0d13: Pull complete 
7d45bd3427da: Pull complete 
Digest: sha256:219ab66601b3acd4a0bb27d43b7600191feb65d1d8b3a8429d286b6cb9fc6f97
Status: Downloaded newer image for mrsarm/mongotail:3.0b2
Version: 4.2.1
Distribution: x86_64
Target OS: linux
OpenSSL running: OpenSSL 1.1.1  11 Sep 2018
Max BSON Object Size: 16777216
Debug: False
Javascript Engine: mozjs

I'm not sure what happens here, the software is closing

➜  ~ docker run -it --rm mrsarm/mongotail:3.0b1 192.168.0.39/360Learning -f
➜  ~ echo $?
0

I do have the exact same behavior with the docker packaged version

docker run -it --rm mrsarm/mongotail 192.168.0.39/360Learning -f
2021-12-03 13:54:25.621 DISTINCT  [groups] : "_id", {"$or": [{"users": ObjectId("5fa80a75b401a755a321c8b5")}], "sys": {"$ne": "marketplace"}}
2021-12-03 13:54:25.624 QUERY     [groups] : {"company": ObjectId("54bd005142c39aba01ffd999"), "sys": "others"}. 1 returned.
2021-12-03 13:54:25.624 QUERY     [learningNeeds] : {"_id": ObjectId("61795e026b8b4b6f553e1863")}. 1 returned.
2021-12-03 13:54:25.625 DISTINCT  [upvotes] : "author", {"learningNeed": ObjectId("61795e026b8b4b6f553e1863")}
2021-12-03 13:54:25.627 DISTINCT  [upvotes] : "author", {"learningNeed": ObjectId("61795e026b8b4b6f553e1863")}
2021-12-03 13:54:25.628 DISTINCT  [posts] : "_id", {"target._id": {"$in": [ObjectId("61795e026b8b4b6f553e1863")]}}
2021-12-03 13:54:25.628 AGGREGATE [posts] : [{"$match": {"target.collection": "posts", "target._id": {"$in": [ObjectId("61795e415b53e57c9d935775"), ObjectId("61795e8a50dc6808e5039018"), ObjectId("61795f1da33bd07d29ea7aee")]}}}, {"$group": {"_id": 1, "n": {"$sum": 1}}}]
2021-12-03 13:54:25.629 AGGREGATE [projects] : [{"$match": {"learningNeed": ObjectId("61795e026b8b4b6f553e1863")}}, {"$limit": 1}, {"$group": {"_id": 1, "n": {"$sum": 1}}}]
2021-12-03 13:54:25.630 AGGREGATE [projects] : [{"$match": {"learningNeed": ObjectId("61795e026b8b4b6f553e1863")}}, {"$limit": 1}, {"$group": {"_id": 1, "n": {"$sum": 1}}}]
Mongotail EXCEPTION - Unknown command operation
Dump: {"op": "command", "ns": "360Learning.courses", "command": {"killCursors": "courses", "cursors": [4530188732135917703], "$db": "360Learning"}, "ts": ISODate("2021-12-03T14:34:19.517Z")}
Mongotail EXCEPTION - Unknown registry
Dump: {"op": "command", "ns": "360Learning.courses", "command": {"killCursors": "courses", "cursors": [4530188732135917703], "$db": "360Learning"}, "ts": ISODate("2021-12-03T14:34:19.517Z")}

If you need more info to reproduce the error / a dump file I can provide I will do

mrsarm commented 2 years ago

Thanks @pp0rtal !

I got the same "empty" output with the connection immediately closed if I didn't set the log level to 1 or 2 before trying to access the DB with the -l option, or the database doesn't exist yet, but once I enable logging I don't have problem connecting with Mongo 4.2.1 also launched with Docker. You are using the "official" MongoDB images, right? any special launch option when executing "docker run ..." or when configuring it with docker-compose?

pp0rtal commented 2 years ago

@mrsarm My bad about the service closing, I switched my network and IP was invalid at this moment :man_facepalming:

So I retried with --rm and still have exceptions

docker run -it --rm mrsarm/mongotail:3.0b1 192.168.0.105/360Learning -f
...
Mongotail EXCEPTION - Unknown command operation
Dump: {"op": "command", "ns": "360Learning.courses", "command": {"killCursors": "courses", "cursors": [4530188732135917703], "$db": "360Learning"}, "ts": ISODate("2021-12-03T14:34:19.517Z")}
Mongotail EXCEPTION - Unknown registry
Dump: {"op": "command", "ns": "360Learning.courses", "command": {"killCursors": "courses", "cursors": [4530188732135917703], "$db": "360Learning"}, "ts": ISODate("2021-12-03T14:34:19.517Z")}

You are using the "official" MongoDB images, right?

Actually I'm using circleci/mongo:4.2.1 (the image of my CI), no options, do you think this can be an issue?

mrsarm commented 2 years ago

Thanks @pp0rtal !! good to know I'm not introducing a new bug related with connections while fixing others :sweat_smile:

About your recent error, it's because you are using the tag 3.0b1, but you should use 3.0b2 instead.

pp0rtal commented 2 years ago

About your recent error, it's because you are using the tag 3.0b1, but you should use 3.0b2 instead.

@mrsarm sorry I'm not focused :grimacing: Still no luck with the revision :/

➜  ~ docker run -it --rm mrsarm/mongotail:3.0b2 192.168.0.105/360Learning -f
...
2021-12-03 13:54:25.630 AGGREGATE [projects] : [{"$match": {"learningNeed": ObjectId("61795e026b8b4b6f553e1863")}}, {"$limit": 1}, {"$group": {"_id": 1, "n": {"$sum": 1}}}]
Mongotail EXCEPTION - Unknown command operation
Dump: {"op": "command", "ns": "360Learning.courses", "command": {"killCursors": "courses", "cursors": [4530188732135917703], "$db": "360Learning"}, "ts": ISODate("2021-12-03T14:34:19.517Z")}
Mongotail EXCEPTION - Unknown registry
Dump: {"op": "command", "ns": "360Learning.courses", "command": {"killCursors": "courses", "cursors": [4530188732135917703], "$db": "360Learning"}, "ts": ISODate("2021-12-03T14:34:19.517Z")}
mrsarm commented 2 years ago

I just released the 3.0b3 on PyPI and Docker Hub, and I'm pretty sure it's fixed this time :crossed_fingers:

pp0rtal commented 2 years ago

@mrsarm Nice! No more Mongotail EXCEPTION :rocket:

However, I still don't have any query logged :confused: the latest is from 2021-12-03 13:54:25.630, Since December, I didn't do any upgrade on my db (still based on circleci/mongo:4.2.1).

I can't test MinKey MaxKey obviously, but I'm sure you fixed it. Probably I should open a new issue (you can close this one) / try to debug a bit before.

mrsarm commented 2 years ago

Thanks @pp0rtal for all your help, version 3.0.0 was released on PyPI and Docker Hub, and it wouldn't be possible without your help.