No support of bytes/binary data coming from MongDB

[resurtm]

Hello @mrsarm!

Looks like mongotail does not support bytes (i.e. binary data) coming from MongoDB. Please consider the following details regarding found bug:

Mongotail

(venv) resurtm@resurtm-desktop:~/dev/mongotail$ mongotail task_tracker -f
2017-11-27 18:34:41.777 COUNT     [users] : {"$or": [{"username": "resurtm"}, {"email": "resurtm@gmail.com"}]}
Traceback (most recent call last):
  File "/home/resurtm/dev/mongotail/venv/lib/python3.5/site-packages/mongotail/out.py", line 60, in print_obj
    query += ', ' + json_encoder.encode(obj['updateobj'])
  File "/home/resurtm/dev/mongotail/venv/lib/python3.5/site-packages/mongotail/jsondec.py", line 57, in encode
    result = super(JSONEncoder, self).encode(o)
  File "/usr/lib/python3.5/json/encoder.py", line 198, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/lib/python3.5/json/encoder.py", line 256, in iterencode
    return _iterencode(o, 0)
  File "/home/resurtm/dev/mongotail/venv/lib/python3.5/site-packages/mongotail/jsondec.py", line 54, in default
    return json.JSONEncoder.default(self, o)
  File "/usr/lib/python3.5/json/encoder.py", line 179, in default
    raise TypeError(repr(o) + " is not JSON serializable")
TypeError: b'$2b$12$d3Kr8DL2nvhCyzvbgUTbROhCS.RqUO0R/wmw7DdXbhvOgqk5Ql2vC' is not JSON serializable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/resurtm/dev/mongotail/venv/bin/mongotail", line 11, in <module>
    sys.exit(main())
  File "/home/resurtm/dev/mongotail/venv/lib/python3.5/site-packages/mongotail/mongotail.py", line 237, in main
    tail(client, db, args.n, args.follow, args.verbose, args.metadata)
  File "/home/resurtm/dev/mongotail/venv/lib/python3.5/site-packages/mongotail/mongotail.py", line 83, in tail
    print_obj(result, verbose, metadata, server_version)
  File "/home/resurtm/dev/mongotail/venv/lib/python3.5/site-packages/mongotail/out.py", line 168, in print_obj
    warn('Unknown registry\nDump: %s' % json_encoder.encode(obj))
  File "/home/resurtm/dev/mongotail/venv/lib/python3.5/site-packages/mongotail/jsondec.py", line 57, in encode
    result = super(JSONEncoder, self).encode(o)
  File "/usr/lib/python3.5/json/encoder.py", line 198, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/lib/python3.5/json/encoder.py", line 256, in iterencode
    return _iterencode(o, 0)
  File "/home/resurtm/dev/mongotail/venv/lib/python3.5/site-packages/mongotail/jsondec.py", line 54, in default
    return json.JSONEncoder.default(self, o)
  File "/usr/lib/python3.5/json/encoder.py", line 179, in default
    raise TypeError(repr(o) + " is not JSON serializable")
TypeError: b'$2b$12$d3Kr8DL2nvhCyzvbgUTbROhCS.RqUO0R/wmw7DdXbhvOgqk5Ql2vC' is not JSON serializable
(venv) resurtm@resurtm-desktop:~/dev/mongotail$ 

MongoDB

Data in MongoDB:

{
    "_id" : ObjectId("5a1bf2658a23bfeed32e7dde"),
    "first_name" : "Timur",
    "username" : "resurtm",
    "email" : "resurtm@gmail.com",
    "password" : { "$binary" : "JDJiJDEyJGZnZ0QzdkRoVVRlRVhvMmdqQUNIeE9scEhSZHk5bGpXL01IOFRXUVhCdjN0RWs4QzlJaFRL", "$type" : "00" },
    "last_name" : "Ruziev"
}

Let me know if you need more details or I'm missing something.

Thank you for such a nice project as mongotail! :+1:

Cheers, @resurtm

[mrsarm]

Thanks @resurtm for the report. I'll try to add this features for the next release, after the final release of MongoDB 3.6.

Meanwhile you can use the option -v, --verbose to see the raw information about the query, but it prints a lots of information about the query and without parsing special types like ObjectId.

Can you provide me an example of how to add a registry with an insert query with a binary field using the mongodb console?

[resurtm]

@mrsarm, thanks!

I'm not sure how it can be reproduced using MongoDB shell. However you can try the following PyMongo code:

import pprint

from pymongo import MongoClient

client = MongoClient()

client.test_db.test_collection.insert_one({
    'test_field': 'проверка'.encode('utf-8'),
})

for item in client.test_db.test_collection.find():
    pprint.pprint(item['test_field'])
    pprint.pprint(item['test_field'].decode('utf-8'))

I think it does what you want:

[mrsarm]

Ok, looks easy to run and test. Thanks for the example!

[mrsarm]

Thanks @resurtm for your report, I'm adding support for this field type. Here are the different ways to represent binary data: https://docs.mongodb.com/manual/reference/mongodb-extended-json/#binary

[mrsarm]

Added support to BinData in upcomming v2.2.0

[mrsarm]

Released!

Anyway the support to serialize binary data from PyMongo driver + Python 2.x is really bad. So any binary field will displayed as an string in the log output with the no ASCII charactesr scaped, I recommend to install mongotail with Python 3 to get a correct visualization:

First uninstall from Python 2 with:

$ pip uninstall mongotail

Then install with:

$ pip3 install mongotail

[resurtm]

@mrsarm Great! Thanks!