Remove $gleStats from server info response

[ShaneHarvey]

Since MongoDB 3.2, the buildinfo command response contains "$gleStats": {"lastOpTime": bson.Timestamp(...), "electionId": bson.ObjectId(...)} when run on a member of a replica set shard. This causes GETS on /servers/<server_id> to fail because these BSON types are not JSON serializable.

Repro: Launch mongo-orchestration with MongoDB 3.4 or 3.2. Start a sharded cluster with a replica set shard. GET on /servers/<shard server id>:

$ curl -XPUT http://localhost:8889/v1/sharded_clusters/myCluster -d '
{"routers": [{"port": 27017}],
 "shards": [{"id": "demo-set-0",
             "shardParams": {"members": [{"procParams": {"port": 27018}}]}}]}
' | python -m json.tool
{
    ...
    "shards": [
        {
            "_id": "94e77611-224f-4e2e-accf-615f6ae6f48a",
            "id": "demo-set-0",
            "isReplicaSet": true,
            "links": [
                {
                    "href": "/v1/sharded_clusters/myCluster/shards/demo-set-0",
                    "method": "GET",
                    "rel": "get-shard-info"
                },
                {
                    "href": "/v1/replica_sets/94e77611-224f-4e2e-accf-615f6ae6f48a",
                    "method": "GET",
                    "rel": "get-replica-set-info"
                }
            ],
            "tags": []
        }
    ]
}
$ curl localhost:8889/v1/replica_sets/94e77611-224f-4e2e-accf-615f6ae6f48a| python -m json.tool
{
    ...
    "members": [
        {
            "_id": 0,
            "host": "localhost:27018",
            "links": [
                {
                    "href": "/v1/servers/f21f9888-f775-46f4-a8f8-29893ddd90fe",
                    "method": "GET",
                    "rel": "get-server-info"
                },
                {
                    "href": "/v1/replica_sets/94e77611-224f-4e2e-accf-615f6ae6f48a/members/0",
                    "method": "GET",
                    "rel": "get-replica-set-member-info"
                }
            ],
            "server_id": "f21f9888-f775-46f4-a8f8-29893ddd90fe",
            "state": 1
        }
    ]
}
$ curl localhost:8889/v1/servers/f21f9888-f775-46f4-a8f8-29893ddd90fe
Traceback (most recent call last):
  File "/Users/shane/git/mongo-orchestration/mongo_orchestration/apps/__init__.py", line 62, in wrap
    return f(*arg, **kwd)
  File "/Users/shane/git/mongo-orchestration/mongo_orchestration/apps/servers.py", line 125, in host_info
    return send_result(200, result)
  File "/Users/shane/git/mongo-orchestration/mongo_orchestration/apps/__init__.py", line 48, in send_result
    result = json.dumps(result)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/__init__.py", line 243, in dumps
    return _default_encoder.encode(obj)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/encoder.py", line 207, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/encoder.py", line 270, in iterencode
    return _iterencode(o, 0)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/encoder.py", line 184, in default
    raise TypeError(repr(o) + " is not JSON serializable")
TypeError: Timestamp(0, 0) is not JSON serializable

Mongo-orchestartion log:

DEBUG:mongo_orchestration.servers:return {'orchestration': 'servers', 'procInfo': {'pid': 49790, 'optfile': '/var/folders/lm/b1r2f8p503xg40r6x2rqv7fr0000gp/T/mongo-CmOacS',       'params': {'dbpath': '/var/folders/lm/b1r2f8p503xg40r6x2rqv7fr0000gp/T/mongo-t_j3KT', 'replSet': '94e77611-224f-4e2e-accf-615f6ae6f48a', 'logpath': '/var/folders/lm/              b1r2f8p503xg40r6x2rqv7fr0000gp/T/mongo-t_j3KT/mongod.log', 'oplogSize': 100, 'bind_ip': 'localhost', 'setParameter': {'enableTestCommands': 1}, u'port': 27018, 'shardsvr':        True}, 'name': 'mongod', 'alive': True}, 'statuses': {'locked': False, 'primary': True, 'mongos': False}, 'mongodb_uri': 'mongodb://localhost:27018', 'serverInfo':                {u'storageEngines': [u'devnull', u'ephemeralForTest', u'mmapv1', u'wiredTiger'], u'maxBsonObjectSize': 16777216, u'ok': 1.0, u'$gleStats': {u'lastOpTime': Timestamp(0, 0),        u'electionId': ObjectId('7fffffff0000000000000001')}, u'bits': 64, u'modules': [], u'openssl': {u'compiled': u'OpenSSL 1.0.2k  26 Jan 2017', u'running': u'OpenSSL 1.0.2k  26 Jan  2017'}, u'javascriptEngine': u'mozjs', u'version': u'3.4.2', u'gitVersion': u'3f76e40c105fc223b3e5aac3e20dcd026b83b38b', u'versionArray': [3, 4, 2, 0], u'debug': False,           u'buildEnvironment': {u'cxxflags': u'-Woverloaded-virtual -Wpessimizing-move -Wredundant-move -std=c++11', u'cc': u'/usr/bin/clang: Apple LLVM version 8.0.0 (clang-800.0.42.1)',  u'linkflags': u'-L/usr/local/opt/openssl/lib -pthread -Wl,-bind_at_load -fstack-protector-strong -mmacosx-version-min=10.12', u'distarch': u'x86_64', u'cxx': u'/usr/bin/clang++:  Apple LLVM version 8.0.0 (clang-800.0.42.1)', u'ccflags': u'-I/usr/local/opt/openssl/include -fno-omit-frame-pointer -fno-strict-aliasing -ggdb -pthread -Wall -Wsign-compare -    Wno-unknown-pragmas -Winvalid-pch -O2 -Wno-unused-local-typedefs -Wno-unused-function -Wno-unused-private-field -Wno-deprecated-declarations -Wno-tautological-constant-out-of-    range-compare -Wno-unused-const-variable -Wno-missing-braces -Wno-inconsistent-missing-override -Wno-potentially-evaluated-expression -fstack-protector-strong -Wno-null-          conversion -mmacosx-version-min=10.12 -fno-builtin-memcmp', u'target_arch': u'x86_64', u'distmod': u'', u'target_os': u'osx'}, u'sysInfo': u'deprecated', u'allocator':            u'system'}}
 ERROR:mongo_orchestration.apps:<function host_info at 0x102545140>
 Traceback (most recent call last):
   File "/Users/shane/git/mongo-orchestration/mongo_orchestration/apps/__init__.py", line 62, in wrap
     return f(*arg, **kwd)
   File "/Users/shane/git/mongo-orchestration/mongo_orchestration/apps/servers.py", line 125, in host_info
     return send_result(200, result)
   File "/Users/shane/git/mongo-orchestration/mongo_orchestration/apps/__init__.py", line 48, in send_result
     result = json.dumps(result)
   File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/__init__.py", line 243, in dumps
     return _default_encoder.encode(obj)
   File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/encoder.py", line 207, in encode
     chunks = self.iterencode(o, _one_shot=True)
   File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/encoder.py", line 270, in iterencode
     return _iterencode(o, 0)
   File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/encoder.py", line 184, in default
     raise TypeError(repr(o) + " is not JSON serializable")
 TypeError: Timestamp(0, 0) is not JSON serializable
 DEBUG:mongo_orchestration.apps:send_result(500)

While simply removing $gleStats fixes the problem now, a more future-proof solution would be to use MongoDB Extended JSON for responses but I think that change should happen after PyMongo implements the spec.

[behackett]

Interesting. Should mongo-orchestration use bson.json_util to avoid issues like this in the future?

[llvtt]

Maybe it's better not to include raw responses from the server (e.g. from buildinfo) in responses from mongo-orchestration; instead we can run these commands from a driver, if it's needed. Then mongo-orchestration won't need to worry about trying to JSONify anything that has unserializable types in it.

As far as buildinfo goes, the way we use mongo-orchestration means we know the build info ahead of time, so we probably don't need to include its output at all in responses from mongo-orchestration.

However, I don't know if someone else may be using this output for their tests right now.