facebookarchive / flashback

Capture and replay real mongodb workloads
Other
217 stars 72 forks source link

Allow sharded clusters to self discover #17

Closed dbmurphy closed 9 years ago

dbmurphy commented 9 years ago

If using a sharded system, it could be useful to have it self discover nodes and have something like

auto_config: true, // profiler could be [ all, secondaries, primary] // additional options could be added later auto_config_options: { profiler : "all" }

One question is due the pyMongos change in 3.0 should we support only the new MongoClient which include the replica_set_connection class, or should we detect the version and then build the connection based on which driver version is installed.

@charity : Thoughts on this?

charity commented 9 years ago

Detecting the version and building the connection based on that sounds better to me. @tredman any thoughts?

dbmurphy commented 9 years ago

I think I found a pretty backward compatible way to do this. Currently about to test this out :)

The new config is below and it will copy (user,password,auth_db) to the profiler/oplog lists as it builds them:

import logging

DB_CONFIG = {
        "auto_config" : True,
        "auto_config_options": {
            "user": "admin",
            "password": "password",
            "auth_db": "admin",
            "use_secondaries": False,
            #mongo_uri in this case is a mongos
            "mongo_uri" : "mongos_hostname:mongos_port"
        },
        "target_databases": [ ],
        "target_collections": None,
        "oplog_servers": [ ],
        "profiler_servers": [   ],
        "oplog_output_file": "./OPLOG_OUTPUT",
        "output_file": "./OUTPUT",
        "duration_secs": 60
    }

APP_CONFIG = {
    "logging_level": logging.DEBUG
}

Any suggestions or asks before I finalize on this pattern?

charity commented 9 years ago

LGTM

dbmurphy commented 9 years ago

This is now tied to the pull request btw :) I am going to close this and reference things there.