implement loader commands through mongos

[leifwalsh]

notes:

• expand client API to pass in the collection options, indexes, and also the shard key and initial splitVector (with options for splitting the keyspace evenly a la presplitting a hashed shard key)
• probably want to make this a new command since it will wrap beginTransaction/beginLoad and commitLoad/commitTransaction, rather than actually allowing beginTransaction through raw
• make sure shard key and set of indexes are compatible (make the UI simple somehow, not worried about specifics at the moment)
• beginTransaction on all shards, then beginLoad on all shards
• need to do a bunch of split work, probably after beginLoad starts, definitely before we return, need to make sure that works when we're in that state
• commitLoad on all shards, then commitTransaction on all shards
• if anything fails, abortLoad and abortTransaction on all shards, then drop collection on any shards that had committed already
• need something to clean up failed loads maybe?
• mongorestore/mongoimport api?

[ankurcha]

In its current form, is there any way to use beginLoad in a sharded environment? It looks like the intention of this issue is to track creation of wrapper commands and the functionality to do this by using commands directly ie db.runCommand is good to go, or am I understanding it wrong?

[leifwalsh]

The only way to do this would be to create a non-sharded collection through mongos, then connect to its primary shard directly (not through mongos) and drop and load the collection there. Right now, beginTransaction and beginLoad are not supported through mongos, which are what you need to do bulk loads.

The work here is to make a mongos command that will start a load on all servers and do the right presplitting so that when the stream of inserts comes in, they'll get spread around to different shards' loaders, then for the mongos to coordinate all of the loader build phases on the shards and commit them.