marcello3d / node-mongolian

[project inactive] Mongolian DeadBeef is an awesome Mongo DB driver for node.js
https://groups.google.com/group/node-mongolian
zlib License
350 stars 50 forks source link

How to run mapreduce on slave? i.e. how to set .setSlaveOk() #74

Open sajal opened 12 years ago

sajal commented 12 years ago

Im trying Master/Slave replication for something. My goal is to use the slave only for mapreduce tasks, get the results inline and then store it into the master.

My full code is given below. When i use master for both serverm and servers , then it works, but if u use mapreduce on the slave server in servers, i get following error :-

> [debug] mongo://slaveserver:27017: Connected
[debug] mongo://slaveserver:27017: Initialized as unknown
[debug] Finished scanning... primary? no
{ stack: [Getter/Setter], arguments: undefined, type: undefined, message: 'Could not connect to primary: ' }
[debug] mongo://slaveserver:27017: Disconnected

AFAIK mongo will allow you to run mapreduce on slave as long as the output is inline and setSlaveOk() is set. How do i set that using mongolian?

in the code below, dbs does not have a method getMongo()

var mongofunctions = require('./mongofunctions'),
  conf = require('./config/conf'),
  helpers = require('./helpers'),
  Mongolian = require("mongolian"),
  serverm = new Mongolian("masterserver:27017"),
  servers = new Mongolian("slaveserver:27017"),
  dbm = serverm.db(conf.dbname),
  dbs = servers.db(conf.dbname),
  beacons = dbs.collection("beacons"),
  hourlystats = dbm.collection("hourlystats"),
  params = {};

var start = new Date((new Date()).getTime() - 1 * 60 * 60 * 1000 ); // 1 hour ago
start.setUTCSeconds(0);
start.setUTCMilliseconds(0);
start.setUTCMinutes(0);
var end = new Date(start.getTime() +  1 * 60 * 60 * 1000); // start plus an hour

params.timestamp = {
  "$gte": start,
  "$lt": end
};

beacons.mapReduce(
  mongofunctions.cronstatgenerator.map,
  mongofunctions.cronstatgenerator.reduce,
  {
    out: {inline:1},
    finalize: mongofunctions.cronstatgenerator.finalize, 
    query: params
  }, function (error, result) {
    console.log(error);
    result.results.forEach(function(item){
      //console.log(item);
      hourlystats.save(item);
    });
    process.exit();
  }
)
sajal commented 12 years ago

Would like to add... that i can run this mapreduce with inline output on the slave using the mongo console directly without any issues.

marcello3d commented 12 years ago

Mongolian doesn't support slave servers yet, so right now your only option would be to hack the Mongolian source.

sajal commented 12 years ago

Aha thanks... it turns out that my map-reduce job was taking hours to finish cause amazon throttles down micro instances big time, even when run from the mongo shell. So for now im doing my processing client side using python.