search

vsivsi / meteor-job-collection

A persistent and reactive job queue for Meteor, supporting distributed workers that can run anywhere.
https://atmospherejs.com/vsivsi/job-collection
Other
389 stars 67 forks source link

Loadbalanced Meteor server setup #188

Open jorisroling opened 8 years ago

jorisroling commented 8 years ago

In a settup where a Meteor app is running on 3 servers (loadbalanced), and 3 workers (using your node client) would connect to each one of them, would then every server know of the 3 workers? (I presume your answer will be 'no').

And would that be a problem?

It look like your try to make MongoDB the arbiter (due to the atomic nature of certain DB actions). Is that true? Do I understand correctly that a worker decides to take a job, not so much that the server has to know it's workers and hand down a job to that specific worker (in wich case the scenario of 3 Meteor servers, a 3 workers connecting to each one of them) would not really work (well).

Any thoughts?

vsivsi commented 8 years ago

Hi, so long as the job-collection on each of the Meteor servers is backed by the same MongoDB collection running on a single MongoDB instance, then it should work fine. The workers can request work from whichever server they are connected to. You are correct that the database is the arbiter through its role as the "source of truth".

If you wish to add redundancy to the MongoDB database as well (through replication and/or sharding) then things will get more complicated, because you suddenly need to care about "write concern" and potentially many other issues related to distributed databases. But so long as you only have a single MongoDB server, none of that complexity comes into play.

jorisroling commented 8 years ago

Actually, that is our situation: we also have e 3 server replica set. How would the write-concern need to be setup in your opinion? I have it on 'majority' now (in the MONGO_URL).

Is the potential problem that more than one worker could start the same job?

vsivsi commented 8 years ago

Majority write concern is sufficient right up until one of your replicas goes down or becomes disconnected from the other two... Honestly, consulting on how to properly architect replicated MongoDB deployments is way out of scope for giving advice on a Github question about a specific package like job-collection.

MongoDB is notoriously finicky about this stuff, and achieving reliable consistency (which is important for a service like job-collection) is highly dependent on how you set it up, and which versions of everything (drivers and server) you are using. My understanding is that this has improved somewhat with MongoDB 3.2 and the latest 2.1 node drivers, but Meteor, notably, still uses the old 1.4.x drivers, so YMMV.

If you are not already well-versed in these issues, these articles should convince you that getting this stuff right has been easy to screw-up in the past, and remains pretty tricky: https://aphyr.com/posts/322-jepsen-mongodb-stale-reads https://aphyr.com/posts/284-call-me-maybe-mongodb

And that these types of issues can affect otherwise well-architected scheduling packages that rely on a database as the "source of truth". https://aphyr.com/posts/326-call-me-maybe-chronos

janat08 commented 7 years ago

So 3.4 is out for meteor.

janat08 commented 7 years ago

Theres also read concern now. https://docs.mongodb.com/manual/reference/read-concern/#readconcern."majority"

janat08 commented 7 years ago

@vsivsi should be intergrated into api i think.

vsivsi commented 7 years ago

Everyone's MongoDB deployment is going to be different. I'm pretty committed to keeping those types of details outside of this package. As I indicated in my answer above, sharded/replicated MongoDB is a tricky and a bit of a minefield. This package will only ever be as good as your deployment and driver setup, and they are both out of scope for my support of the package.