Scalability and background tasks discussion

[ido43210]

Hi, I have several concerns and questions (more of a discussion then an issue).

1. Scalability
2. Background tasks

The first is a question about the proper way to scale up the parse server. Is using the nodejs vanila cluster module will suffice, or you would suggest using cloud base scalability feature like the heroku apps. (In the LiveQuery wiki you suggested a diagram that uses redis as the intermediate). Link: [(https://devcenter.heroku.com/articles/node-concurrency)]

The second is more of a suggestion to add a background task scheduler like node-resque [https://github.com/taskrabbit/node-resque] natively to the parse server. And make it accessible to the cloud code. The benefit for using a redis base background queue with the parse server is incredible, especially with the new LiveQuery feature. The use case I had in mind is incorporating with the parse server a high-cpu task that the client needs like image-processing, now the process I had in mind is:

1. The client can save a request with an image file and subscribe to changes via LiveQuery.
2. The parse object will be saved and will have a beforeSave hook in the cloud code.
3. Inside the hook the server will queue a long cpu task to process this image.
4. After the processing is complete (in the same machine or maybe another) the task will end with saving this in a result object that the client subscribed too.
5. Now using the LiveQuery (and or push) the client will receive the processed image.

P.S. really looking forward using parse server in production, the android+ios native SDKs really speeds us developing native apps.

[goggelj]

To add to the above, the questions I have regarding scalability are:

1. On Parse.com I have a slider that allows me to increase workers/requests per second. How is this scaling managed? Does it actually create another vm (shared or other) behind nginx?
2. Is there a cluster code that has not been incorporated into parse-server code? If so, what's the eta? If not what is the recommendation? Heroku's docs suggest throng but then what is the use of dynos (other than incurring higher costs)?

[flovilmart]

I suggest that you move this question to the Stack Overflow and Server Fault communities as you may find better answers than what we can provide here.

For the background queues etc... I don't believe we'll add another dependency to it as you can very easily manage it yourself inside your cloud code code by importing the module you like.

There are so many background queue implementations that choosing one I believe will create a disservice to the community. Some users would want to use AWS SQS, others redis, others rabbitMQ or zeroMQ etc...

For the clustering, you can use PM2 or another process manager that would be responsible for it, but there is no plan to add it to parse-server.

Again, those are valid questions but the Stack Exchange communities are more suited.

[goggelj]

Could you at least comment on what happens in Parse when you use the 'scale' slider?

[flovilmart]

I believe this it toggles the billing and a software limiter on API requests, but that don't seem relevant here as there is no such sliders on parse-server

[goggelj]

hmmm....but how is actual scaling takes place in parse.com? Do you know? If the slider creates threads to handle more requests on a single vm, then somewhere in parse-server there is a throng like feature that picks up on number of threads. If not then somewhere it adds a vm behind nginx (or similar). Thoughts?

[flovilmart]

Again, you can discuss that on stackoverflow as there is multiple technics and strategies to handle scaling. One that applied to parse.com may not apply to your use case

[otymartin]

@flovilmart As a GCP user on app engine, in regards to scalability, I'm under the assumption that App Engine will automatically to handle any load i'm hit with. Am I right with this assumption (I have no server-side knowledge).