search

phantombuster / nickjs

Web scraping library made by the Phantombuster team. Modern, simple & works on all websites. (Deprecated)
https://nickjs.org
ISC License
500 stars 48 forks source link

I have a question for many requests #57

Closed msaus closed 4 years ago

msaus commented 5 years ago

Hello,

Do you use docker for many requests?

For example, do you create multiple docker instances in 1 server and handle many requests?

Best regards,

paps commented 5 years ago

Hey,

Yes, that's basically what our service Phantombuster does. We have a cluster of servers that each have a docker deamon and load balancing / management code to handle everything.

Regards

msaus commented 5 years ago

Hello there,

Thanks for that.

To load balancing, do you use nginx or litespeed?

Or, other middleware?

regards,

paps commented 4 years ago

In production we use AWS ALB to load balance incoming requests. Then we have a custom made load balancer to select the correct VM in which to launch a new docker container with Chrome inside.