thammegowda
commented
8 years ago My opinion: Priority(basic things) > Priority(fancy features)
My definition of basics:
- Horizontally scalable: distributes the workload to all nodes in the cluster
- Open source: licence is Apache Licence 2.0 or atleast compatible with it.
- Portable: the only assumption about the environment is a standard Java Runtime and nothing more.
In other words, I love to get some JS engine working within JVM. Thus we can distribute the load to all the nodes in the cluster. If that is not possible, the second option would be to proxy to an HTTP API (assuming the JS execution is still distributable behind the proxy).
@smadha Come on board, research about the options, and get us some working piece of code. Thanks!
smadha
Are there any thoughts on this? Are we planning to use Splash / Selenium / phantomjs ?
https://github.com/scrapinghub/splash