search

Netflix-Skunkworks / Scumblr

Web framework that allows performing periodic syncs of data sources and performing analysis on the identified results
Apache License 2.0
2.64k stars 317 forks source link

Question: is that possible to run few Scumblr instances at one box? #148

Closed annbyl closed 7 years ago

annbyl commented 7 years ago

Hi! We have a need to run >2000 queries, and as far I understand, there's a limitation in Google CSE on # of queries or other limit. Has someone performed such an implementation for large # of queries? Thanks in advance, Anna.

ahoernecke commented 7 years ago

Hi Anna,

CSE does have limits, (https://developers.google.com/custom-search/json-api/v1/overview#pricing). It looks like currently it's 100/per day for free. However there are paid options if you want to search more and that works fine with Scumblr's google task.

As far as the question in the title, I'm not sure what you're asking. I think you could run separate instances of Scumblr assuming they're on separate ports, but it wouldn't really help with the Google api limits.

Please feel free to jump into our gitter channel if you'd like to discuss further: https://gitter.im/Netflix/Scumblr