search

Rostlab / JS16_ProjectF

In this project we will build a web portal for our GoT data analysis and visualization system. The website will integrate all the apps created in projects B-D with the help of the integration team assigned to Project E.
GNU General Public License v3.0
10 stars 8 forks source link

added multicore support #356

Closed yashha closed 8 years ago

yashha commented 8 years ago

Closes #51

yashha commented 8 years ago

Should we run only one instance of d4 and d5 or it doesn't matter? At the moment I get often: [crawler/mobile][WARN] Too Many Requests - Retry ... @CavidSalahov @jonny3576

sacdallago commented 8 years ago

Mannnnn... I don't know... :D I love multicore, don't get me wrong, but I think for the twitter things you would need something like as many twitter keys as cores you run the thing on... Sooooo 16 cores = 16x2 twitter keys :D :D

What you could do is the following, very interesting actually:

  1. Take the num of cores, like you already do. If this number is > 2, then
  2. reserve 1 fork for the first twitter package and
  3. 1 core for the second twitter package (they anyway always run in the background, right?)
  4. Use the other forks to handle HTTP requests/servicing sites.

I guess you would need to take care of understanding which worker died in that case, so that you respawn the right one.

jcebel commented 8 years ago

Yes D5 always runs in the background. And I think this depends on your infrastructure, if you have to run them on several cores, but it is not inevitable needed, as far as I understood D5. @sacdallago's suggestion sounds reasonable. Maybe you think of using two Twitter accounts (request limit blablabla). So D4 and D5 do not interfere with each other.

sacdallago commented 8 years ago

Sorry @yashha . The idea was really good, the code also, but this needs some more thought :) Future work, future work