search

chnm / serendipomatic

http://serendipomatic.org/
26 stars 9 forks source link

run api searches in parallel to avoid unnecessary slowdowns #55

Open rlskoeser opened 11 years ago

rlskoeser commented 11 years ago

have a preliminary implementation of this with gevent in a branch; just need to do some testing to see if it works under apache and/or on heroku

rlskoeser commented 11 years ago

I have working code in a separate branch b137e6cab611759acd98f2834a59f9a1d3465e52 and it seems to be working quite well; this also should add timeout functionality in case an api request hangs.

My only concern is still the apache/mod wsgi config. I'd like to be able to test it on a non-production apache server to make sure I've configured it correctly, although I'm not certain what the risk is... It's possible that if it's configured correctly we just lose the concurrency, but still gain timeouts and don't get any errors, but I'm not certain.

mialondon commented 11 years ago

Does it need to be an AWS server for best replicability? Can we image the one we have and clone it for testing?

rlskoeser commented 11 years ago

Not certain, I think any apache modwsgi set up should be close, but there are some peculiarities to our AWS server setup that I haven't completely ironed out. I may try setting it up on my dev linux box at work when I have the time.

mialondon commented 10 years ago

Is there anything we can do to help test this and close the call?

rlskoeser commented 10 years ago

I tried testing gevent under apache a while back and it seemed to hang indefinitely. I think it might be related to the C library gevent uses, and it's possible the 1.0 release of gevent might help as I believe they switched from libevent to libev (there's an RC available). It's also possible that if I spent more time fiddling with the configuration/setup for gevent or apache I could make it work, but I'm not sure.

mialondon commented 10 years ago

I've come back to this because discussion at an event on sustainability in digital history made me realise we should try to make sure the whole thing doesn't fall over if one API goes.

@rlskoeser should we try to find another solution if the one you were looking it isn't looking viable?