search

singhj / locality-sensitive-hashing

MIT License
28 stars 11 forks source link

FIFOQueue running on multiple machines #3

Open singhj opened 10 years ago

singhj commented 10 years ago

The current implementation of FIFOQueue is based on in-memory structures. Since App Engine can spin up multiple server instances, those server instances will not share the memory and you have to give up on the asynchronous features.

We'll need to create a FIFOQueue version based on Task Queues. Another alternative is the Pipeline API but Task Queues seems simpler and more portable.

tbrooks007 commented 10 years ago

Just saw this, sorry I missed this comment before. Yep you are right. For a demo that doesn't use contrived (faux) data (like using tweets) we will need something like the Task Queues or Pipeline API. Just note though that a while ago I tested running this in Google App Engine and it works..for a more scalable solution though you are right. We need something else.

I will look into using Task Queues or Pipeline API. Once I have LSH working in a non-flakey way I'll look into this as we'll need it for nay work we do with tweets.