search

jeffcjlee / ylplines

Yelp Trends
GNU General Public License v3.0
1 stars 1 forks source link

No restrictions on fetching amount can cause getting banned #8

Closed jeffcjlee closed 8 years ago

jeffcjlee commented 8 years ago

A user can click to fetch multiple businesses at a time, and the server will process them independently without restrictions.

If they chose to fetch 10 businesses with each having large # reviews, it can cause too many requests to Yelp and get the site banned.

jeffcjlee commented 8 years ago

I see some good things about leveraging Celery and Redis in order to create a async job queue that runs in background. This probably is the way to. Look more into it.

jeffcjlee commented 8 years ago

Fixed.

The workflow has been converted with a task queue system. Now, fetching businesses are individual tasks that wait in a queue. The daemon only does one task at a time. The client polls the server every 5 seconds for if their respective task is completed. No longer hitting time outs.