taskqueues rate limits are not respected

[GoogleCodeExporter]

What steps will reproduce the problem?
1. In queue.yaml put:
queue:
- name: default
  rate: 1/s

- name: maps-api
  rate: 2500/d

2. Queue something in the maps-api
3. See how it's consumed as fast as it can, instead of at the rate of 2500/d.

Original issue reported on code.google.com by e98cu...@gmail.com on 1 Oct 2010 at 6:14

[GoogleCodeExporter]

Unfortunately, TyphoonAE currently does not interpret the queue.yaml file. 
Therefore, our implementation of the Task Queue service, based on RabbitMQ, 
handles tasks only in two ways:

1) Immediate task execution:

Each task gets executed as soon as possible in the exact order in which it is 
enqueued.

2) Deferred tasks:

If a Task instance provides an 'eta' it is automatically sent to the 'deferred 
queue', which guarantees an absolute time when the task should execute.

Task Queue configuration at a more granular level (e.g. via queue.yaml) is 
definitely on our roadmap, though. Could you please explain in which way the 
current behavior of our implementation compromises your application? This would 
definitely help to better understand different use-cases.

Thanks a lot for giving TyphoonAE a go and for reporting this issue!

Original comment by tobias.r...@gmail.com on 2 Oct 2010 at 12:40

• Changed state: Accepted

[GoogleCodeExporter]

For the sake of documenting the reasons that not respecting the rate limit may 
impact the application, I currently have a queue with a limit because it makes 
queries to another service (Google Geocoder) that has a rate limit of 2500 
queries / day.

Another case where ignoring rate limit may impact your application is if you 
hit a backend that you know may only handle a maximum of a few queries / 
seconds.

Cheers,

Original comment by e98cu...@gmail.com on 14 Oct 2010 at 8:07

[GoogleCodeExporter]

Thanks for clarifying! These are indeed good reasons to use Celery as default 
Task Queue backend.

Original comment by tobias.r...@gmail.com on 14 Oct 2010 at 10:04

• Changed state: Started