conveyal / otpa-cluster

Cluster backend for otpa many-to-many queries.
0 stars 1 forks source link

Don't let queue size grow when processing single point requests #36

Closed mattwigway closed 9 years ago

mattwigway commented 9 years ago

When processing single point requests, a particular worker will often get one cluster request at a time and be idle when not processing. This is fine. However, the code that autoscales the queue size assumes that if it emptied its queue the queue was not long enough. However, it's often the case that the queue was simply not full. Thus, if you run a bunch of single point requests, the queue size will eventually hit the ceiling. If this worker is then reassigned to a multipoint job, its first batch will be much more than it can handle, jobs will be retried, etc. Chaos will be the upshot.