search

xtracthub / xtract-service

Globus Labs Xtract: Extract metadata from distributed data sets.
6 stars 1 forks source link

Smarter polling prioritization #59

Closed tskluzac closed 2 years ago

tskluzac commented 3 years ago

Polling across all tasks sequentially really isn't working. For instance, if you get past the oldest tasks, a lot of the early-polled tasks will finish while the 'just submitted' tasks are polled. I think the 'queue' needs to be changed to a priority queue, probably based on (initially):

[] age [] extractors to be applied

tskluzac commented 2 years ago

Closing as we now handle this in the xtract-service (after crawl)