Data Service Driver - auto replenish queued tasks

marklogic-community / marklogic-state-conductor

An event-based state-machine engine for manipulating MarkLogic database documents.

Other

1 stars 4 forks source link

Data Service Driver - auto replenish queued tasks #80

Closed aclavio closed 4 years ago

aclavio commented 4 years ago

The Data Services Driver should utilize an auto replenishing queue of state conductor jobs .

Currently the queue of jobs is only replenished after all batches are completed by the ExecutorService. This means that a single long-running batch can cause the other executors to starve.

aclavio commented 4 years ago

created branch /feature/80-auto-replenish-tasks

aclavio commented 4 years ago

To prevent the task queue from being filled up with duplicate tasks, the getJobs API will need to be updated to support pagination. This means it will need to support a "start" parameter in addition to the existing "count" parameter. The response will also need to be changed to a json object with a "total" key indicating the total number of jobs, and "jobs" key containing the list of job uris.