singer-io / tap-marketo

GNU Affero General Public License v3.0
9 stars 17 forks source link

add backoff for job queue full errors #93

Open haleemur opened 9 months ago

haleemur commented 9 months ago

Description of change

Queue

The bulk extract APIs use a job queue (shared between leads, activities, program members, and custom objects). Extract jobs must first be created, and then enqueued by calling Create Export Lead/Activity/Program Member Job and Enqueue Export Lead/Activity/Program Member Job endpoints. Once enqueued, the jobs are pulled from the queue and started when computing resources become available.

The maximimum number of jobs in the queue is 10. If you try to enqueue a job when the queue is full, the Enqueue Export Job endpoint will return an error “1029, Too many jobs in queue”. A maximum of 2 jobs can run concurrently (status is “Processing”).

Manual QA steps

Risks

Rollback steps