yalelibrary / YUL-DC

Preliminary issue tracking for Yale University Libraries Digital Collections project
3 stars 0 forks source link

Segment large batch processes #2873

Open K8Sewell opened 4 months ago

K8Sewell commented 4 months ago

Story As described in a comment in #2859, sometimes a job may time out and fail before all records in a CSV are processed. This causes some jobs to run multiple times. We would like to change the batch process behavior to process CSVs in segments of 50 rows at a time, to prevent process from timing out and re-running.

This behavior should be applied to the following batch processes:

Acceptance The following jobs run in segments of 50 rows until completion:

Engineering Notes Jobs that have batching patterns to pull from: