load_jobs first tries to do a bulk insert into the database with all the jobs. This can fail in two cases: (1) there are too too many jobs, (2) some of the jobs to be inserted already exist in the database. If bulk insert fails, load_job reverts to inserting one job at a time, which is slow.
Proposed change:
If bulk commit fails, revert to "batched commits," wherein batches of jobs (say 1000 jobs at a time) are insert into the database. By using the IGNORE keyword, duplicates can be handled.
alexandraBara
commented
4 months ago
load_jobs first tries to do a bulk insert into the database with all the jobs. This can fail in two cases: (1) there are too too many jobs, (2) some of the jobs to be inserted already exist in the database. If bulk insert fails, load_job reverts to inserting one job at a time, which is slow.
Proposed change:
If bulk commit fails, revert to "batched commits," wherein batches of jobs (say 1000 jobs at a time) are insert into the database. By using the
IGNOREkeyword, duplicates can be handled.