This is a major refactor of the batch job submission and updating. I have renamed the table to "Processing Job Table" and removed all references to "status tracking".
I find this logic more serial and easier to follow, but please let me know if you find this harder to follow or anything isn't clear. I tried to avoid the upstream/downstream terminologies when used together and instead switched to "potential jobs" for the downstream case. So we iterate through potential jobs. For each potential job we look "upstream" to see if the required files are available. If any are missing, abort. If all are available, try to submit the record. The record submission will fail if it isn't unique, so we just abort and log that there was already a record for that case.
I also added the unique ID from the table to the job-name so that we can directly get the proper record to update in the database. Before we were just getting the first, so not guaranteed to hit the right job to update if we had multiple failures or attempts.
This builds upon #308 with the final two commits here being relevant for review. Refactoring the batch starter and the indexer functions.
Change Summary
Overview
This is a major refactor of the batch job submission and updating. I have renamed the table to "Processing Job Table" and removed all references to "status tracking".
I find this logic more serial and easier to follow, but please let me know if you find this harder to follow or anything isn't clear. I tried to avoid the upstream/downstream terminologies when used together and instead switched to "potential jobs" for the downstream case. So we iterate through potential jobs. For each potential job we look "upstream" to see if the required files are available. If any are missing, abort. If all are available, try to submit the record. The record submission will fail if it isn't unique, so we just abort and log that there was already a record for that case.
I also added the unique ID from the table to the job-name so that we can directly get the proper record to update in the database. Before we were just getting the first, so not guaranteed to hit the right job to update if we had multiple failures or attempts.
This builds upon #308 with the final two commits here being relevant for review. Refactoring the batch starter and the indexer functions.