Within job launch logic there was a helper method which would query the job status and based on the returned value proceed with some logic to either update it or fall back to other logic. This works ok if all requests to the persistence service implementation go to a single cosistent backend. If, however, read only queries go to a read replica which may have lag or some other implementation entirely this breaks down without the service actually knowing why or how.
Moving the logic for this behind the persistence API and letting the launch service only act the returned job status from the source of truth api should fix this problem.
Coverage decreased (-0.01%) to 93.788% when pulling 556028cbd02bc7cf136a903f382ff54aacd46544 on tgianos:jobStatusFix into cfe9cfd0dfee750d0778936782472439d4e70719 on Netflix:master.
Within job launch logic there was a helper method which would query the job status and based on the returned value proceed with some logic to either update it or fall back to other logic. This works ok if all requests to the persistence service implementation go to a single cosistent backend. If, however, read only queries go to a read replica which may have lag or some other implementation entirely this breaks down without the service actually knowing why or how.
Moving the logic for this behind the persistence API and letting the launch service only act the returned job status from the source of truth api should fix this problem.