EP Merge: Investigate/Fix possible bug in resume jobs logic

[dfitchett]

Summary

While investigating TTL behavior, I noticed some weird behavior that I'd like to look into regarding jobs that are in progress while EP Merge is starting.

Acceptance Criteria

• [x] document results of investigation and procedures to repeat
• [x] implement fix
• [x] update unit tests
• [x] update integration tests

[dfitchett]

Investigation

The following undesired behavior was noticed:

• Currently there is a 5 second delay in resuming any in-progress jobs that are in the database when the EP Merge Application is started. This 5 second delay is meant to allow time for the resources (RabbitMQ and RDS) to become available. Due to the way program is started, the application will begin accepting requests as soon as those resources are available.
• If vets-api sends a request after the resources are available, but before this 5 second period is up, then the job that is already in progress will start again on a separate thread, and the thread that finishes first will have the final write to the database, with every call to BIP/BGS being duplicated. This can result in unexpected duplicate contentions and claim notes.
• This bug is a race condition that is very unlikely to happen with minimal traffic going to the EP Merge App. However, with more expected traffic going to the application, we risk encountering the behavior with every new deployment.

Steps to repeat:

1. Start all platform services & BIP. Note BGS is not started to allow sending a requests during the same time

   COMPOSE_PROFILES="platform,bip" ./gradlew :dockerComposeUp
   COMPOSE_PROFILES="bip,bgs" ./gradlew -p mocks dockerComposeUp

2. Within 5 seconds sent a merge request to the /merge endpoint with the following body:

   {
   "pending_claim_id": 10000,
   "ep400_claim_id": 10001
   }

3. The request will be accepted and will likely run to the stage where the request is made to BGS to add the claim note.
4. After the initial 5 seconds is up, the app will attempt to resume in-progress jobs and will log a statement showing event=resumeJobsInProgress status=started total=1
5. The job with the same id as the request that was sent will be "resumed" in a separate thread and you'll see duplicate requests being sent for each step of the job process.