department-of-veterans-affairs / abd-vro

To get Veterans benefits in minutes, VRO software uses health evidence data to help fast track disability claims.
Other
19 stars 6 forks source link

EP Merge: Set TTL on requests sent to downstream VRO queues #3175

Closed dfitchett closed 1 month ago

dfitchett commented 2 months ago

Summary

Currently messages that are sent from EP Merge to any VRO service that is down will remain in the queue until that service comes online and consumes the messages from the queue. EP Merge will always reject the responses in those cases if they have reached the max latency set for the clients in the hoppy_service.py which is 30s.

Possible Effects:

None of the GET endpoints would have any adverse effects, but there can be issues if the requests are modifying downstream data. For all of the cases below, actions could be completed in any order even though the merge job might have been completed unsuccessfully:

Temporary solution

Request that VRO apply queue level TTL to 0 seconds for all BGS/BIP queues used by EP Merge.

Best / Long term solution

Update lib-hoppy to send requests to queue with per message TTL option set to client's max-latency value.

Acceptance Criteria

Image