lisac
commented
3 months ago approach and findings from attempt # 1 to dev
Approach: triggered deployments of svc-bip-api and domain-ee-ep-merge-app to dev, using b995774 (latest on default branch at the time); observed an error in svc-bip-api app logs:
2024-08-14T22:00:09.275Z ERROR 1 --- [20.223.225:5672] o.s.a.r.c.CachingConnectionFactory : Shutdown Signal: channel error; protocol method: #method<channel.close>(reply-code=406, reply-text=PRECONDITION_FAILED - inequivalent arg 'auto_delete' for queue 'vroDeadLetterQueue' in vhost '/': received 'true' but current is 'false', class-id=50, method-id=10)
Restarted the apps (using Restart option on the respective Deployments in Lens), to see if that might have any effect; however, we continued to see the Shutdown Signal in svc-bip-api logs. (ep-merge logs were quiet, a consequence of no requests being fielded in dev).
In debugging, we inspected the queues in RabbitMQ. sample commands executed from the vro-rabbitmq pod, using guest:guest as the RabbitMQ user:password (actual values can be discovered when inspecting the pod):
# get the list of queues, displayed as pretty json.
$ curl -s -u guest:guest http://localhost:15672/api/queues | json_pp
# look for existence of a specific queue, such as updateClaimContentionsResponseQueue
$ curl -s -u guest:guest http://localhost:15672/api/queues | json_pp | grep updateClaimContentionsResponseQueue
# look for existence of a specific queue and display surrounding lines
$ curl -s -u guest:guest http://localhost:15672/api/queues | json_pp | grep -B 10 -A 10 updateClaimContentionsResponseQueue sample command to delete the BIP queues (7) (see also: https://github.com/department-of-veterans-affairs/abd-vro/blob/develop/svc-bip-api/src/main/resources/application.yaml#L35) :
curl -X DELETE -u guest:guest --location 'http://localhost:15672/api/queues/%2f/getClaimDetailsQueue' --header 'Content-Type: application/json'
curl -X DELETE -u guest:guest --location 'http://localhost:15672/api/queues/%2f/putClaimLifecycleStatusQueue' --header 'Content-Type: application/json'
curl -X DELETE -u guest:guest --location 'http://localhost:15672/api/queues/%2f/cancelClaimQueue' --header 'Content-Type: application/json'
curl -X DELETE -u guest:guest --location 'http://localhost:15672/api/queues/%2f/getClaimContentionsQueue' --header 'Content-Type: application/json'
curl -X DELETE -u guest:guest --location 'http://localhost:15672/api/queues/%2f/createClaimContentionsQueue' --header 'Content-Type: application/json'
curl -X DELETE -u guest:guest --location 'http://localhost:15672/api/queues/%2f/updateClaimContentionsQueue' --header 'Content-Type: application/json'
curl -X DELETE -u guest:guest --location 'http://localhost:15672/api/queues/%2f/putTempStationOfJurisdictionQueue' --header 'Content-Type: application/json' sample code to delete the DLQ:
curl -X DELETE -u guest:guest --location 'http://localhost:15672/api/queues/%2f/vroDeadLetterQueue' --header 'Content-Type: application/json' After trying a number of things (argh, I didn't document the exact sequence - but at one point included deleting BIP and DLQ queues and re-starting apps) - we re-inspected the Shutdown Signal (posted near top of this comment), in particular this part:
inequivalent arg 'auto_delete' for queue 'vroDeadLetterQueue' in vhost '/': received 'true' but current is 'false'
This led to finding an issue in the queue declaration. potential fix in #3308.
Plan for next steps
- merge in #3308
- on
dev: delete thevroDeadLetterQueuefrom rabbitmq OR redeploy rabbitmq (theory: the latter would have a new volume) - the point is to ensure there is not avroDeadLetterQueuealready in-place, and in theory the app code can create the queue without contention - deploy
svc-bip-apitodev; observe app logs for errors - deploy
ep-mergetodev; observe app logs for errors (although we don't expect to see any, due to the app not receiving traffic on thedevenvironment.
BerniXiongA6
summary
We anticipate potential complexities with deploying the dead letter queue feature (#3238). In particular, it might be necessary to sequence deployments of
svc-bip-apiandep-mergein a particular order so that downtime can be minimized, if not avoided; and coordination might also be required with the RabbitMQ service. This ticket is is to track the approach and progress, for the purposes of 1) accounting for the work during the sprint; and 2) informing future situations where a change to RabbitMQ behavior needs to be deployed.VRO planned to deploy this change on 8/14.
tentative plan of action
(source: #benefits-vro-engineering 8/14)