Investigation: Acapy status endpoints can hang up during some operations and cause probe failures

[loneil]

I noticed that /status/live and /status/ready will hang and not return when doing certain longer running operations like creating a tenant or registering a did, or even in a smaller way while getting a token.

https://traction-acapy-admin-dev.apps.silver.devops.gov.bc.ca/api/doc#/server/get_status_ready

So since those are the k8s probes that will fail readiness/liveliness and potentially kill the pod (and crash out Acapy) if they occur enough (which happens more readily in lower resource envs like PRs, but will occur in others as well).

So seems important to figure out if this is:

• A known issue in Acapy (that maybe needs resolving)
• Possibly related to any plugin implementation
• Should the status endpoints not be treated as reliable and the k8s probes be something else like just / instead. Or the timeout adjusted way higher?
• Anything relate to Askar? Don't think we saw this behaviour before but we weren't really doing heavy usage.

[loneil]

@usingtechnology has discussed. At the least, going with a higher probe timeout like other Acapy projects use, since these calls do seem to return, just hang for extra time while these operations are happening. Can see later if Aca-py ticket is needed?

[usingtechnology]

talked with @WadeBarnes and confirmed he has previously deployed using liveness probe time of 10 seconds. we had 2 seconds. this timeout/pod restore happened with 3 seconds in test namespace so did a PR that bumped to 10 - seems to be ok.

[usingtechnology]

see PR 473