Mediator deployment with REDIS

[esune]

Deployment of an ACA-Py mediator with REDIS persistent queue plugin.

[WadeBarnes]

Note from yesterday:

I've deployed and tested a mediator with a persistent queue (PQ) (connected to a Redis cluster). It's working though performance isn't great at the moment as I need to tune (increase) the resources on the Redis cluster. I'll let people know when that's done. I've also got a little cleanup to do on the updated mediator configurations before the PR goes in. So far the PQ instance is slower than the regular instance, but seems to be more stable (consistent). I'm expecting some of the slowness to go away once the Redis cluster has more resources.

[WadeBarnes]

I've got the new persistent queue mediator tuned and stable. All pods in the mix have very low throttling percentages. 8% and lower for the Redis luster pods, and <1% for the mediator pods. I've also turned down the logging levels on ACA-Py to squeeze as much out of it as possible. Invite provided to team.

I was getting some decent performance out of it this morning until the Locust container started to fall over.

Since then there's a few thousand more connection records in the secure storage and the performance is lower, but so far fairly stable.

I had to add a second CPU the mediator agent with the introduction of the Redis plugin. Without the second CPU the pod was going over allocation and being throttled over 60% of the time. Following the addition of the second CPU, throttling dropped to under 1%.

Environment settings:

Number of users: 200
Spwan Rate: 25

LOCUST_FILES=locustMediatorPing.py
LOCUST_MIN_WAIT=0.05
LOCUST_MAX_WAIT=0.1

aries-mediator-agent-pq:
        - resources:
            limits:
              cpu: '2'
              memory: 512Mi
            requests:
              cpu: '2'
              memory: 256Mi

aries-mediator-db-pq:
        - resources:
            limits:
              cpu: '1'
              memory: 512Mi
            requests:
              cpu: '1'
              memory: 256Mi

aries-mediator-proxy-pq:
        - resources:
            limits:
              cpu: '1'
              memory: 512Mi
            requests:
              cpu: '1'
              memory: 256Mi

redis-shared (6 nodes):
        - resources:
            limits:
              cpu: 250m
              memory: 256Mi
            requests:
              cpu: 125m
              memory: 64Mi

Longer run test results:

requests_1685035542.126043.csv failures_1685035543.9566092.csv exceptions_1685035544.99205.csv

[WadeBarnes]

A little bit of cleanup left to do.

[esune]

@WadeBarnes I believe this is complete. Can you please confirm/close?

[WadeBarnes]

Yes, complete now