Add Support for Redis Cluster

[Jsyro]

Acapy can install a plugin that bolsters reliability on cloud deployments with a redis based persistent queue to manage incoming and outgoing messages, so that, in the case where a pod gets evacuated in the middle of a transmission, the new container will recover and continue the processing any interrupted messages.

AATH has support for managing supporting services (universal resolver, von-network, tails server etc), and those artifacts persist througout the entire test run, which simplifies the first automated testing at scale of this pattern.

This paradigm may expand to other implementations who also gain capabilities to use a redis cluster, so I'm curious if others think it is appropriate for AATH to include this option.

Expected outcome: On image build, a --redis option would be available, and if supplied would:

• Include the appropriate plugin to the acapy image build,
• Include the required plugin-config.yml file and a known path.

On docker run, a --redis option would be available, and if supplied would:

• Ensure that the redis cluster is available on the configured ports (and if not, start one)
• Provide the appropriate startup commands to the acapy agent, as far as I know the plugin makes no functional changes to the endpoints or procedures, so the backchannel itself needs no changes.

Please share any thoughts or suggestions on a more appropriate way to test this plugin's impact to aca-py.

[swcurran]

I like the idea, as this gives us a long term way to support the functionality. We are adding the flags at the harness level, but leave it to the Backchannel implementations to choose to support or not the use of Redis. Tests should run the same regardless of the presence of Redis, and a similar approach could be applied to other comparable capabilities. In the Runsets, the use of the service could be added or not, as appropriate.

I think this is implied, but there should be added the redis service on the ./manage service subcommand.

This is a deviation from the direct path to scaled testing or production deployment of Redis/PQ capabilities, but I think necessary for demonstrating how to use the capability and automated verification of the functionality.

[Jsyro]

Preliminary Run from this branch with -d acapy-redis.

Failing scenarios:
  features/0025-didcomm-transports.feature:28  Create DIDExchange connection between two agents with overlapping transports -- @2.1 DIDExchange connection with both agents using WS for inbound and outbound transport
  features/0183-revocation.feature:380  Issuer revokes a credential and sends a v1 revocation notification -- @1.1
  features/0211-coordinate-mediation.feature:106  Two agents creating a connection using a mediator without having inbound transports -- @1.1 Acme and Faber creating a DIDExchange connection using Bob as a mediator without having inbound transports
  features/0211-coordinate-mediation.feature:111  Two agents creating a connection using a mediator without having inbound transports -- @2.1 Acme and Faber creating a 0160 connection using Bob as a mediator without having inbound transports
  features/0453-issue-credential-v2.feature:63  Issue a JSON-LD credential with the Holder beginning with a proposal -- @4.1
  features/0453-issue-credential-v2.feature:68  Issue a JSON-LD credential with the Holder beginning with a proposal -- @5.1
  features/0454-present-proof-v2.feature:36  Present Proof of specific types and proof is acknowledged with a Drivers License credential type with a DID Exchange Connection -- @4.1
  features/0454-present-proof-v2.feature:41  Present Proof of specific types and proof is acknowledged with a Drivers License credential type with a DID Exchange Connection -- @5.1
  features/0454-present-proof-v2.feature:75  Present Proof of specific types and proof is acknowledged with a Citizenship credential type with a DID Exchange Connection -- @3.1   
  features/0454-present-proof-v2.feature:80  Present Proof of specific types and proof is acknowledged with a Citizenship credential type with a DID Exchange Connection -- @4.1   

5 features passed, 5 failed, 4 skipped
85 scenarios passed, 10 failed, 58 skipped
722 steps passed, 10 failed, 588 skipped, 0 undefined
Took 43m22.186s

[Jsyro]

Comparison run from the same branch with -d acapy-redis

Failing scenarios:
  features/0023-did-exchange.feature:45  Establish a connection with DID Exchange between two agents with an implicit invitation
  features/0025-didcomm-transports.feature:28  Create DIDExchange connection between two agents with overlapping transports -- @2.1 DIDExchange connection with both agents using WS for inbound and outbound transport
  features/0211-coordinate-mediation.feature:106  Two agents creating a connection using a mediator without having inbound transports -- @1.1 Acme and Faber creating a DIDExchange connection using Bob as a mediator without having inbound transports
  features/0211-coordinate-mediation.feature:111  Two agents creating a connection using a mediator without having inbound transports -- @2.1 Acme and Faber creating a 0160 connection using Bob as a mediator without having inbound transports
  features/0453-issue-credential-v2.feature:63  Issue a JSON-LD credential with the Holder beginning with a proposal -- @4.1
  features/0453-issue-credential-v2.feature:68  Issue a JSON-LD credential with the Holder beginning with a proposal -- @5.1
  features/0454-present-proof-v2.feature:36  Present Proof of specific types and proof is acknowledged with a Drivers License credential type with a DID Exchange Connection -- @4.1
  features/0454-present-proof-v2.feature:41  Present Proof of specific types and proof is acknowledged with a Drivers License credential type with a DID Exchange Connection -- @5.1
  features/0454-present-proof-v2.feature:75  Present Proof of specific types and proof is acknowledged with a Citizenship credential type with a DID Exchange Connection -- @3.1   
  features/0454-present-proof-v2.feature:80  Present Proof of specific types and proof is acknowledged with a Citizenship credential type with a DID Exchange Connection -- @4.1   

5 features passed, 5 failed, 4 skipped
85 scenarios passed, 10 failed, 58 skipped
722 steps passed, 10 failed, 588 skipped, 0 undefined
Took 43m16.144s

Slightly different failing scenarios, but overall success. Will work to clean up the management of the images and the redis service.

[Jsyro]

https://github.com/hyperledger/aries-agent-test-harness/pull/680

Looking for review.

[Jsyro]

PR merged, closing issue.