Increasing the tests run by AFJ

swcurran commented 1 year ago

Currently, the AFJ runset (the tests run through GHActions for AFJ) are limited to those selected as supported back in 2021 or so. Since then Aries Framework JavaScript has been substantially expanded. I wanted to see what AFJ would look like if (almost) all the AATH tests were run in an all AFJ scenario -- the same set run by ACA-Py. The following shows the results.

To get many of these to run, some effort is needed:

Pass through the list of executed tests and see which ones are failing because the AFJ does not support the necessary functionality, and revise the set of tags to be included/excluded.
Pass through the test results and see which tests are failing because the AFJ backchannel does not support the test, and update the backchannel to enable the test to be run.
Pass through the failing tests and determine which ones are failing because:
- The test is wrong
- The backchannel is wrong
- AFJ is wrong
- The RFC upon which the test was written is wrong/unclear

From there, we can update the runsets for AFJ with the other Aries Frameworks and verify/fix the interop between AFJ and other Aries components.

Failing scenarios:
  features/0023-did-exchange.feature:9  Establish a connection with DID Exchange between two agents with an explicit invitation
  features/0023-did-exchange.feature:27  Establish a connection with DID Exchange between two agents with an explicit invitation with a public DID
  features/0023-did-exchange.feature:45  Establish a connection with DID Exchange between two agents with an implicit invitation
  features/0023-did-exchange.feature:65  Establish a connection with DID Exchange between two agents with attempt to continue after protocol is completed
  features/0025-didcomm-transports.feature:23  Create DIDExchange connection between two agents with overlapping transports -- @1.1 DIDExchange connection with both agents using HTTP for inbound and outbound transport
  features/0025-didcomm-transports.feature:35  Create DIDExchange connection between two agents with overlapping transports -- @3.1 DIDExchange connection with both agents using WS for inbound and outbound transport, but also supporting http for outbound transport
  features/0025-didcomm-transports.feature:40  Create DIDExchange connection between two agents with overlapping transports -- @4.1 DIDExchange connection with both agents using HTTP and WS for inbound and outbound transport
  features/0025-didcomm-transports.feature:41  Create DIDExchange connection between two agents with overlapping transports -- @4.2 DIDExchange connection with both agents using HTTP and WS for inbound and outbound transport
  features/0025-didcomm-transports.feature:46  Create DIDExchange connection between two agents with overlapping transports -- @5.1 DIDExchange connection with one agent using http for inbound and the other using ws and both agents supporting ws and http outbound
  features/0025-didcomm-transports.feature:47  Create DIDExchange connection between two agents with overlapping transports -- @5.2 DIDExchange connection with one agent using http for inbound and the other using ws and both agents supporting ws and http outbound
  features/0036-issue-credential.feature:9  Issue a credential with the Holder beginning with a proposal
  features/0036-issue-credential.feature:23  Issue a credential with the Holder beginning with a proposal with DID Exchange Connection
  features/0036-issue-credential.feature:37  Issue a credential with the Holder beginning with a proposal with negotiation
  features/0036-issue-credential.feature:53  Issue a credential with the Issuer beginning with an offer
  features/0036-issue-credential.feature:66  Issue a credential with the Issuer beginning with an offer with negotiation
  features/0037-present-proof.feature:19  Present Proof where the prover does not propose a presentation of the proof and is acknowledged -- @1.1 
  features/0037-present-proof.feature:20  Present Proof where the prover does not propose a presentation of the proof and is acknowledged -- @1.2 
  features/0037-present-proof.feature:37  Present Proof where the prover does not propose a presentation of the proof and is acknowledged with a DID Exchange Connection -- @1.1 
  features/0037-present-proof.feature:38  Present Proof where the prover does not propose a presentation of the proof and is acknowledged with a DID Exchange Connection -- @1.2 
  features/0037-present-proof.feature:55  Present Proof of specific types and proof is acknowledged with a Drivers License credential type -- @1.1 
  features/0037-present-proof.feature:56  Present Proof of specific types and proof is acknowledged with a Drivers License credential type -- @1.2 
  features/0037-present-proof.feature:74  Present Proof of specific types and proof is acknowledged with a Biological Indicators credential type -- @1.1 
  features/0037-present-proof.feature:91  Present Proof of specific types and proof is acknowledged with multiple credential types -- @1.1 
  features/0037-present-proof.feature:109  Present Proof where the prover does not propose a presentation of the proof and is acknowledged -- @1.1 
  features/0037-present-proof.feature:110  Present Proof where the prover does not propose a presentation of the proof and is acknowledged -- @1.2 
  features/0037-present-proof.feature:129  Present Proof where the prover has proposed the presentation of proof in response to a presentation request and is acknowledged -- @1.1 
  features/0037-present-proof.feature:130  Present Proof where the prover has proposed the presentation of proof in response to a presentation request and is acknowledged -- @1.2 
  features/0037-present-proof.feature:150  Present Proof where the prover has proposed the presentation of proof from a different credential in response to a presentation request and is acknowledged -- @1.1 
  features/0037-present-proof.feature:151  Present Proof where the prover has proposed the presentation of proof from a different credential in response to a presentation request and is acknowledged -- @1.2 
  features/0037-present-proof.feature:207  Present Proof where the prover starts with a proposal the presentation of proof and is acknowledged -- @1.1 
  features/0183-revocation.feature:24  Credential revoked by Issuer and Holder attempts to prove with a prover that doesn't care if it was revoked -- @1.1 
  features/0183-revocation.feature:42  Credential revoked by Issuer and Holder attempts to prove with a prover that doesn't care if it was revoked with a DID Exchange connection -- @1.1 
  features/0183-revocation.feature:60  Credential revoked by Issuer and Holder attempts to prove with a prover that doesn't care if it was revoked -- @1.1 
  features/0183-revocation.feature:79  Credential revoked and replaced with a new updated credential, holder proves claims with the updated credential with timesstamp -- @1.1 
  features/0183-revocation.feature:116  Proof in process while Issuer revokes credential before presentation and the verifier doesn't care about revocation status -- @1.1 
  features/0183-revocation.feature:153  Credential is revoked inside the non-revocation interval -- @1.1 
  features/0183-revocation.feature:154  Credential is revoked inside the non-revocation interval -- @1.2 
  features/0183-revocation.feature:155  Credential is revoked inside the non-revocation interval -- @1.3 
  features/0183-revocation.feature:173  Credential is revoked before the non-revocation instant -- @1.1 
  features/0183-revocation.feature:174  Credential is revoked before the non-revocation instant -- @1.2 
  features/0183-revocation.feature:175  Credential is revoked before the non-revocation instant -- @1.3 
  features/0183-revocation.feature:176  Credential is revoked before the non-revocation instant -- @1.4 
  features/0183-revocation.feature:194  Credential is revoked before the non-revocation interval -- @1.1 
  features/0183-revocation.feature:195  Credential is revoked before the non-revocation interval -- @1.2 
  features/0183-revocation.feature:213  Credential is revoked before the non-revocation instant -- @1.1 
  features/0183-revocation.feature:214  Credential is revoked before the non-revocation instant -- @1.2 
  features/0183-revocation.feature:215  Credential is revoked before the non-revocation instant -- @1.3 
  features/0183-revocation.feature:216  Credential is revoked before the non-revocation instant -- @1.4 
  features/0183-revocation.feature:330  Revocable Credential not revoked and Holder attempts to prove without a non-revocation interval -- @1.1 
  features/0183-revocation.feature:347  Non-revocable Credential, not revoked, and holder proves claims with the credential with timesstamp -- @1.1 
  features/0183-revocation.feature:364  Revocable Credential, not revoked, and holder proves claims with the credential with timesstamp -- @1.1 
  features/0183-revocation.feature:380  Issuer revokes a credential and sends a v1 revocation notification -- @1.1 
  features/0211-coordinate-mediation.feature:21  Request mediation with the mediator accepting the mediation request -- @1.1 DIDExchange connection
  features/0211-coordinate-mediation.feature:45  Request mediation with the mediator accepting the mediation request and creating a connection using the mediator -- @1.1 DIDExchange connection
  features/0211-coordinate-mediation.feature:144  Two agents creating a connection using a mediator with overlapping transports -- @1.1 Acme and Faber creating a DIDExchange connection using Bob a mediator with overlapping transports
  features/0434-out-of-band.feature:9  Issue a v1 indy credential using connectionless out of band invitation
  features/0434-out-of-band.feature:42  Issue a v2 credential using connectionless out of band invitation -- @1.1 
  features/0434-out-of-band.feature:47  Issue a v2 credential using connectionless out of band invitation -- @2.1 
  features/0434-out-of-band.feature:50  Present a v1 indy proof using connectionless out of band invitation
  features/0434-out-of-band.feature:80  Present a v2 proof using connectionless out of band invitation -- @1.1 
  features/0434-out-of-band.feature:85  Present a v2 proof using connectionless out of band invitation -- @2.1 
  features/0453-issue-credential-v2.feature:23  Issue a Indy credential with the Holder beginning with a proposal -- @1.1 
  features/0453-issue-credential-v2.feature:28  Issue a Indy credential with the Holder beginning with a proposal -- @2.1 
  features/0453-issue-credential-v2.feature:48  Issue a JSON-LD credential with the Holder beginning with a proposal -- @1.1 
  features/0453-issue-credential-v2.feature:53  Issue a JSON-LD credential with the Holder beginning with a proposal -- @2.1 
  features/0453-issue-credential-v2.feature:58  Issue a JSON-LD credential with the Holder beginning with a proposal -- @3.1 
  features/0454-present-proof-v2.feature:20  Present Proof of specific types and proof is acknowledged with a Drivers License credential type with a DID Exchange Connection -- @1.1 
  features/0454-present-proof-v2.feature:21  Present Proof of specific types and proof is acknowledged with a Drivers License credential type with a DID Exchange Connection -- @1.2 
  features/0454-present-proof-v2.feature:26  Present Proof of specific types and proof is acknowledged with a Drivers License credential type with a DID Exchange Connection -- @2.1 
  features/0454-present-proof-v2.feature:31  Present Proof of specific types and proof is acknowledged with a Drivers License credential type with a DID Exchange Connection -- @3.1 
  features/0454-present-proof-v2.feature:46  Present Proof of specific types and proof is acknowledged with a Drivers License credential type with a DID Exchange Connection -- @6.1 
  features/0454-present-proof-v2.feature:65  Present Proof of specific types and proof is acknowledged with a Citizenship credential type with a DID Exchange Connection -- @1.1 
  features/0454-present-proof-v2.feature:70  Present Proof of specific types and proof is acknowledged with a Citizenship credential type with a DID Exchange Connection -- @2.1 
  features/0454-present-proof-v2.feature:85  Present Proof of specific types and proof is acknowledged with a Citizenship credential type with a DID Exchange Connection -- @5.1 

1 feature passed, 9 failed, 4 skipped
11 scenarios passed, 74 failed, 68 skipped
179 steps passed, 74 failed, 1067 skipped, 0 undefined
Took 4m41.079s

Exporting Agent logs.

swcurran commented 1 year ago

FYI - @TimoGlastra @cvarjao @nodlesh

swcurran commented 1 year ago

I did a quick check of the logs for the run and I think this is a list of all of the endpoints that failed because they don't exist -- e.g. the BDD Test Endpoints that the AFJ Backchannel doesn't support. Seems like a small list... Hopefully that bodes well for the second task in the list above being fairly easy.

.logs/acme_agent.log:    stack: 'NOT_FOUND: Resource "/agent/command/did-exchange/create-request-resolvable-did/" not found\n' +
.logs/acme_agent.log:  stack: 'NOT_FOUND: Resource "/agent/command/did-exchange/create-request-resolvable-did/" not found\n' +
.logs/acme_agent.log:    stack: 'NOT_FOUND: Resource "/agent/command/issue-credential-v2/prepare-json-ld/" not found\n' +
.logs/acme_agent.log:  stack: 'NOT_FOUND: Resource "/agent/command/issue-credential-v2/prepare-json-ld/" not found\n' +
.logs/acme_agent.log:    stack: 'NOT_FOUND: Resource "/agent/command/out-of-band/send-invitation-message/" not found\n' +
.logs/acme_agent.log:  stack: 'NOT_FOUND: Resource "/agent/command/out-of-band/send-invitation-message/" not found\n' +
.logs/bob_agent.log:    stack: 'NOT_FOUND: Resource "/agent/command/out-of-band/send-invitation-message/" not found\n' +
.logs/bob_agent.log:  stack: 'NOT_FOUND: Resource "/agent/command/out-of-band/send-invitation-message/" not found\n' +
.logs/faber_agent.log:    stack: 'NOT_FOUND: Resource "/agent/command/out-of-band/send-invitation-message/" not found\n' +
.logs/faber_agent.log:  stack: 'NOT_FOUND: Resource "/agent/command/out-of-band/send-invitation-message/" not found\n' +

openwallet-foundation / owl-agent-test-harness

Increasing the tests run by AFJ #696