search

openwallet-foundation / owl-agent-test-harness

Aries agent test framework, with agent backchannel support
https://aries-interop.info
Apache License 2.0
60 stars 66 forks source link

Feature afj/credo backchannel enhancements #842

Closed nodlesh closed 2 months ago

nodlesh commented 4 months ago

This PR creates a new backchannel for Credo, leaving the AFJ one in place for now. It expands the backchannel further for more supported protocols of Credo(though the new tests are not running in this PR).

This PR also adds a Dev Container for the Credo Backchannel.

All historical AFJ tests are now working with Credo. Runsets have been created for credo with ACA-Py and VCX. Credo Allure projects are created, and the gen-interop script has been updated to reference credo instead of aft.

The removal of the AFJ runsets and the AFJ backchannel will happen in a separate PR after we have a few successful runs of Credo. Though the afj runsets have been set to skip, so they will not run in the interop test pipeline.

This PR also doesn't have new tests outside of the historical AFJ tests. They need a little more work and will come in a separate PR. This PR is getting too large as it is.

To test this PR you can run the following. ./manage runset credo ./manage runset ariesvcx-credo ./manage runset credo-ariesvcx ./manage runset acapy-credo ./manage runset credo-acapy

swcurran commented 3 months ago

How’s the progress on this? Getting close?

nodlesh commented 3 months ago

I thought I was close a few weeks ago but got hung up solving an issue with starting a server with some newer TsED libraries in the upgrade of the Credo Backchannel. I'd like to say I'm close now. I've solved the issues in the Credo backchannel that were causing the schema errors in the AFJ backchannel seen in the nightly test runs for a while. I believe there is just a credential issuance problem I'm dealing with right now. I'd say I'm 3-5 days away from having credo running in the interop pipeline and retiring the AFJ runs in the pipeline. I'm away next week so it will be into Sept.

nodlesh commented 2 months ago

I'm having a couple of failures in the ariesvcx-credo runset that I'm investigating.

swcurran commented 2 months ago

Running the tests now.

When this is merged, are we adding credo as a new backchannel in the results in Allure and so to https://aries-interop.info, or does this push its results to the “javascript” Allure results so everything stays the same?

nodlesh commented 2 months ago

If I understand how the interop.info results are generated, I think it will list as credo not afj. The credo projects are already created in Allure.

swcurran commented 2 months ago

OK — I see how the results are pushed to allure. I’ll adjust the script configuration to get the credo results instead of the AFJ (aka “javascript”) ones.

Here is the results from the ACA-Py / Credo tests I ran locally. Look right?

4 features passed, 2 failed, 9 skipped 25 scenarios passed, 5 failed, 130 skipped 198 steps passed, 5 failed, 1180 skipped, 0 undefined Took 12m11.948s

swcurran commented 2 months ago

Looks great!! I’ll leave it to you to push the merge button.

nodlesh commented 2 months ago

I could be wrong, but I think I've adjusted the scripts to get the credo results instead of AFJ. Check the gen-interop.sh changes in this PR.

My ACA-Py - Credo runset only has 2 failures. These are not failing when running in VS code and the Dev Container. Looking at that now.

I think there is an issue with VCX talking to Credo, I think the problem is on the VCX side. Will have to engage the VCX guys.

swcurran commented 2 months ago

The gen-interop change looks good.

nodlesh commented 2 months ago

There are still a couple of failures here across the runsets, but I think they are problems with the frameworks, Credo and VCX mainly. I'm merging, and will follow up with them on the remaining issues.