search

w3c / vc-test-suite-implementations

List of W3C Verifiable Credentials Implementations
BSD 3-Clause "New" or "Revised" License
3 stars 8 forks source link

Add test metadata to each request #24

Open filip26 opened 3 months ago

filip26 commented 3 months ago

Hi, because there is no way to get a reason nor an origin request from generated reports, it would be helpful to add test metadata to options or as a separate object.

e.g.

options: {  checks: [..],  testNumber:  ... ,  testType: "Interop", test params}

The syntax does not matter, Just anything what would allow a developer to target log search query more finely than going manually through all requests trying to figuring out if the 400 is correct or not.

BigBlueHat commented 3 months ago

Are you wanting these sent in with each request? Not sure adding to the request payload would be a good idea...

filip26 commented 3 months ago

Yes, the goal is to help an implementer to find a request associated with a test case execution. You already have an option object in the payload, you can add another one, e.g. meta, but the name nor a location is not important as long as it's part of payload (a search in payload is usually supported, search in HTTP headers is not in my case) with no issue keeping it backward compatible.

filip26 commented 3 months ago

A side note, helping an implementer to pass a suite is critical, and this is the minimum you can do, considering the reports give nothing but status. I see tendention to prioritize the suite operator perspective over an implementers` needs, which is destructive at the end of the day.

An implementer has no motivation to spend hours trying to figure out why a test has failed in a report, especially in a case of interop tests where only the test operator has the data.

aljones15 commented 3 months ago

Just so you know suite.log is published along side each test run and the newer https://github.com/w3c/vc-di-ecdsa-test-suite/blob/gh-pages/index.json index.json give out the report in json form where you can in turn see the actual error object, so I think what we should do here is record more metadata, and figure out a way of linking to the index.json file so you can see the raw json from the test run.

filip26 commented 3 months ago

thanks, but the log contains no useful data allowing to reproduce the test, a full request body is needed - generally I'm constantly asking (for more than two years) for a way how to get the origin request, easily if possible and you keep refusing any constructive solution I bring up, even it's almost no job on your side but life safer on impleterer's side.

Hm...this looks like a hostile approach towards implementers which is not helpful nor constructive.