ga4gh-beacon / beacon-verifier

Tool to verify that a Beacon implementation follows the specification
2 stars 3 forks source link

Better error messages #8

Open MrRobb opened 2 years ago

MrRobb commented 2 years ago

The verifier should be way more expressive about fails etc, too. E.g.:

beacon-verifier http://progenetix.org/beacon

... giving e.g. a

    "Bioinformatics analysis": [
      {
        "name": "Bioinformatics analysis all entries",
        "url": "http://progenetix.org/beacon/analyses",
        "valid": false,
        "error": "Response does not match the schema: \"https://progenetix.org/services/schemas/analysis/\" is not of type \"object\"\n"
      },
...

... not really being very ... transparent, especially since the https://progenetix.org/services/schemas/analysis/ schema looks legit.

Originally posted by @mbaudis in https://github.com/ga4gh-beacon/beacon-verifier/issues/4#issuecomment-956519403

MrRobb commented 2 years ago

But the verifier should be way more expressive about fails etc, too. E.g.:

beacon-verifier http://progenetix.org/beacon

... giving e.g. a

    "Bioinformatics analysis": [
      {
        "name": "Bioinformatics analysis all entries",
        "url": "http://progenetix.org/beacon/analyses",
        "valid": false,
        "error": "Response does not match the schema: \"https://progenetix.org/services/schemas/analysis/\" is not of type \"object\"\n"
      },
...

... not really being very ... transparent, especially since the https://progenetix.org/services/schemas/analysis/ schema looks legit.

@mbaudis True, the error messages should be improved, but if you run the verifier you would see that the full error log is:

ERROR beacon_verifier::beacon > NOT VALID:
ERROR beacon_verifier::beacon >    ERROR: Type { kind: Single(Object) } - "https://progenetix.org/services/schemas/analysis/" is not of type "object" (/meta/receivedRequestSummary/requestedSchemas/0)

So, it is not referring to the result of https://progenetix.org/services/schemas/analysis/, but that when you ping https://progenetix.org/beacon/analyses the property in /meta/receivedRequestSummary/requestedSchemas/ should be an object (https://github.com/ga4gh-beacon/beacon-framework-v2/blob/4e134cd07c08e31bff00d9b9fb63bc4eff86f227/common/beaconCommonComponents.json#L50).

I have created this new issue to track the progress of the error messages of the verifier.

mbaudis commented 2 years ago

@MrRobb Thanks - found the error & fixed this & some other ones (though they mostly seemed of the "failed due to random required parameter in response schema" kind).

IMO it would be already good to be able to call the offending test url for a fast look; in the offline version it is a bit more approachable, but having the url directly trough the UI would be even better (e.g. page rendering the comments/errors and the direct server response below).

jrambla commented 2 years ago

@MrRobb there is a reason the text error in the log couldn't be used in the problematic example above?