search

digitalbazaar / pyld

JSON-LD processor written in Python
https://json-ld.org/
Other
611 stars 131 forks source link

Question about EARL reports #155

Closed hsolbrig closed 3 years ago

hsolbrig commented 3 years ago

When I run python tests/runtests.py ../json-ld-api/tests/ ../json-ld-framing/tests/ -e pyld-earl.jsonld, I get FAILED (failures=18, errors=168, skipped=7)

This doesn't seem to line up with the summary printed in https://w3c.github.io/json-ld-api/reports/

Digging into the report itself, the first failure shows at:

{
      "@type": "earl:Assertion",
      "earl:assertedBy": "https://github.com/dlongley",
      "earl:mode": "earl:automatic",
      "earl:test": "https://w3c.github.io/json-ld-api/tests/compact-manifest#tc009",
      "earl:result": {
        "@type": "earl:TestResult",
        "dc:date": "2021-05-25T21:31:04Z",
        "earl:outcome": "earl:failed"
      }
    },

Which, if I understand the reports correctly, appeared to have passed previously. Help!

hsolbrig commented 3 years ago

At least some of the failures seem to fit the pattern below:


EXPECTED:  {
  "@context": {
    "@vocab": "http://example/",
    "Foo": {
      "@context": {
        "baz": {
          "@type": "@vocab"
        }
      }
    }
  },
  "@type": "Foo",
  "bar": {
    "baz": "buzz"
  }
}
ACTUAL:  {
  "@context": {
    "@vocab": "http://example/",
    "Foo": {
      "@context": {
        "baz": {
          "@type": "@vocab"
        }
      }
    }
  },
  "@type": "Foo",
  "bar": {
    "baz": {
      "@id": "http://example/buzz"
    }
  }
}```
davidlehn commented 3 years ago

I haven't looked into this, but it may in part be a case of the the test suites changing and pyld hasn't caught up. But looks like you got a bunch of failures and errors so maybe it's something else too.

The reports really should be clear that results are for a test suite and implementation as of a certain date.

As far as tracking down and fixing the issues, help would be appreciated.

hsolbrig commented 3 years ago

Happy to pitch in -- just wanted to be certain I understood the starting point.

hsolbrig commented 3 years ago

Ok - part of the issue is that URL's of the form https://json-ld.org/test-suite/tests/frame/0005-in.jsonld are no longer valid.

curl https://json-ld.org/test-suite/tests/frame/0005-in.jsonld
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>404 Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL was not found on this server.</p>
<hr>
<address>Apache/2.4.29 (Ubuntu) Server at json-ld.org Port 80</address>
</body></html>

If I do curl https://json-ld.org/test-suite/ I get

<div class="hero-unit">
    <h1>JSON-LD Test Suite</h1>
    <br>
    <p>Please use the Working Group version of the test suite at <a href="https://w3c.github.io/json-ld-api/tests/">https://w3c.github.io/json-ld-api/tests/</a> and <a href="https://w3c.github.io/json-ld-framing/tests/">https://w3c.github.io/json-ld-framing/tests/</a> instead.</p>
  </div>

So - do we update the manifest and any embedded links? How to proceed?

gkellogg commented 3 years ago

Tests should now all be at https://w3c.github.io/json-ld-api/tests/ and /json-ld-framing/tests/. Those in json-ld.org are obsolete.

hsolbrig commented 3 years ago

Will update the manifests accordingly.

hsolbrig commented 3 years ago

Ah - I see what is going on. The instructions for running the tests are a bit ambiguous....

hsolbrig commented 3 years ago

Was using https://github.com/digitalbazaar/pyld/blob/master/CONTRIBUTING.rst , which wasn't clear on where the tests came from. Will file a separate issue