search

solid-contrib / specification-tests

Test coverage report
https://solid-contrib.github.io/specification-tests/
MIT License
7 stars 4 forks source link

readme

Solid Specification Conformance Tests

This repository contains the tests that can be executed by the Solid Conformance Test Harness (CTH). The best way to run the harness is by using the Docker image.

The tests are written in a language called KarateDSL. This is a simple Behaviour-Driven Development (BDD) testing language based on Gherkin but which has been extended specifically for testing HTTP APIs. Further Solid-specific capabilities are added by the test harness. The difference to Cucumber's use of Gherkin is that this is actually executable code rather than just a human readable layer on top of functions that the tester must write. It also has an embedded JavaScript engine supporting ES6 syntax and provides the ability to call Java classes. The conformance tests are expected to be written in KarateDSL and JavaScript. Additional capabilities added to the test harness as Java libraries will be called from these without the need for the test implementer to know Java.

Run script

There is a handy script run.sh that you can use to run tests. It has options to specify the target server, to choose whether to use the published CTH image or a locally built one, and whether to use the tests embedded in that image or tests available locally.

For each test subject you want to test, create a {subject}.env file in this directory according to the instructions here.

If you are testing a local server, add the following line to your /etc/hosts file rather than use http://localhost:

127.0.0.1       server

To see the usage instructions:

./run.sh

The reports will be created in the reports/ directory.

If you want to only run specific test(s), you can add the filter option, such as:

./run.sh css --filter=content-negotiation

Running local tests

You can clone this repository, work on tests, and run them locally.

git clone git@github.com:solid-contrib/specification-tests.git
cd specification-tests

Use the script with the -d option to use the local tests: 

./run.sh -d . css

Tips for CSS

For testing CSS, the run script has the ability to register test accounts on a fresh server installation. It depends on node 18+ being available. It calls clientCredentials.js to add the required client ids and secrets to a temporary env file.

Creating a script for a CI workflow

If you just want to run tests against a single test subject, for example in a CI workflow, you can create a script such as this one which will run the tests embedded in the latest published CTH image:

#!/bin/bash

mkdir -p config reports

cat > ./config/application.yaml <<EOF
subjects: /data/test-subjects.ttl
sources:
  # Protocol specification & manifest (latest published version)
  - https://solidproject.org/TR/protocol
  - https://github.com/solid-contrib/specification-tests/blob/main/protocol/solid-protocol-test-manifest.ttl

  # WAC specification & manifest (latest published version)
  - https://solidproject.org/TR/wac
  - https://github.com/solid-contrib/specification-tests/blob/main/web-access-control/web-access-control-test-manifest.ttl

  # This is an example of how you could run tests for a specific version of a specification 
#  - https://solidproject.org/TR/2021/wac-20210711
#  - https://github.com/solid-contrib/specification-tests/blob/main/web-access-control/web-access-control-test-manifest-20210711.ttl

mappings:
  - prefix: https://github.com/solid-contrib/specification-tests/blob/main
    path: /data
EOF

docker pull solidproject/conformance-test-harness
docker run -i --rm \
  -v "$(pwd)"/:/data \
  -v "$(pwd)"/config:/app/config \
  -v "$(pwd)"/reports:/reports \
  -v "$(pwd)"/target:/app/target \
  --env-file=.env solidproject/conformance-test-harness \
  --output=/reports --target=https://github.com/solid/conformance-test-harness/css "$@"

Just change the target option and create a .env file for the server as mentioned above.

KarateDSL

The following is a high level overview of KarateDSL, focused on the most common aspects required in these specification tests. For more detail please go to:

Structure of a Test Case

The basic structure of a KarateDSL test file is:

@tag1
Feature: The title of the test case

  Background: Set up steps performed for each Scenario
    * def variable = 'something'

  @tag2
  Scenario: The title of the scenario
    Given url 'https://example.org/test.ttl'
    And header Content-Type = 'text/turtle'
    When method GET
    Then status 200
    And match header Content-Type contains 'text/turtle'
    And match response contains 'some-text'

  Scenario: The title of another scenario
    * another set of steps

The example above shows the use of tags. If you tag a feature, the tag is applied to all scenarios and combined with any tags on individual scenarios. The tag names should describe functionality that is optional so that testers can choose to skip tests that are not appropriate to the server being tested. For example, you would tag any tests that only work for servers implementing ACP with @acp.

The keywords Given, And, When, and Then in the Scenario are for the benefit of human readers of the script. To the test harness they simply denote steps in the procedure and have the same meaning as *. They make it easier to gain an understanding of the test.

The Background steps are executed before every Scenario in the file. This is important to understand as it allows the scenarios to be run in parallel but can also cause confusion if you expect a scenario to depend on something done in a previous scenario (i.e., chained together). If you need to perform a sequence of interactions in a single test then they should all be added to the same scenario. It is important to think of each scenario as a Test and the background as a BeforeEach method as you might see in other testing frameworks.

The first scenario represents a single HTTP interaction with the server. It has the following conceptual structure:

  1. Start with any variables you need to set up. Normally this is done with a * prefix.
  2. Then use Given to start describing the context for the test. Often this will be where you set up the URL or path for the interaction.
  3. Use And to provide more details for the context (e.g., setting up request headers).
  4. The When keyword represents the action, but remember it has the same meaning as *. The request is actually triggered by the use of method.
  5. Next you can begin to make assertions about the response starting with the Then keyword. Often the status is checked first.
  6. Finally, you can use And to describe additional assertions. You could use * if you need to create variables and analyze the response as this allows you to reserve And for assertions. It is a style choice since the keywords have no meaning to the test harness as already stated.

Data related keywords

JSON and XML are supported directly so you can express things such as:

* def cat = { name: 'Billie', color: 'black' }
* assert cat.color == 'black'
* def myCat = <cat><name>Billie</name><color>black</color></cat>
* assert myCat.cat.color == 'black'

When handling large amounts of data, you can either read it in from external files or express it on multiple lines using """. This can apply to commands such as def, request, and match:

* def cat =
  """
  {
    name: 'Billie',
    color: 'black'
  }
  """

KarateDSL attempts to parse the multiline data, so if you need to avoid this you can use the text keyword instead of def. This is particularly useful when you have Turtle data that Karate thinks might be malformed XML:

* text data =
  """
  @base <https://example.org> .
  <#hello> <#linked> <#world> .
  """

To read data from an external file there is a read keyword. However, this attempts to parse the data as JSON or XML so to get raw data you should use a Karate function as follows:

* def data = karate.readAsString('../fixtures/example.ttl')

HTTP related keywords

Setting the URL

There are 2 keywords for setting the URL to be used in a request: url and path. The full URL can be set in the background. It is good practice to use the * keyword in the background and the Given keyword in a scenario:

* url 'https://example.org/test`

You could set a base URL in the background which applies to all scenarios, and then use the path keyword to alter it for each scenario:

* url 'https://example.org/`
Given path 'test'

An alternative would be to set the base URL as a variable in the background, and use url in the scenarios:

* def baseUrl = 'https://example.org/`
Given url baseUrl + 'test1'

Configuring the request

The keywords: param, header, cookie, form field, and multipart field are used for setting key-value pairs in the relevant part of the request:

And param key = "value" # adds ?key=value to the query string
And header Accept = 'text/turtle'
And cookie foo = 'bar'
And form field username = 'john'

You can set multiple values at the same time with JSON using params, headers, cookies, and form fields.

And headers { Accept: 'text/turtle', 'Accept-Charset': 'iso-8859-5' }

Note:

If you want to set up some headers to be used across multiple requests, you can use the following command:

* configure headers = { 'Content-Type': 'application/json' }

If you use this in the Background, the headers will apply to all scenarios and cannot be overwritten with other methods of adding headers. If you need to replace these headers in one of those scenarios, you will need to use configure again to either replace them or set them to null and then set them as normal using either header or headers. The recommendation is to only use configure headers in the Background for any headers that should apply to requests in all scenarios and won't need to be overwritten. Use header or headers within Scenarios to add extra headers. Avoid using configure headers inside a Scenario unless you are setting up common headers for a series of requests within the one Scenario.

Setting the request body

To set up the body of the request use the request keyword. Note that for methods that expect a body (e.g., PUT, POST), you must use this keyword even if you set the content as an empty string. You would normally be using this in conjunction with the And keyword: 

And request 'data'
And request ''
And request { name: 'Billie', color: 'black' }
And request karate.readAsString('../fixtures/example.ttl')

Since Karate has special handling of JSON you need to be careful when passing JSON which includes local variables in the request. You can either use #() to embed an expression or wrap the whole JSON object in () so that it is treated as Javascript and not processed by Karate. Also be aware that when sending JSON-LD you need to wrap any keywords in quotes. Both of the following approaches will work but the first is preferred as it is more obvious:

* def url = 'http://localhost/test'
And request {@context: ['https://www.w3.org/ns/solid/notification/v1'], type: 'WebSocketSubscription2021', topic: '#(url)'}
And request ({'@context': ['https://www.w3.org/ns/solid/notification/v1'], type: 'WebSocketSubscription2021', topic: url})

Sending the request

The HTTP request is sent when you use the method keyword and a specific method. You would normally use this in conjunction with the When keyword:

When method PUT

Note:

Checking the response code

Finally, there is a shorthand for asserting the value of the response code if you are only matching one code:

Then status 200

In cases where you need to match multiple possible codes, you need to revert to using the responseStatus variable:

* Then match [200, 201, 202] contains responseStatus
* Then assert responseStatus >= 200 && responseStatus < 300
* Then match karate.range(200, 299) contains responseStatus

Note that the last option creates and array of 100 values so the error message in not particularly helpful as it lists all the options that the code did not match!

Checking the response payload

The important keywords for this are match and assert. They are very similar but generally match should be used as it is better at reporting errors than assert. The match keyword is very powerful. It has the ability to ignore parts of the data when matching and to apply fuzzy matching. The full details are available here: Payload Assertions.

In their simplest forms, match and assert simply take a JavaScript expression that evaluates to a boolean:

* match foo == bar && foo2 != 10

The left-hand side can be a variable name, a JSON/XML path, a function call, or anything in parentheses which evaluates as JavaScript. The right-hand side can be any Karate expression. Some of the important operators are outlined below.

contains

This can be a simple text comparison:

* match hello contains 'World'`
* match hello !contains 'World'`

It can also work with arrays and maps:

* def foo = { bar: 1, baz: ['hello', 'world'] }
* match foo contains { bar: 1 }
* match foo.baz contains 'world'

For matching the contents of an array independent of order:

* def data = { foo: [1, 2, 3] }
* match data.foo contains only [2, 3, 1]
* match data.foo contains any [9, 2, 8]

The any operator works with objects too:

* def data = { a: 1, b: 'x' }
* match data contains any { b: 'x', c: true }

If you want to match deeper into an object, you need contains deep:

* def original = { a: 1, b: 2, c: 3, d: { a: 1, b: 2 } }
* def expected = { a: 1, c: 3, d: { b: 2 } }
* match original contains deep expected

Special variables

You can access various parts of the HTTP response using special variables such as response, responseHeaders, and responseStatus.

The response body is saved into response after a request. Depending on the content type, the returned value will be a string, JSON, or XML object. You can apply matches to this or perform other logic on it:

* match response contains 'Billie'
* match response.name == 'Billie' # if the response is JSON

The headers are available as responseHeaders. However, this can be tricky to use. It is a map of all the header values in the form Map<String, List<String>>, and it preserves the case of the returned header names even though they should be treated as case insensitive. Because of this, there is a shortcut header which matches the header name case-insensitively and matches any value of that key. In the example below, the first 3 are equivalent but the 4th fails. There is a new function which helps with this problem as the last example shows:

* match header Content-Type contains 'text/turtle'
* match header content-type contains 'text/turtle'
* match responseHeaders['Content-Type'][0] contains 'text/turtle'
* match responseHeaders['content-type'][0] contains 'text/turtle'  # fails as responseHeaders['content-type'] returns null
* match karate.response.header('content-type') contains 'text/turtle'

Note that it is safer to use contains instead of == in this case since the header value may contain an encoding element such as ; charset=UTF-8. You can also get an array of all the values of a header with karate.response.headerValues().

Using the responseStatus variable as an alternative to status was mentioned earlier.

Karate Object

Within a test case, you have access to the Karate object which has a number of useful methods described here. This includes methods to manipulate data, call functions with a lock so they only run once, read from files, create loops, and handle async calling.

You can also access additional environment values using karate.properties['OPTION'] where the values are defined in your environment files using JAVA_TOOL_OPTIONS. For example:

JAVA_TOOL_OPTIONS=-Dproperty1=value1 -Dproperty2=value2

You should confirm the value is set before tests continue:

* def externalProperty = karate.properties['property1']
* match externalProperty == '#notnull'

Calling Functions

See https://karatelabs.github.io/karate/#code-reuse--common-routines

Sometimes you may want to set up something in the Background section that is only done once for all scenarios whereas typically these steps are run for every Scenario. This can be achieved using callonce. This would be similar to the difference between BeforeEach and BeforeAll in other testing frameworks. You can set up a function in the Background section (or even in another feature file), and on calling it receive a single object back:

  Background: Setup (effectively BeforeAll)
    * def setupFn =
    """
      function() {
        // do some setup
        return something;
      }
    """
    * def something = callonce setupFn

Although the Background is run for every Scenario, the function will only be called once.

Data driven tests

When a group of tests are very similar it is good to use a Scenario Outline with Examples to set up tests on a data- driven basis. There is a good description of this here: https://github.com/karatelabs/karate#data-driven-tests and an example below.

Scenario Outline: Test status <status> with <method>
  Given url 'https://httpbin.org/status/<status>'
  When method <method>
  Then status <status>
  Examples:
    | method | status |
    | GET    | 200    |
    | GET    | 300    |
    | DELETE | 400    |

This runs 3 tests, replacing placeholders in the scenario name and throughout the test. This is a simple example and the documentation linked above shows more possibilities, including generating tests in a function call.

Note If you are putting non-string data (e.g. a boolean or json) in a column then add ! as a suffix to the column name to make Karate handle the values correctly.

Comments

Comments in Karate test files are handled according to their position in the file. When they are added to reports this is done with the dcterms:description predicate, applied as follows:

The following example explains how comments in different positions are used. Note that in each case the comment could run over multiple lines and that any spaces before the # are stripped off.

# Feature level comments
Feature: Feature title
  # Comments here are treated as comments on the background and are prepended to scenario comments for each scenario
  Background: Set up the test
    # This applies to the variable declaration line below
    * def myVar = 'variable'
    # All comments between sections (Background or Scenario) are attached to the subsequent section

  # This would be the second comment for this Scenario as noted above
  Scenario: Run a test
    # This applies to the Given step below
    Given url '/test'
    When method GET
    Then Status 200

  Scenario: Run a test without its own comment but it would still pick up the background comments
    Given url '/test2'
    When method GET
    Then Status 200
    # There are no further sections so this comment will not be attached to the report

Change logs

It would be good to keep a log of changes in a test case but we don't want to attach this to the report as it may make them unnecessarily long. The best place to put the change log is therefore the bottom of the file where, as explained above, comments are not included in the reports.

Test Harness Capabilities

Global Variables

The test harness makes some variables available to all tests:

Helper Functions

RDF Model Support

KarateDSL 'natively' supports JSON and XML, but sadly it does not yet support RDF. As a result you will need a library to parse RDF documents and query them.

iri(iri)

This function returns an IRI object from the string version provided. The IRI can be used in functions querying RDF models.

iri(namespace, localName)

This function returns an IRI object by combining the namespace and localName strings provided. The IRI can be used in functions querying RDF models.

literal(value, option)

Returns a Literal value that can be used in functions querying RDF models. There are different variants of this function depending on the type of value and whether the option parameter is provided: type:

Parsing Functions

parse(data, contentType, baseUri?) (static function)

Parses an RDF document into a queryable model. The supported content types are: text/turtle, application/ld+json, text/html, application/xhtml+xml.

contains(model)

Returns true if the model passed in is a subset of the model the function is applied to. If it is not a subset it logs information highlighting the differences.

getMembers() or read-only property members

Designed for use with container contents, this function extracts any objects that match the pattern <url> ldp:contains <object> where <url> a ldp:BasicContainer and returns them as a list of strings.

asTriples()

Convert the model to triples.

subjects(predicate, object)

Returns a list of subjects matching the predicate and object.

predicates(subject, object)

Returns a list of predicates matching the subject and object.

objects(subject, predicate)

Returns a list of objects matching the subject and predicate.

contains(subject, predicate, object)

Tests whether any statements exist that match the subject, predicate and object.

HTTP Header Support

parseWacAllowHeader(headers)

This function accepts the response headers, locates the WAC-Allow header, and parses it into a map object.

It extracts all the access modes, and adds them as a list to the relevant group. The result can be treated as a JSON object such as:

{
  user: ['read', 'write', 'append'],
  public: ['read', 'append']
}

In a test, it could be used like this:

* def result = parseWacAllowHeader(responseHeaders)
And match result.user contains only ['read', 'write', 'append']
And match result.public contains only ['read', 'append']
parseLinkHeaders(headers)

Ths processes the response headers to extract all the Link headers and return them in the form of List<Map<String, String>>, where each item in the list is a map of the components of the link using the keys: rel, uri, title, type. The rel and uri values are mandatory. The returned object can be treated as a JSON object.

For example:

[
  {
    rel: 'type',
    uri: 'http://www.w3.org/ns/pim/space#Storage'
  },
  {
    rel: 'acl',
    uri: 'https://example.org/test/resource.acl'
  }
]

This could be used in a test like this:

* def hasStorageType = (ls) => ls.findIndex(l => l.rel == 'type' && l.uri == 'http://www.w3.org/ns/pim/space#Storage') != -1
* def links = parseLinkHeaders(responseHeaders)
And assert hasStorageType(links)

Other useful functions

resolveUri(base, target)

Apply the target URI to the base URI to return a new URI.

For example resolveUri('https://example.org/test/resource', '/inbox/') would return https://example.org/inbox/.

Libraries

As previously mentioned, the global variable clients holds instances of SolidClient which are set up to be able to access the test server as a specific user. There are some additional capabilities provided by this class. Most tests will deal with resources and containers (which is a subclass of a resource). These objects are represented by 2 classes in the test harness: SolidResouce and SolidContainer. For handling access controls in a universal way there are two classes: AccessDatasetBuilder and AccessDataset. Finally, there are two libraries (Utils and RDFModel) providing parsing functions for headers and RDF respectively. Their functions are exposed globally within the Karate environment and have already been described above.

SolidClient

The SolidClient class provides access to the test server on behalf of a user. This is mostly used internally within SolidResouce and SolidContainer but there are three public API functions available.

getAuthHeaders(method, uri)

This returns a map of the authorization headers required for a request on behalf of the user registered in this instance of the SolidClient. The map can be applied to a request being prepared in Karate using the headers keyword: 

  And headers clients.alice.getAuthHeaders('GET', url)

send(method, uri, data, headers, version) and sendAuthorized(method, uri, data, headers, version)

This is an alternative way to send a request which allows testers to have full control over the HTTP method and request headers. Karate will not allow invalid HTTP methods and sets some headers by default so this method allows some additional edge case tests to be performed. The only request header these methods include by default is the User-Agent so that requests are still traceable to the CTH. There are two versions of this method, the first sends un-authenticated requests, whereas the second adds the correct authorization headers for the user registered in this instance of the SolidClient.

The header parameter must be set carefully due to limitations in the way Karate translates JSON objects to Java. For simple examples you can pass headers inline:

* def response = clients.alice.send('POST', resource.url, 'data', {'Content-Type': 'text/plain', Accept: 'text/plain'})

If a header has multiple values they must be set once using an array but this cannot be inline:

* def values = ['a', 'b', 'c']
* def response = clients.alice.send('GET', resource.url, 'data', {'X-test': values})

Obviously you can set up the whole map as a variable:

* def headers = {'X-test1': '1', 'X-test2': ['a', 'b', 'c']}
* def response = clients.alice.send('GET', resource.url, 'data', headers)

SolidResource

The SolidResource class represents a resource or container on the server. Since this is also the base class for SolidContainer, it includes methods that are related to containers. It is not common to need use this class directly in a test as most resources and containers are created from the starting point of the rootTestContainer. Since this class conforms to the JavaBeans standard, getters and setters can be treated as properties when accessed from JavaScript.

SolidResource.create(solidClient, url, body, contentType)

A static method that can create a resource on the server.

getUrl() or read-only property url

Get the URL of this resource.

getContainer() or read-only property container

Gets the SolidContainer instance representing the parent container of this resource or ultimately returns the root container.

getAclUrl() or read-only property aclUrl

Get the ACL URL for this resource.

findStorage()

Get the storage root for this resource by working up the path hierarchy looking for the pim:Storage link header.

isStorageType() or read-only property storageType

Does this resource have the pim:Storage link header identifying it as a storage-type container?

getAccessDatasetBuilder() or read-only property accessDatasetBuilder

There are currently 2 access control implementations supported by the test harness:

Get an object that can build a set of access control statements in a WAC/ACP agnostic way. The object is initialized by adding owner access for the specified owner (not applicable in ACP mode).

getAccessDataset() and setAccessDataset(accessDataset) or read-write property accessDataset

Get or set an object representing the access control document/policy.

delete()

Delete this resource and, if it is a container, recursively delete its members.

SolidContainer

The SolidContainer class represents a container on the server and is a subcvlass of SolidResource. It is not common to need use this class directly in a test as most resources and containers are created from the starting point of the rootTestContainer. Since this class conforms to the JavaBeans standard, getters and setters can be treated as properties when accessed from JavaScript.

SolidContainer.create(solidClient, url)

A static method that can create a container on the server.

instantiate()

Create this previously reserved container on the server.

reserveContainer()

Reserve a container as a child of this container using a UUID as the name but do not instantiate it on the server.

createContainer()

Create a container as a child of this container using a UUID as the name. This is the equivalent of reserving a container and then instantiating it.

reserveResource(suffix)

Reserve a SolidResource as a child of this container using a UUID as the name with the provided suffix, but do not instantiate it on the server.

createResource(suffix, body, contentType)

Create a SolidResource as a child of this container using a UUID as the name with the provided suffix, then put the provided contents into it.

listMembers()

Get a list of all the members of this container.

deleteContents()

Recursively delete the contents of this container but not the container itself.

AccessDatasetBuilder

This universal access dataset builder can be used to build up a set of rules which are then built into a WAC or ACP specific AccessDataset. Many of the methods below take an access parameter which is a list of access modes to be granted. This list can contain any of read, write, append, control, controlRead, controlWrite or an IRI representing an access mode not known to the test harness. The known modes are translated to the appropriate IRI for WAC or ACP. There are special conditions relating to the control modes. In WAC you can only set control however in ACP it is possible to add read and/or write access to the ACL resource. When you use the mode control is used for ACP it is translated to read and write but it is possible to set only one of these using the controlRead and controlWrite variants.

For example:

const access = testContainer.accessDatasetBuilder
        .setAgentAccess(testContainer.url, webIds.bob, ['write'])
        .setInheritableAgentAccess(testContainer.url, webIds.bob, ['append', 'write', 'control'])
        .build();
testContainer.accessDataset = access;

When the AccessDatasetBuilder is first created for a resource, it is configured with full access for the owner of that resource and the base URI of the ACL is automatically set so there is no need to call setOwnerAccess or setBaseUri.

setOwnerAccess(target, owner)

Add a rule granting the owner full access to the target resource. If the target is a container then this access is made inheritable by current and future child resources.

setBaseUri(uri)

Set the base URI of the ACL document.

setAgentAccess(target, webId, access)

Add a rule granting access to the target for an agent, specified by their webId, with the specified access modes.

setGroupAccess(target, members, access)

Add a rule granting access to the targetfor a group, specified as a list (members) of their web IDs, with the specified access modes.

setPublicAccess(target, access)

Add a rule granting access to the target for an unauthenticated user, with the specified access modes.

setAuthenticatedAccess(target, access)

Add a rule granting access to the targetfor any authenticated user, with the specified access modes.

setInheritableAgentAccess(target, webId, access)

Add a rule granting access to any child resources of target for an agent, specified by their webId, with the specified access modes.

setInheritableGroupAccess(target, members, access)

Add a rule granting access to any child resources of target for a group, specified as a list (members) of their web IDs, with the specified access modes.

setInheritablePublicAccess(target, access)

Add a rule granting access to any child resources of target for an unauthenticated user, with the specified access modes.

setInheritableAuthenticatedAccess(target, access)

Add a rule granting access to any child resources of target for any authenticated user, with the specified access modes.

build()

Returns an AccessDataset constructed in WAC or ACP format using the rules added to the builder.

AccessDataset

asTurtle()

Returns a string with the turtle representation of the set of rules in the WAC or ACP format.

asSparqlInsert()

Returns a string with the turtle representation of the set of rules in the WAC or ACP format wrapped as a SPARQL INSERT statement. This is only used for ACP.

parseTurtle(data, baseUri)

In the rare case where you want to manually construct an ACL document you may use this function to replace the content of an AccessDataset object and then apply it to a resource.

TestHarnessException

Any exceptions generated within the libraries below are wrapped as TestHarnessException errors and will be trapped and reported by the Karate test engine. If you need to trap these exceptions within a test you can use a JavaScript try/catch mechanism:

    * eval
    """
    try {
      resource = testContainer.createResource('.ttl', exampleTurtle, 'text/turtle');
    } catch(err) {
      resource = null
      karate.log(err.message);
    }
    """

Example Test Cases

The following are a selection of example tests that demonstrate different features of the test harness, and show various approaches to writing tests.

protocol/content-negotiation/content-negotiation-turtle.feature

The purpose of this test is to confirm that a Turtle resource can be fetched as either JSON-LD or Turtle using content negotiation.

Feature: Requests support content negotiation for Turtle resource

  Background: Create a turtle resource
    * def testContainer = rootTestContainer.reserveContainer()
    * def exampleTurtle = karate.readAsString('../fixtures/example.ttl')
    * def resource = testContainer.createResource('.ttl', exampleTurtle, 'text/turtle');
    * def expected = parse(exampleTurtle, 'text/turtle')
    * configure headers = clients.alice.getAuthHeaders('GET', resource.url)
    * url resource.url

  Scenario: Alice can GET the TTL example as JSON-LD
    Given header Accept = 'application/ld+json'
    When method GET
    Then status 200
    And match header Content-Type contains 'application/ld+json'
    And assert parse(response, 'application/ld+json', resource.url).contains(expected)

  Scenario: Alice can GET the TTL example as TTL
    Given header Accept = 'text/turtle'
    When method GET
    Then status 200
    And match header Content-Type contains 'text/turtle'
    And assert parse(response, 'text/turtle', resource.url).contains(expected)

The Background for this test:

  1. Sets up a test container (which isn't yet instantiated).
  2. Loads example Turtle data into a variable from a file.
  3. Puts this data into a resource inside the test container.
  4. Convert the example data into an array of triples for later comparisons.
  5. Sets up the URL and authorization headers for the HTTP requests used in the scenarios.

Note that this Background is run for each Scenario so in reality 2 test files are created. That may seem inefficient, but it allows all scenarios to be run in parallel.

There are 2 scenarios based on this setup which perform the following steps:

  1. Set an Accept header to get the resource as JSON-LD or as Turtle.
  2. Send a GET request for this resource.
  3. Confirm that the response code is 200.
  4. Confirm the Content-Type header matches the requested type.
  5. Confirm that the response body, when converted to an array of triples contains the triples saved in the background setup.

protocol/wac-allow/access-Bob-W-public-RA.feature

The purpose of this test is to set up a resource with a combination of access controls and then confirm that the WAC-Allow header reports the correct permissions.

Feature: The WAC-Allow header shows user and public access modes with Bob write and public read, append

  Background: Create test resource giving Bob write access and public read/append access
    * def setup =
    """
      function() {
        const testContainer = rootTestContainer.reserveContainer();
        const resource = testContainer.createResource('.ttl', karate.readAsString('../fixtures/example.ttl'), 'text/turtle');
        const access = resource.accessDatasetBuilder
          .setAgentAccess(resource.url, webIds.bob, ['write'])
          .setPublicAccess(resource.url, ['read', 'append'])
          .build();
        resource.accessDataset = access;
        return resource;
      }
    """
    * def resource = callonce setup
    * url resource.url

  Scenario: There is an acl on the resource containing Bob's WebID
    Given url resource.aclUrl
    And headers clients.alice.getAuthHeaders('GET', resource.aclUrl)
    And header Accept = 'text/turtle'
    When method GET
    Then status 200
    And match header Content-Type contains 'text/turtle'
    And match response contains webIds.bob

  # Note this test is not applicable to a server using ACP as the WAC-Allow header is not supported currently.
  # In ACP mode, the following step would fail as there will be an ACL on the parent - this step needs to change
  # to ask the test harness to confirm there is no inherited access.
  Scenario: There is no acl on the parent
    Given url resource.container.aclUrl
    And headers clients.alice.getAuthHeaders('HEAD', resource.container.aclUrl)
    And header Accept = 'text/turtle'
    When method HEAD
    Then status 404

  Scenario: Bob calls GET and the header shows RWA access for user, RA for public
    Given headers clients.bob.getAuthHeaders('GET', resource.url)
    When method GET
    Then status 200
    And match header WAC-Allow != null
    * def result = parseWacAllowHeader(responseHeaders)
    And match result.user contains only ['read', 'write', 'append']
    And match result.public contains only ['read', 'append']

  Scenario: Bob calls HEAD and the header shows RWA access for user, RA for public
    Given headers clients.bob.getAuthHeaders('HEAD', resource.url)
    When method HEAD
    Then status 200
    And match header WAC-Allow != null
    * def result = parseWacAllowHeader(responseHeaders)
    And match result.user contains only ['read', 'write', 'append']
    And match result.public contains only ['read', 'append']

  Scenario: Public calls GET and the header shows RA access for user and public
    When method GET
    Then status 200
    And match header WAC-Allow != null
    * def result = parseWacAllowHeader(responseHeaders)
    And match result.user contains only ['read', 'append']
    And match result.public contains only ['read', 'append']

  Scenario: Public calls HEAD and the header shows RA access for user and public
    When method HEAD
    Then status 200
    And match header WAC-Allow != null
    * def result = parseWacAllowHeader(responseHeaders)
    And match result.user contains only ['read', 'append']
    And match result.public contains only ['read', 'append']

The Background for this test:

  1. Sets up a function which will be called once for the whole set of scenarios:
    1. Creates a test container (which isn't yet instantiated).
    2. Creates a resource in this container using an example Turtle file.
    3. Adds an ACL for the resource which grants Bob write access and the public read and append access.
  2. Use callonce to run the setup process once for all scenarios.
  3. Sets up the URL for the HTTP requests used in most of the scenarios.

The first 2 scenarios check the ACLs which could impact this test. The first fetches the resource's ACL and confirms it contains the ACL document we just created. The second fetches the container's ACL to confirm there isn't one from which permissions could be inherited.

The subsequent scenarios have the following pattern:

  1. Set up the authorization headers for requests from Bob but not for public requests.
  2. Send a GET or HEAD request for this resource.
  3. Confirm that the response code is 200.
  4. Confirm that the WAC-Allow header exists.
  5. Parse the WAC-Allow header and save this to a variable.
  6. Confirm the expected set of permissions for each of Bob and the public user.

protocol/writing-resource/containment.feature

The purpose of this test is to check that all containment triples are created on intemediate containers if a resource is created on a path that doesn't exist using PUT or PATCH.

Feature: Creating a resource using PUT and PATCH must create intermediate containers

  Background: Set up clients and paths
    * def testContainer = rootTestContainer.reserveContainer()
    * def intermediateContainer = testContainer.reserveContainer()
    * def resource = intermediateContainer.reserveResource('.txt')

  Scenario: PUT creates a grandchild resource and intermediate containers
    Given url resource.url
    And headers clients.alice.getAuthHeaders('PUT', resource.url)
    And header Content-Type = 'text/plain'
    And request 'Hello'
    When method PUT
    Then status 201

    * def parentUrl = intermediateContainer.url
    Given url parentUrl
    And headers clients.alice.getAuthHeaders('GET', parentUrl)
    And header Accept = 'text/turtle'
    When method GET
    Then status 200
    And match parse(response, 'text/turtle', parentUrl).members contains resource.url

    * def grandParentUrl = testContainer.url
    Given url grandParentUrl
    And headers clients.alice.getAuthHeaders('GET', grandParentUrl)
    And header Accept = 'text/turtle'
    When method GET
    Then status 200
    And match parse(response, 'text/turtle', grandParentUrl).members contains intermediateContainer.url

  Scenario: PATCH creates a grandchild resource and intermediate containers
    Given url resource.url
    And headers clients.alice.getAuthHeaders('PATCH', resource.url)
    And header Content-Type = 'application/sparql-update'
    And request 'INSERT DATA { <#hello> <#linked> <#world> . }'
    When method PATCH
    Then assert responseStatus >= 200 && responseStatus < 300

    * def parentUrl = intermediateContainer.url
    Given url parentUrl
    And headers clients.alice.getAuthHeaders('GET', parentUrl)
    And header Accept = 'text/turtle'
    When method GET
    Then status 200
    And match parse(response, 'text/turtle', parentUrl).members contains resource.url

    * def grandParentUrl = testContainer.url
    Given url grandParentUrl
    And headers clients.alice.getAuthHeaders('GET', grandParentUrl)
    And header Accept = 'text/turtle'
    When method GET
    Then status 200
    And match parse(response, 'text/turtle', grandParentUrl).members contains intermediateContainer.url

The Background for this test:

  1. Creates a resource as a grandchild of a test container (nothing is instantiated at this point).

Note that the 2 scenarios are independent as they each run the background steps, setting up their own test resource.

The pattern for the scenarios is based on making 3 HTTP requests. They each set up the URL and authorization headers first, then the sequence is:

  1. Send a PUT request to put data in this resource so it is actually created.
  2. Confirm that the response code a success code.
  3. Send a GET request for the resource's immediate container.
  4. Confirm that the response code is 200.
  5. Parse the response and confirm that the resource is a member of this container.
  6. Send a GET request for the resource's grandparent container (the original test container)
  7. Confirm that the response code is 200.
  8. Parse the response and confirm that the resource's immediate container is a member of this container.

web-access-control/protected-operation/read-resource-access-R.feature

The purpose of this test is to set up a resource with a combination of access controls and then confirm that the WAC-Allow header reports the correct permissions.

Feature: Bob can only read an RDF resource to which he is only granted read access

  Background: Create test resource with read-only access for Bob
    * def setup =
    """
      function() {
        const testContainer = rootTestContainer.reserveContainer();
        const resource = testContainer.createResource('.ttl', karate.readAsString('../fixtures/example.ttl'), 'text/turtle');
        const access = resource.accessDatasetBuilder
          .setAgentAccess(resource.url, webIds.bob, ['read'])
          .build();
        resource.accessDataset = access;
        return resource;
      }
    """
    * def resource = callonce setup
    * url resource.url

  Scenario: Bob can read the resource with GET
    Given headers clients.bob.getAuthHeaders('GET', resource.url)
    When method GET
    Then status 200

  Scenario: Bob can read the resource with HEAD
    Given headers clients.bob.getAuthHeaders('HEAD', resource.url)
    When method HEAD
    Then status 200

  Scenario: Bob cannot PUT to the resource
    Given request '<> <http://www.w3.org/2000/01/rdf-schema#comment> "Bob replaced it." .'
    And headers clients.bob.getAuthHeaders('PUT', resource.url)
    And header Content-Type = 'text/turtle'
    When method PUT
    Then status 403

  Scenario: Bob cannot PATCH the resource
    Given request 'INSERT DATA { <> a <http://example.org/Foo> . }'
    And headers clients.bob.getAuthHeaders('PATCH', resource.url)
    And header Content-Type = 'application/sparql-update'
    When method PATCH
    Then status 403

  Scenario: Bob cannot POST to the resource
    Given request '<> <http://www.w3.org/2000/01/rdf-schema#comment> "Bob replaced it." .'
    And headers clients.bob.getAuthHeaders('POST', resource.url)
    And header Content-Type = 'text/turtle'
    When method POST
    Then status 403

  Scenario: Bob cannot DELETE the resource
    Given headers clients.bob.getAuthHeaders('DELETE', resource.url)
    When method DELETE
    Then status 403

The Background for this test:

  1. Sets up a function which will be called once for the whole set of scenarios
    1. Creates a test container (which isn't yet instantiated).
    2. Creates a resource in this container using an example Turtle file.
    3. Adds an ACL for the resource which grants Bob read access. It logs this to make it visible in the reports.
  2. Use callonce to run the setup process once for all scenarios.
  3. Sets up the URL for the HTTP requests used in each of the scenarios.

The scenarios then:

  1. Set up the authorization headers for Bob to make each request.
  2. Send a request using each type of HTTP method.
  3. Confirm that the status codes for the GET and HEAD requests are all 200.
  4. Confirm that the status codes for PUT, PATCH, POST and DELETE requests are all 403.

Specifications

Annotations

All test cases should be linked to the related requirements in one of the Solid specifications. This depends on the specifications being annotated:

The intention is that this will be done as RDFa annotations in the specification documents, but it is understood that this will take some time. As a workaround, the same data may be provided in Turtle format separate to the specification.

For example:

prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>
prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#>
prefix xsd: <http://www.w3.org/2001/XMLSchema#>
prefix doap: <http://usefulinc.com/ns/doap#>
prefix spec: <http://www.w3.org/ns/spec#>

prefix wac: <https://solidproject.org/TR/2021/wac-20210711#>

<https://solidproject.org/TR/2021/wac-20210711>
  a doap:Specification ;
  spec:requirement
        wac:access-modes ,
        wac:access-objects
.

wac:access-modes
  spec:requirementSubject spec:Server ;
  spec:requirementLevel spec:MUST ;
  spec:statement "text of the requirement"@en .

wac:access-objects
  spec:requirementSubject spec:Server ;
  spec:requirementLevel spec:MUST ;
  spec:statement "text of the requirement"@en .

The specification vocab used above is under development, but the latest version is at https://github.com/solid/vocab/blob/specification-terms/spec.ttl.

Versions

It may be necessary to run tests against a specific version of a specification. This is simple to achieve by changing the files passed to CTH either via the --source option or in application.yaml. A specification must be paired with a test manifest, for example:

sources:
  - https://solidproject.org/TR/2021/wac-20210711
  - https://github.com/solid-contrib/specification-tests/blob/main/web-access-control/web-access-control-test-manifest-20210711.ttl

The manifest file might be a copy of the one used for the current version of a specification, however internal references must point to the right specification. This is easiest to do by modifying a namespace prefix and using that prefix for all such references.

prefix wac: <https://solidproject.org/TR/2021/wac-20210711#>
prefix manifest: <#>
manifest:protected-operation-not-read-resource-access-AWC
  a td:TestCase ;
  spec:requirementReference wac:access-modes ;
  td:reviewStatus td:approved ;
  spec:testScript
    <https://github.com/solid-contrib/specification-tests/blob/main/web-access-control/protected-operation/not-read-resource-access-AWC.feature> .

If the test implementation has changed between specification versions, then you could have an alternative version of the feature file and point to the relevant one in the manifest file.

The test report header shows the link to the specification making it clear which version was used.

Test Manifest

The test cases themselves need to be described in a manifest file. For each test case, this provides:

Note that there may be more than one test case linked to a requirement as shown in the example below:

prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>
prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#>
prefix xsd: <http://www.w3.org/2001/XMLSchema#>
prefix dcterms: <http://purl.org/dc/terms/>
prefix td: <http://www.w3.org/2006/03/test-description#>
prefix spec: <http://www.w3.org/ns/spec#>

prefix wac: <https://solidproject.org/TR/2021/wac-20210711#>

prefix manifest: <#>

manifest:protected-operation-not-read-resource-access-AWC
  a td:TestCase ;
  spec:requirementReference wac:access-modes ;
  td:reviewStatus td:unreviewed ;
  spec:testScript
    <https://github.com/solid-contrib/specification-tests/blob/main/web-access-control/protected-operation/not-read-resource-access-AWC.feature> .

manifest:protected-operation-not-read-resource-default-AWC
  a td:TestCase ;
  spec:requirementReference wac:access-modes ;
  td:reviewStatus td:unreviewed ;
  spec:testScript
    <https://github.com/solid-contrib/specification-tests/blob/main/web-access-control/protected-operation/not-read-resource-default-AWC.feature> .

manifest:acl-object-none
  a td:TestCase ;
  spec:requirementReference wac:access-objects ;
  td:reviewStatus td:unreviewed ;
  spec:testScript
    <https://github.com/solid-contrib/specification-tests/blob/main/web-access-control/acl-object/container-none.feature> .

Versioning

This repository will tag releases to make it possible to track report results to the specific versions of tests that were run. Changes to the tests for each version will be described in the CHANGELOG.md.

The release process is: