search

solid-contrib / test-suite

An automated test of Solid specification technical compliance
MIT License
23 stars 10 forks source link

enable test involvement by clients #58

Open bblfish opened 4 years ago

bblfish commented 4 years ago

Currently testing of ACL (and indeed LDP) is limited to a test suite that can only test some very limited aspects of a server. It won't be able to test most aspects since most resources on the Solid Web will be access controlled.

What is needed is a way to allow clients to report potential spec incompatibilities to servers, or client authors, or indeed to users so that the cause of problems can be correctly attributed to the rightful party.

Imagine for example that the rel="acl" Web ACLs resource is readable. This would allow clients to decide whether to bother authenticating to a resource, which credential they should use, or if they even could get access by becoming a member of a group (see linked issue).

But it would also allow a client that had followed the rules and supplied an ID they believe complies with the ACL to point out that they nevertheless could not access it. This may reveal a flaw on the server, in the accessibility of data needed to for authentication (perhaps a WebID Profile is not online), in the access control reasoning of the App, tokens sent by the OAuth provider... etc. etc... Without this people using apps that do not work could end up being completely in the dark, and not knowing who to blame, end up blaming Solid itself. With it, one would be in a much better position to improve the App ecosystem.

Such a feature would allow testing to grow as deployment grew, essentially making every resource open to testing by millions of agents using Solid every day.

kjetilk commented 4 years ago

It very much depends on what you mean by "Solid Web". If mean Solid when deployed in the wild, then yes, little of that will be available. That would be an extension to the present framework.

However, a server implementation can be nearly fully tested by adding a root ACL and authorizing a "fake IDP". The only thing that really can't be tested is to verify that the server returns a 500 when the root ACLs is not present.

So, we can subject willing server implementors to the test suite by asking them to set up an instance and populating it with some initial data (notably the root ACL, but see #40 ). If they are not willing, then we will not be able to run the test suite, but I hope and think that the value of having a test report turn green is sufficient to make implementors submit that.

bblfish commented 4 years ago

yes, the idea is to allow solid to be tested in the wild as deployed, continuously. So that would complement your current work. But this idea could guide the current work too, as it would of course be really useful if the reporting framework and all could be the same.