Make the test framework generic, distribuable and reusable

[lecoqlibre]

In order to make the test framework generic, distribuable and reusable, we should decouple its components into Solid apps!

We would have:

• A Solid app to generate and write test data on PODs.
• A Solid app to generate different indexes from this test data (the indexing agent).
• A Solid app to test the different indexes made by the indexing agent (our current framework).

This way, anybody could generate the data he wants to test indexing strategies onto. Then he would create these indexes using the indexing agent. Finally he would test different indexing strategies using our test app.

[pchampin]

I'm not sure this is a good idea. Granted, that makes the test framework more portable.

But generating test data in my browser, transferring it to my pod over HTTP, then retrieving it from my pod over HTTP to feed docker containers on my local machine, seems like a lot of data transfer with no added value. All the more that test data could be really big in some cases.

This "app" targets developers and researchers, tech-savvy people who can install a python interpreter in order to test the data. And actually, since Docker is a requirement to run the test, the most straightforward way to make the framework portable would be to make it runnable via Docker!

[lecoqlibre]

Python scripts are hard-writing data into PODs. This means it is only working for only one targeted Solid server (currently CSS). Having the possibility to test on different Solid servers should be considered. Even if we generate the data to a folder that a Solid server Docker container must mount, we will have to write (and maintain) specific scripts to do so (especially if the server does not provide a nice way to do it with Docker).

It is possible to mitigate the HTTP transfers by generating dumps that the indexing agent could use. Note that if you are on your local machine HTTP requests should resolve fast. But using HTTP is the price to pay for genericity.

Moreover using HTTP requests allows to deploy and test data and indexes on distant servers and architectures.

Another added value is to have a generic indexing agent that can be used for any other use cases, including production. That will be a nice contribution to the Solid community. This will complete and promote the work done on the indexing agent during the hackathon.

Edit: The Docker compose file is an helper to deploy multiple servers easily. But one should be able to deploy Solid servers like he wants to (without using Docker for instance). We could keep a Python script for generating this compose although more intelligent solutions might be preferred.

[pchampin]

Ok, I had missed a part of your point: you want the Solid app to write the data directly to the pods used for the tests. This is indeed much more interesting. And OK, it makes this agnostic to the pod implementation, which is a nice to have.

So, if I understand correctly, the plan would be to first generate a bunch of dockerized pod servers, then run in sequence the 3 solid apps that you describe above to populate these pods and run the tests.

Note that the solid apps can be pod-implementation-agnostic, but the 1st step would still need some implementation-specific tweaks to generate the pods on each server, and configure them all to accept writes from my WebID.

[lecoqlibre]

So, if I understand correctly, the plan would be to first generate a bunch of dockerized pod servers, then run in sequence the 3 solid apps that you describe above to populate these pods and run the tests.

Yes, with or without Docker (a compose file will still be provided to help). This gives the commodity to try new strategies and indexes on different data without leaving the browser. One could still write additional scripts for specific use cases.

Note that the solid apps can be pod-implementation-agnostic, but the 1st step would still need some implementation-specific tweaks to generate the pods on each server, and configure them all to accept writes from my WebID.

Very good remark, thank you. Indeed until there is no standard way to create PODs, one will still have to create them specifically. Making Solid apps instead of Python scripts will help to contain these tweaks into the smaller area possible (waiting for the standard to provide something). To help with the WebID configuration one way might be to create these PODs using an existing WebID (the one from the tester). CSS allows that, at least in the web interface.

[pchampin]

waiting for the standard to provide something

Don't hold your breath :-D I don't think that Pod creation/provisioning is (nor should be) in scope of the Solid protocol.

[pchampin]

Apart from that, you convinced me that Solid apps are the way to go for populating the test pods (and the indexes).