Type index consolidation service

[rubensworks]

Pitch

The Solid type index improves data discovery within pods. However, since client-side apps are expected to update these indexes whenever they add or update data, this can lead to data inconsistencies or incomplete type indexes. This challenge aims to tackle this problem by having an external agent that can periodically crawl your pod and update your type indexes if needed.

Desired solution

• An external agent to which you can register your WebID via a client-side app.
  • Upon registration, the app should ask permissions to update your type index.
  • It can also ask for optional read-access to your private resources.
• The agent should crawl over the registered pods periodically (configurable), determine all available resource types, and update the type index entries if needed.
• If the agent has read-access to private resources, it should not only update the public type index, but also the private type index.

Acceptance criteria

• A client-side app using which you can register your WebID, give permissions, and configure a crawling rate
• A crawler agent for pods that can update type indexes
• The generated type indexes should be annotated with a new class to mark that this type index has been auto-generated (so that apps can consider this type index as trusted).
• The generated type indexes should be annotated with the last crawling time (to mark staleness).

Pointers

• https://solid.github.io/type-indexes/

Scenarios

[pheyvaer]

This challenge will be able to built upon the results of https://github.com/SolidLabResearch/Challenges/issues/68 and https://github.com/SolidLabResearch/Challenges/issues/70.

[RubenVerborgh]

Thanks @rubensworks. However, I am not convinced that the type index is the way forward, as it clashes with the actual permission structure of Solid. Furthermore, I would definitely see this as a server-provided data structure.

So yes to the idea of this challenge, but perhaps just as a server-side implementation? And the fact that it can be deployed as an agent as optional?

[rubensworks]

However, I am not convinced that the type index is the way forward, as it clashes with the actual permission structure of Solid.

If you refer to the public/private distinction, then I guess this is something that can be easily resolved by introducing the ability to expose multiple permissions-based type indexes. I do think the fundamental idea of a type-based index is very promising for (query-based) data discovery, and we have preliminary experimental results to confirm this.

[RubenVerborgh]

@rubensworks I have experimented in the past with (other) indexes for every different agent that has some kind of access to a pod, which is feasible under the assumption that the number of pods is low.

Alternatively, the index could be a server-permissioned resource, where you basically get back what you're allowed to see.

An in any case, if not a type index, we need to figure this out for indexes in general. Okay, let's proceed then.