Globbing, sorting and paging

JornWildt commented 5 years ago

Would it be possible to add query parameters for sorting and paging when requesting a "globbing" resource?

When presenting a list of items (blog comments, chat entries, pets, whatever), it is very common to let the server do paging and sorting to improve efficiency for browsing through large amounts of data (as compared to loading all N-thousand Turtle statements and throw away 99% of them in order to show the top 1%).

It could work as follows:

A globbing request like "/items/*" normally returns the Turtle content of all resources stored under "/items". Each of theses resources contains a set of Turtle statements.
One URL parameter, like for instance "orderBy", could define a predicate identifier (or multiple) to order the resources by.
Other URL parameters, like for instance "count" and "offset" could define which "slice" of the sorted resources to return.
As a result, only the content of the first "count" resources, from index "offset", when ordering the resources by the value of the predicates in the resources by "orderBy", would be returned.

Example, get the first 10 items ordered by the value of http://schema.org/name: /items/*?orderBy=https%3A%2F%2Fschema.org%2Fname&offset=0&count=10

The "orderBy" predicate is only evaluated for statements where the subject is identical to the resource URL.

I kind of assume this suggestion is covered in existing work on linked data.

JornWildt commented 5 years ago

I believe it is different from SPARQL in that it relates to whole resources instead of single statements - a granularity which is quite different from a giant un-organized set of statements for which SPARQL normally is used (AFAIK).

RubenVerborgh commented 5 years ago

Unless you consider a document a graph and do a CONSTRUCT over the graphs you want 🙂

JornWildt commented 5 years ago

Does that mean it (paging and sorting) is doable with the Solid server as it is today? It would be great to know how.

RubenVerborgh commented 5 years ago

No not yet; Solid servers do not support SPARQL queries.

kjetilk commented 5 years ago

@dmitrizagidulin you were talking about this, IIRC, care to voice an opinion?

dmitrizagidulin commented 5 years ago

@kjetilk My thoughts on this are:

We should deprecate globbing. It was always a placeholder, until better querying came along. Instead, we should focus on:

Implementing the LDP Paging spec
Implementing Triple Pattern Fragments (and via those and Comunica, SPARQL)
Promoting use of LD Flex querying

JornWildt commented 5 years ago

The LDP paging spec's use of HTTP headers (link relations) for paging information seems like a better idea then using URL parameters. Cool.

But I don't see any way to specify the ordering? The spec has a way to show what ordering the server selected - but I cannot find a way for the client to specify it? Given that the Solid server has no way to know what values to order by, it doesn't help much.

JornWildt commented 5 years ago

I also think globbing still has value. In example 13 (https://www.w3.org/2012/ldp/hg/ldp-paging.html#ldpc-ex-ordering-nopaging) the LDP paging spec spec shows the content of a container - but here it includes (some? all?) statements from the container items ... but the Solid server only returns a few file system details about the container items. If I want the full resource content of the items, I need to use globbing.

melvincarvalho commented 5 years ago

We should deprecate globbing.

-1 on this. It's in use, and that kind of action is the sort of thing people moan about for years. Would not be a good advert for the dev team.

melvincarvalho commented 5 years ago

It was always a placeholder

That's not true at all. Do you have any reference to back this up?

RubenVerborgh commented 5 years ago

We likely want to differentiate between desired functionality and concrete implementation. The desired functionality is cross-document data access.

Globbing is one way. It is easy conceptually, but has a couple of limitations. (DDOS on large documents, no cross-pod access, ...)

Querying is another way. Harder to implement, but more powerful when we do.

It's important that we are open to changing decisions of the past, since tech is changing too. And progress might mean breakage of a small number of things to enable a much larger set of things.

But we shouldn't move on this before an alternative is in place. And that will likely not happen before we get a better server.

melvincarvalho commented 5 years ago

In this case the desired functionality is to get all the triples in a container. LDPC (aka folders) have a special role in solid. What you describe is more generic and also useful.

But we shouldn't move on this before an alternative is in place. And that will likely not happen before we get a better server.

Glad to hear this!

nodeSolidServer / node-solid-server

Globbing, sorting and paging #956