search

LDflex / Query-Solid

Simple access to data in Solid pods through LDflex expressions
https://solid.github.io/query-ldflex/
MIT License
66 stars 15 forks source link

How to process transformations on a container #35

Closed matthieu-fesselier closed 5 years ago

matthieu-fesselier commented 5 years ago

For the needs of startinblox, we need to process some transformations on a container to sort, filter, group... resources.

To do this, we recreate an array of resources from the AsyncIterator returned by query-ldflex to perform the transformations. We have two main concerns with this approach:

Is there a smarter way to handle this?

RubenVerborgh commented 5 years ago

You will probably like https://www.npmjs.com/package/iter-tools 🙂

matthieu-fesselier commented 5 years ago

Thanks, it sounds interesting! I'll take a look ;)

RubenVerborgh commented 5 years ago

Feel free to reopen if you don't find what you need!

matthieu-fesselier commented 5 years ago

I added iter-tools to startinblox and it fits almost all my needs. The only problem I face is that I can't sort a list of resource, because of the asynchronicity of the proxies (see https://github.com/iter-tools/iter-tools/issues/286) Should the sort feature be added to query-ldflex as a parameter? I guess it make sense to sort resources before creating an iterator?

rubensworks commented 5 years ago

Sorting may indeed be valuable, as this could be pushed down to the SPARQL level by LDflex.

RubenVerborgh commented 5 years ago

Good idea; tracking sorting in https://github.com/RubenVerborgh/LDflex/issues/41/.

Note however that, in several case, sorting makes operations blocking, i.e., you receive all of them at the end instead of incrementally. So this is why we, by default, prefer to leave it to the consumer to perform sorting. For instance, if you are feeding a UI, you could take unsorted results and then insert them in the right place as they arrive. But that all depends on the use case.

As iter-tools also says:

For example: sorting, shuffling and in general all operations that rely on having the full array at your disposal. In that case the way to go is to convert the iterable in an array and use that.

So you could do https://www.npmjs.com/package/iter-tools#to-array and take it from there for now?

Or, for your specific case, map to sorting keys (asynchronously) and then sort?

matthieu-fesselier commented 5 years ago

Yes the solution would be to re-create an array with the sorting key and then sort the resources. This was our last choice but I think this is the only one possible for now

RubenVerborgh commented 4 years ago

Sorting landed via https://github.com/RubenVerborgh/LDflex/pull/43.