features needed for implementing Linked Data Fragment - Datasource interface

[elf-pavlik]

This issue captures conversation started in https://github.com/beautifulinteractions/node-quadstore/issues/3#issuecomment-288134220

I think node-quadstore could provide a nice addition to the data sources already available for Linked Data Fragments server.

https://github.com/LinkedDataFragments/Server.js#configure-the-data-sources

Support for new sources is possible by implementing the Datasource interface.

I recall work on having such LDF server Datasource interface for LevelGraph and it seems that it requires approximate count without fetching result https://github.com/levelgraph/levelgraph/pull/72

/cc @RubenVerborgh

[jacoscaz]

@elf-pavlik Yes. This would also help offloading querying logic to the already working LDF ecosystem, which would definitely help (we're struggling a bit with allocating enough time to figure that out at the moment). However, it is mandatory for us to have full support for quads, not triples. What's the state of graphs in the LDF ecosystem?

[jacoscaz]

/cc @jmatsushita

[RubenVerborgh]

@jacoscaz We have a branch with quad support, called Quad Pattern Fragments. Demo at http://data-beta.linkeddatafragments.org/nanopublications, and further details can be obtained through @rubensworks.

[jacoscaz]

@RubenVerborgh does that branch allow for queries like the following?

give me ?s ?g such that ?s ex:ps ex:os ?g and ?g ex:pg ex:og?

[RubenVerborgh]

The server only supports single quad patterns at a time; more complex patterns are resolved by the client.

[jacoscaz]

I see. https://github.com/LinkedDataFragments/Client.js/commits/feature-qpf-latest looks promising. What's the ETA for this to be released?

I will have a look at adding an implementation of the Datasource interface to quadstore and test it using the QPF versions of the LDF client and server. If that gets us to support complex queries and SPARQL... Oh boy, that would be huge.

[RubenVerborgh]

What's the ETA for this to be released?

ASAP 😄 I'll let @rubensworks comment on this.

I will have a look at adding an implementation of the Datasource interface to quadstore

Great, let me know if there's any trouble. Note that, in the upcoming version 3.x, we are introducing a packaging system such that datasources can be released as independent npm packages easily.

[jacoscaz]

@RubenVerborgh can I configure and start ldf-server programmatically? The package does export everything in /lib through module.js but configuration parsing seems to happen in /bin/ldf-server. What I'm trying to do is fire up an instance of ldf-server with a custom datasource whilst retaining access to the datasource's underlying store.

EDIT: I'm working on the feature-qpf-latest branch.

[RubenVerborgh]

can I configure and start ldf-server programmatically?

Yes! For now, just fork the repo and define your datasource inside of it, as you can see for example here: https://github.com/KMiOpenBlockchain/Server.js

[jacoscaz]

I've seen what https://github.com/KMiOpenBlockchain/Server.js/commit/364a8477094d968ef24e2cbf9c21e48d867d2ffc does but it seems to still require launching ldf-server via cmd line as the initialisation of the configured datasources happens in /bin/ldf-server.

Basically, what I'm after is something like:

const store = new Store();
const datasources = [ new CustomDatasource({ store }) ];
const ldfServer = new require('ldf-server').Server({ datasources });
ldfServer.listen(8080);

// do something else with the store instance here

Is this possible ATM?

[RubenVerborgh]

That is possible indeed, it's more or less what /bin/ldf-server is doing.

[elf-pavlik]

What I'm trying to do is fire up an instance of ldf-server with a custom datasource whilst retaining access to the datasource's underlying store.

@jacoscaz I might sidetrack here, with https://github.com/substack/level-party one should have possibility to run ldf-server as one process and 'something else' as another process and both processes would access the same leveldb.

[jacoscaz]

@elf-pavlik @RubenVerborgh I've added preliminary support for retrieving the approximate count and managed to port all that's happening in ldf-server/bin/ldf-server.js to a function that allows me to create an instance of ldf-server from within quadstore code with a custom Datasource. It definitely needs some polishing but it does seem to work well. Thanks to the LDF server I was able to browse the contents of quadstore with a browser!

Code here: https://github.com/jacoscaz/node-quadstore/tree/ldf/

If this all works I think I will redirect my efforts in dealing with advanced queries towards other LDF projects.

[rubensworks]

I see. https://github.com/LinkedDataFragments/Client.js/commits/feature-qpf-latest looks promising. What's the ETA for this to be released?

QPF-support will be part of LDF server 3.x.x, which is feature complete at the moment. It just needs some more testing, which is something I will continue doing next week. So if everything goes well, the release should follow soon after that. A new QPF-supporting LDF client will be released together with that.

[jacoscaz]

@RubenVerborgh @elf-pavlik @rubensworks @jmatsushita

I've done a little more experimental work on this and it most definitely works. A few notes:

1. As a HTTP API is on our radars and I wasn't too happy with how the integration with the ldf-server module was going, I've written a proper server module from scratch that exposes an LDF endpoint (following https://www.hydra-cg.com/spec/latest/triple-pattern-fragments/).

2. The same server module also exposes HTTP equivalents of RDF/JS' .import(), .remove() and .match() methods, which is what we were aiming for.

3. I've added a .sparql() method to RdfStore that uses ldf-client (feature-qpf-latest branch) to query the database through the server module. There's some redundant/useless (de)serialization happening at the moment but the results are quite spectacular in that RdfStore now seems to support SPARQL queries with results formatted as dictionaries of RDF/JS Node instances.

I'm fairly happy with where this is going and I wanted to check-in with you all to gather some feedback. As @RubenVerborgh's query engine is way ahead than any sort of query engine we'd be able to come up with in the near future, I'm inclined to delegate all query logic but the essentials needed for RDF/JS and the LDF endpoint to ldf-client. What do you think?

[jacoscaz]

This would help getting rid of the HTTP roundtrip: https://github.com/LinkedDataFragments/Client.js/issues/12

[RubenVerborgh]

I've written a proper server module from scratch that exposes an LDF endpoint

Great!

the results are quite spectacular in that RdfStore now seems to support SPARQL queries

Cool!

As @RubenVerborgh's query engine is way ahead than any sort of query engine we'd be able to come up with in the near future

Perhaps, but it is also very incomplete still. We are working on a next-generation engine, but there is no ETA yet.

I'm inclined to delegate all query logic but the essentials needed for RDF/JS and the LDF endpoint to ldf-client.

That definitely makes sense!

[elf-pavlik]

Awesome work @jacoscaz and very fast progress :running_man:

I hope to take couple of days soon to work on some experiments I have in mind which will use LDF and node-quadstore. Should have some more useful feedback to offer then!

As @RubenVerborgh's query engine is way ahead than any sort of query engine we'd be able to come up with in the near future

Perhaps, but it is also very incomplete still. We are working on a next-generation engine, but there is no ETA yet.

Do you work on it in some public repo or currently still keep this project private?

[RubenVerborgh]

Currently private until we figure things out, plan is open!

[jacoscaz]

Closing this one and tracking all the issues in #51, #52, #54 and #53 . Thanks everyone!

[jacoscaz]

@elf-pavlik if and when you get around to tinker with quadstore, use branch v3.0 straight from github and the undocumented store.sparql() method.

[jacoscaz]

@elf-pavlik have you had any chance to tinker with quadstore? No pressure meant, just wondering if you have some feedback :)