New library: quadstore

[jacoscaz]

Hello!

I'm the maintainer of quadstore (npm, github), an embedded graph database for JS runtimes that implements the Store, Source and Sink RDF/JS interfaces, backed by LevelDB and compatible backends.

I was wondering if you would consider adding quadstore to the set of tested libraries on https://rdfjs.dev ? And, if so, whether there would be something I could help you with?

I mostly ask because I'd be very interested in hearing about your experience developing against it, considering how many different points of comparison you have accrued while reviewing all of these libraries. In addition to that, having quadstore listed on https://rdfjs.dev would also make quadstore itself more visible, which always helps.

Quadstore has ties with two libraries that you've already reviewed:

• it is accompanied by quadstore-comunica (npm, github), a custom configuration and distribution of Comunica specifically optimized for bundle size and ease of integration.

• it is used within @gsvarovsky 's m-ld as the local storage backend

Feel free to close this right away if you're not interested in reviewing quadstore specifically or if you're not planning on expanding the set of tested libraries. In any case, thanks for https://rdfjs.dev - it's a brilliant website!

[megoth]

Apologies for late response, I haven't had much surplus of energy in a couple of months, and I've been ignoring GH notifications for a while.

But to your question: Yes, I would very much be interested in including quadstore into the list of libraries on rdfjs.dev. I'll do a bit of research into it and get back to you on how it would look, but on the top of my head I imagine it should support a local demo and a Solid demo using N3.js to parse and serialize the turtle (I did something similar in the local demo in the guide on Inrupt's JavaScript client libraries). Let me know if you recommend other tools to do that part.

I'll probably also update the guides on the libraries you mentioned, so that the readers will be aware of the connections.

Thank you so much for creating this issue, and thank you also for your kind words on rdfjs.dev ❤️ I hope to be able to develop it into an even more useful resource for Linked Data and JavaScript developers 😸

[jacoscaz]

Hello @megoth ! No apologies needed, I totally understand that. I myself go through periods in which I mute all notifications - it's good for one's mental health.

Thank you for considering quadstore! For an in-browser local demo you could use the browser-level backend (persistent via IndexedDB) or the memory-level backend.

I'm not sure about the Solid demo as quadstore's scope is relatively narrow compared to Solid's and it might make more sense to demo it individually. But, I defer to you! :)

Let me know if I can help with anything.

[megoth]

@jacoscaz I've created a branch for the new quadstore content: https://github.com/megoth/rdfjs/tree/library/quadstore

I've created three functional demos: Local, SPARQL, and Solid. I'm using the N3 parser and serializer to handle the Turtle.

I'm planning to write the texts soon, but wanted to share the progress with you in case you have feedback on the current work 😸

[megoth]

A preview is available at https://rdfjs-git-library-quadstore-megoth.vercel.app/quadstore

[jacoscaz]

Hello @megoth ! Thank you for taking a look at quadstore! I don't have time to do an in-depth review this week but can surely do so early next week. In the meantime, however, I've had a cursory look and one thing that stood out to me is the following pattern:

await Promise.all(quads.map(async ({subject, predicate, object, graph}) => {
  await store.put(df.quad(subject, predicate, object, graph));
}));

Two things:

1. The quads returned by N3 should already implement the relevant RDF/JS interfaces, meaning that you should be able to pass them to quadstore without deconstructing them into their individual terms and then reconstructing into a new object: await Promise.all(quads.map(quad => store.put(quad)));.
2. Quadstore offers dedicated methods for multi-quad operations: await store.multiPut(quads);. This is significantly faster than putting quads one by one while also resulting in transactional (read: atomic) changes to the underlying LevelDB-like backend (if supported).

[jacoscaz]

Another thing:

Given that you depend on N3 already, you don't need to also depend on rdf-data-factory for a DataFactory implementation. You can use N3's own implementation as I've done in https://github.com/jacoscaz/quadstore-perf/blob/a9136c865629fdb712eca25a55ebfbdae4ae57e2/src/loadfile.ts#L25-L28 .

[jacoscaz]

Last thing for today:

In addition to the .match() method from the low-level RDF/JS interfaces, quadstore also gives you the .get() commodity method that resolves to an array of quads rather than to the underlying stream of quads. Which one to use really depends on which aspects you'd like to highlight in the demo.

[jacoscaz]

Had a better look at the code and noticed that you're using the memory-level backend, which stores data in-memory and doesn't deliver persistence across page reloads, but you also add persistence via the useLocalStorage hook such as in https://github.com/megoth/rdfjs/compare/main...library/quadstore#diff-fcc653809c8fce1be045724efe84ad82f756b5b6c57ad7d45d8193defd908d55R22 and https://github.com/megoth/rdfjs/compare/main...library/quadstore#diff-fcc653809c8fce1be045724efe84ad82f756b5b6c57ad7d45d8193defd908d55R54 .

This is somewhat redundant if you consider the browser-level backend, which delivers persistence via IndexedDB. I mention this backend in a dedicated section of the README.md file but, looking back at it from your perspective, having that section placed at the bottom of the document with no mention of such backend in storage backends is really silly and I will rectify that.

That said, the main point here is that using the browser-level backend would further simplify your code by not having to depend on the useLocalStorage hook and basically dumping the entire database on every onSubmit() invocation.

[megoth]

I'm working on an update where I'll adress your feedback. But just wanted to thank you for everything that you've pointed out till now ^_^

[megoth]

I've integrated all of your feedback into a couple of commits (731424a7ce4b793bc3f9b3e672d88a2ee4e989ff, 91ac1fea9f74d1b671c51c957db24e071bddd364), hopefully it looks better now ^_^

I'll start writing the guides for the demos, and expand upon the text in general.

[jacoscaz]

@megoth looking great! The only suggestion I have left is more of a nitpick: technically speaking, browser-level persists to IndexedDB rather than localStorage (at least right now [1]). Practically speaking it's still storage that is local to the page but it's not the same thing. But this is just a nitpick, really; Well done!

Any feedback for quadstore on your end?

[1]: I hope one day we'll get a WASM build of LevelDB itself using OPFS.

[megoth]

@megoth looking great! The only suggestion I have left is more of a nitpick: technically speaking, browser-level persists to IndexedDB rather than localStorage (at least right now [1]). Practically speaking it's still storage that is local to the page but it's not the same thing. But this is just a nitpick, really; Well done!

Thank you! I've updated the text to reflect the use of IndexedDB, and it's updated on https://rdfjs-git-library-quadstore-megoth.vercel.app/quadstore.

There's a bug with my local demos, but once I've fixed them I'll merge Quadstore into the main branch ^_^

Any feedback for quadstore on your end?

Quadstore seems to do what it does very well. I would've appreciated methods to parse and serialize with it directly, but using N3 (or other RDF/JS-compliant libs) is a good solution as well. Maybe it would be good to point this out for developers looking for ways to do that while using your library? (I might put up a resource for this later on, I'm not aware of any explicit listing of this, except what you can find through rdf.js.org.)

[1]: I hope one day we'll get a WASM build of LevelDB itself using OPFS.

Cool! Hope to see that happen 😸

[jacoscaz]

Maybe it would be good to point this out for developers looking for ways to do that while using your library?

Definitely! Tracking this at https://github.com/jacoscaz/quadstore/issues/169 .

[megoth]

I'm closing this issue as I think Quadstore is satisfyingly presented on rdfjs.dev. I've also promoted it on the Solid Community Forum and r/solid.

Thank you @jacoscaz for bringing the library to my attention ^_^

[jacoscaz]

Thank you @megoth for considering quadstore and working on this!