Frontend usage with Vite?

[KonradHoeffner]

Is it possible to use node-quadstore in the frontend with Vite instead of webpack? I tried to adapt https://github.com/beautifulinteractions/node-quadstore-webpack-bundle to Vite in the following way:

import leveljs from "level-js";
import { newEngine } from "quadstore-comunica";
import { Quadstore } from "quadstore";
import { DataFactory } from "rdf-data-factory";

const dataFactory = new DataFactory();

const store = new Quadstore({
    dataFactory,
    backend: leveljs("quadstore"),
    comunica: newEngine(),
});

But it throws the following error:

Uncaught ReferenceError: global is not defined
    ta2str deserialize.js:5
    js deserialize.js:10
    __require2 chunk-FPJRMZPJ.js:14
    js iterator.js:6
    __require2 chunk-FPJRMZPJ.js:14
    js index.js:10
    __require2 chunk-FPJRMZPJ.js:14
    <anonymous> level-js:1

Even prepending this polyfill doesnt help:

if (typeof (window as any).global === "undefined") {
    (window as any).global = window;
}

When I reorder the import statements and add leveljs to the end of them, I get the following error instead:

Uncaught ReferenceError: Buffer is not defined
    js quadstore.js:13
    __require2 chunk-FPJRMZPJ.js:14
    js index.js:15
    __require2 chunk-FPJRMZPJ.js:14
    <anonymous> quadstore:1

[jacoscaz]

@KonradHoeffner disclaimer: I've never used Vite, I am not testing with Vite.

That said, I've just published quadstore@9.2.0-alpha.0 and quadstore-comunica@1.2.0-alpha.0 and updated node-quadstore-webpack-bundle accordingly. No major nor minor changes, these new versions are only about updating dependencies, which includes switching to Webpack 5.x for the browser-side build.

This means working around the fact that Webpack 5.x doesn't come with pre-baked polyfills like Webpack 4.x used to do, which (I think) aligns Webpack's behaviors with Vite. Is there a way for you to configure Vite in a manner that is equivalent to https://github.com/beautifulinteractions/node-quadstore-webpack-bundle/blob/2daf76bc34c4eac39a61ad2698ab3698aa46514a/webpack.config.js#L18-L29 ?

[jacoscaz]

@KonradHoeffner further to my previous comment, judging from https://github.com/Level/level-js/search?l=JavaScript&q=global and https://webpack.js.org/configuration/node/ it looks like Webpack 5.x still polyfills global, __filename and __dirname. Assuming Vite does not, you will indeed have to use your polyfill for the global object .

[KonradHoeffner]

Unfortunately, I cannot replicate the original error anymore. Right now it cannot even load level-js as Vite reports error: Do not know how to load path: level-js.

[KonradHoeffner]

I tried it now in a completely new vite project using npm init vite@latest (using vanilla JavaScript instead of TypeScript) and after adding global, now I get:

Uncaught TypeError: class heritage events_1.EventEmitter is not an object or null
    cjs asynciterator.cjs:74
    __require2 chunk-GEH2LQCB.js:32
    js quadstore.js:6
    __require2 chunk-GEH2LQCB.js:32
    js index.js:15
    __require2 chunk-GEH2LQCB.js:32
    <anonymous> quadstore:1
asynciterator.cjs:74
    cjs asynciterator.cjs:74
    __require2 chunk-GEH2LQCB.js:32
    js quadstore.js:6
    __require2 chunk-GEH2LQCB.js:32
    js index.js:15
    __require2 chunk-GEH2LQCB.js:32
    <anonymous> quadstore:1
    InnerModuleEvaluation self-hosted:2371
    InnerModuleEvaluation self-hosted:2371
    evaluation self-hosted:2332

[jacoscaz]

@KonradHoeffner seems like you're getting there! This should be fixable by making Vite fallback to something like https://www.npmjs.com/package/events for the events module.

[KonradHoeffner]

@jacoscaz thanks! I'm making slow progress. The events error is indeed fixable by npm install events.

The workaround for the buffer error is the following vite.config.js (after 'npm install rollup-plugin-polyfill-node'):

import { defineConfig } from 'vite'
import polyfillNode from 'rollup-plugin-polyfill-node'

export default defineConfig({
  plugins: [
    polyfillNode()
  ],
  optimizeDeps: {
    exclude: ['quadstore'] // <= The libraries that need shimming should be excluded from dependency optimization.
  }
})

[KonradHoeffner]

Next error:

Uncaught ReferenceError: exports is not defined
    <anonymous> http://localhost:3000/node_modules/quadstore/dist/index.js?v=db8b2a82:12
index.js:12:1
    <anonymous> http://localhost:3000/node_modules/quadstore/dist/index.js?v=db8b2a82:12
    InnerModuleEvaluation self-hosted:2371
    InnerModuleEvaluation self-hosted:2371
    evaluation self-hosted:2332

[jacoscaz]

This might come from Vite expecting a ES module whereas quadstore is currently built as CommonJS modules. Something like https://www.npmjs.com/package/vite-plugin-commonjs-externals might help?

[jacoscaz]

Support for building as a set of ES modules is being tracked at https://github.com/beautifulinteractions/node-quadstore/issues/138

[jacoscaz]

@KonradHoeffner any progress on using quadstore with Vite?

[KonradHoeffner]

@jacoscaz: Unfortunately not. I tried vite-plugin-commonjs-externals but now I get the error Uncaught SyntaxError: import not found: Quadstore. I will probably put this task aside for a while until this whole interconnectivity between Vite, Quadstore, various RDF JavaScript libraries, ES modules and the browser is more stable.

[jacoscaz]

@KonradHoeffner the problem with that is that it might take a long time for quadstore and all of its dependencies, particularly the LevelDB modules, to adopt the ES modules spec. Something that might be easier and faster for you is to load the quadstore.bundle.js UMD bundle from https://github.com/beautifulinteractions/node-quadstore-webpack-bundle via a dedicated script tag in your page while keeping the rest of your application working via Vite (or something along the same lines).

[KonradHoeffner]

Great advice, thanks! The following doesn't report any errors, I will import and query data and report back if there are any problems with that.

<script src="https://cdn.jsdelivr.net/gh/beautifulinteractions/node-quadstore-webpack-bundle/quadstore.bundle.js"></script>
...
const { leveljs, Quadstore, DataFactory, newEngine } = quadstore;

const dataFactory = new DataFactory();

const store = new Quadstore({
    dataFactory,
    backend: leveljs("quadstore"),
    comunica: newEngine(),
});

[jacoscaz]

Good! This is instructive, perhaps we could create pre-bundled distributions of quadstore as a stop-gap solution until we sort out ES module support (tracked in https://github.com/beautifulinteractions/node-quadstore/issues/138).

[KonradHoeffner]

Data loading and querying seems to work as well with a small test query!

const URL = "http://0.0.0.0:8000/test.nt";
const response = await fetch(URL);
const text = await response.text();
const parser = new Parser({ format: "rdf/ntriples" });
await store.open();
const quads = parser.parse(text);
store.multiPut(quads);
const {type, items} = await store.query("SELECT * WHERE {<http://my.test/resource> ?p ?o.}");

Real world queries seem to never finish, however, but I guess that is a different issue.

[jacoscaz]

That's good! As for the queries, I would suggest starting with using the memdown backend rather than level-js and see where that takes you. level-js uses IndexedDB, which in my experience can be incredibly slow at times.

Other than that, depending on the number of quads and the kinds of queries you are running, you might be stumbling into https://github.com/beautifulinteractions/node-quadstore/issues/115 .

Simple matching queries should be fine, though.

[KonradHoeffner]

I tried using memdown, but that throws the original errors again: no buffer, require not found, and so on.

[jacoscaz]

@KonradHoeffner how is quadstore working out for you?

[KonradHoeffner]

I could not get it to the speed required for my use case and will revisit it in the future, when the whole technology stack of Vite, CommonJS modules, RDF JavaScript libraries, Node/browser interoperability and so is more robust and polished. In the meantime, I am using Docker containers with Virtuoso SPARQL endpoints and normal JavaScript fetch queries to get JSON bindings and found that to be easier to setup and faster but thanks for all the swift and helpful support!

[jacoscaz]

No worries! Thank you for taking the time to share. Comments like yours really help with deciding where to allocate future effort. I will close this for now but feel free to follow up at any point.