Unable to setup link-traversal query-engine

[tfrancart]

Issue type:

• :question: Question

---

Question:

Dear @rubensworks this is following a previous discussion at https://github.com/comunica/comunica/issues/1438

How can I package a query engine using the same link traversal feature as the demo at https://comunica.github.io/comunica-feature-link-traversal-web-clients/builds/solid-single-pod/ ?

Despite our efforts together with @antoine37120, we are unable to achieve the same behavior. In the demo page we query 3 sources, one of them being a Solid pod requiring Solid authentication.

We have compiled our own engine with the following config file : https://github.com/sparna-git/spoty/blob/main/custom_pacckage/config/config-default.json. the config is identical to the one we see at https://github.com/comunica/comunica-feature-link-traversal/blob/master/engines/config-query-sparql-link-traversal/config/config-solid-single-pod.json (except we sticked with v3, not v4).

Can you confirm this config will traverse the links in the LDP container of the Solid POD ?

(The complete compiled engine you can see at https://github.com/sparna-git/spoty/blob/main/custom_pacckage, and the test code is at https://github.com/sparna-git/spoty/blob/main/src/index.ts, where you can see how we call the engine)

What we observe in our integration is that the request is made to the Solid pod, the response from the query looks identical to the one we get from the demo page, but the LDP resources in the contained are not "followed".

Could we benefit from a little of your expertise to help us on this ? We are probably missing something.

Many many many thanks

[rubensworks]

Hi @tfrancart,

At first glance, your config looks correct to me, and should follow LDP links in Solid pods.

So if I understand correctly, if you execute your query using https://comunica.github.io/comunica-feature-link-traversal-web-clients/builds/solid-single-pod/ it gives the expected results, but not when using your custom compiled engine?

If so, the first low-effort thing you could try is bumping to Comunica v4 instead of v3, as this would be the only difference I see atm between your setup and the web client.

[tfrancart]

Thanks for looking into our question !

So if I understand correctly, if you execute your query using https://comunica.github.io/comunica-feature-link-traversal-web-clients/builds/solid-single-pod/ it gives the expected results, but not when using your custom compiled engine?

Correct, and that is our problem.

If so, the first low-effort thing you could try is bumping to Comunica v4 instead of v3, as this would be the only difference I see atm between your setup and the web client.

We will try to do that however the demo page in v3 did work also for us.

Here is the screenshot from the demo page showing the request cascades. Resources starting with "Personal_xxxx" are fetched from the Solid Pod (whose source URI ends with "spoty/"):

We see the query returns some results (the query joins between the Solid POD and one of the other file source).

The same with our demo page:

The request cascade shows that only the "spoty" Solid POD request is made (with a success, indicating correct login), with the other 2 sources, but no other resources are fetched from the LDP container, we don't see any fetch for "Personal_xxxx" resources

Thanks again for your help

[tfrancart]

One thing that puzzles me is that the request for the initial POD URL ("/pa/spoty") is said to return n-quads however the response payload looks like binary:

And here are the response headers from the Solid POD response:

[tfrancart]

@pchampin FYI

[rubensworks]

I had another look at https://github.com/sparna-git/spoty/blob/main/custom_pacckage/package.json, and it looks like you're missing a dependency on @comunica/query-sparql-link-traversal-solid (probably on v0.5.0 as long as you're using comunica v3). Without this, the relevant link traversal packages won't be installed. But I'm surprised this worked so far, as building the config should have lead to errors.

[antoine37120]

This link in default config get error : https://linkedsoftwaredependencies.org/bundles/npm/@comunica/config-query-sparql-link-traversal/^0.0.0/components/context.jsonld => Error: Unable to resolve version ^0.0.0 for bundle @comunica/config-query-sparql-link-traversal but have no error when run build:engine.

And if I change version to 0.1, 0.2, 0.3, 0.4 0.5 0.6, I have for all the same error when I run npm run build:engine : Failed to load remote context https://linkedsoftwaredependencies.org/bundles/npm/%40comunica%2Fconfig-query-sparql-link-traversal/0.2.0/components/context.jsonld: Detected remote context lookup for 'https://linkedsoftwaredependencies.org/bundles/npm/%40comunica%2Fconfig-query-sparql-link-traversal/0.2.0/components/context.jsonld' in config/config-default.json. This may indicate a missing or invalid dependency, incorrect version number, or an invalid context URL.

dependencies installed :

    "@comunica/config-query-sparql-link-traversal": "^0.5.0",
    "@comunica/query-sparql-link-traversal": "^0.5.0",
    "@comunica/query-sparql-link-traversal-solid": "^0.5.0",
    "@comunica/query-sparql-solid": "^3.1.1",

[rubensworks]

This link in default config get error : https://linkedsoftwaredependencies.org/bundles/npm/@comunica/config-query-sparql-link-traversal/^0.0.0/components/context.jsonld => Error: Unable to resolve version ^0.0.0 for bundle @comunica/config-query-sparql-link-traversal but have no error when run build:engine.

You can ignore that one. The dependency injection framework will never dereference those links. The error occurs because @comunica/config-query-sparql-link-traversal is in the pre-1.0.0 range.

Failed to load remote context https://linkedsoftwaredependencies.org/bundles/npm/%40comunica%2Fconfig-query-sparql-link-traversal/0.2.0/components/context.jsonld: Detected remote context lookup for 'https://linkedsoftwaredependencies.org/bundles/npm/%40comunica%2Fconfig-query-sparql-link-traversal/0.2.0/components/context.jsonld' in config/config-default.json. This may indicate a missing or invalid dependency, incorrect version number, or an invalid context URL.

This error can occur if the dependency injection framework is unable to find this package (@comunica/config-query-sparql-link-traversal) in the node_modules directory. This usually indicates a missing dependency or something unexpected has happened during installation. Could you manually check if this package exists in your node_modules?

In any case, I recommend using the latest version for all packages, to avoid conflicting dependency versions.

[antoine37120]

Hello, thanks for th e help, checking all dependencies and don't see where is the error and tryed with last version.

I see a difference whith the demo https://comunica.github.io/comunica-feature-link-traversal-web-clients/builds/solid-default/ It use service workers. But not in my case. We need that ? I don't see anything about service workers in docs.

We try to use it has js browser client. Are we realy in a good way for this ?

[antoine37120]

Good evening I finally found

I was inspired by the demo code and reproduced part of the code to implement the service worker. These files were very useful to me: https://github.com/comunica/jQuery-Widget.js/tree/master/src

[rubensworks]

@antoine37120 Do I understand correctly that everything works now for you?

The worker should not be required, but can be useful if you want to run queries in the background without blocking the main/render thread too much. Comunica processes everything in an async manner, so it should already be quite friendly for running on the main thread.

[antoine37120]

Not completly good, Need to create my own engine. Think good exemple is here ? https://github.com/comunica/examples/tree/master/packages/configure-sparql-http-solid

[rubensworks]

Those examples are a bit outdated, so I would not rely on those too much. The guides on https://comunica.dev/ are up-to-date though.