Client not requesting next page (custom server)

[ghost]

Client sends request:

``` GET /Dataset/0 HTTP/1.1 accept: text/turtle;q=0.6,application/n-triples;q=0.3,text/n3;q=0.2 accept-encoding: gzip,deflate user-agent: Triple Pattern Fragments Client referer: http://10.0.2.2:49881/Dataset/0 Host: 10.0.2.2:49881 Connection: keep-alive ```

My server responds:

``` HTTP/1.1 200 OK Date: Sun, 28 Jan 2018 12:39:25 GMT Content-Type: text/turtle;charset=utf-8 Server: Kestrel Transfer-Encoding: chunked Access-Control-Allow-Origin: * 4d9c @prefix rdf: . @prefix rdfs: . @prefix owl: . @prefix xsd: . @prefix hydra: . @prefix void: . hydra:member . a void:Dataset, hydra:Collection; void:subset ; void:uriLookupEndpoint "http://10.0.2.2:49881/Dataset/0{?s,p,o}"; hydra:search _:triplePattern. _:triplePattern hydra:template "http://10.0.2.2:49881/Dataset/0{?s,p,o}"; hydra:variableRepresentation hydra:ExplicitRepresentation; hydra:mapping _:subject, _:predicate, _:object. _:subject hydra:variable "s"; hydra:property rdf:subject. _:predicate hydra:variable "p"; hydra:property rdf:predicate. _:object hydra:variable "o"; hydra:property rdf:object. void:subset . a hydra:PartialCollectionView; a hydra:PartialCollectionView; "Linked Data Fragment of Common Procurement Vocabulary"; "Triple Pattern Fragment of the 'Common Procurement Vocabulary' dataset containing triples matching the pattern { ?s ?p ?o }."; ; hydra:totalItems "290002"^^xsd:integer; void:triples "290002"^^xsd:integer; hydra:itemsPerPage "100"^^xsd:integer; hydra:first ; hydra:next . "2008-01-01"^^ . "Common Procurement Vocabulary 2008"@en . ... ```

The client loads the first 100 but doesnt follow up. As you can see there is a hydra:next so it should load next 100 right? Am i missing something? Is there something wrong with the server response?

[RubenVerborgh]

Which server are you using?

Since you are requesting http://10.0.2.2:49881/Dataset/0, the response should contain a triple

<http://10.0.2.2:49881/Dataset/0> hydra:next <http://10.0.2.2:49881/Dataset/0?offset=100>.

However, it only contains a triple

<http://10.0.2.2:49881/Dataset/0?offset=0> hydra:next <http://10.0.2.2:49881/Dataset/0?offset=100>.

[ghost]

I am trying to make my own server in asp.net and c#. I tried removing the ?offset=0, unfortunately it did not help. I also tried changing offset parameter to paging.

I tried poking in your clients code, adding console.log(links); in lib/extractors/ControlExtractor.js:

``` // Parse the links LINK_TYPES.forEach(function (property) { // check first the correct hydra:next, then the deprecated hydra:nextPage var links = controlData[property] || controlData[property + 'Page'], linkTargets = links && links[controls.fragment]; console.log(links); // added by me if (linkTargets && linkTargets.length > 0) Object.defineProperty(controls, property, { value: linkTargets[0], enumerable: true }); }); ```

Then I ran the client against your Node.js server and against my C# server with same dataset. The console output seems the same except the difference in parameter names and case in escaped characters.

client console.log() for Node.js server

``` { 'http://localhost:5000/0': [ 'http://localhost:5000/0?page=1' ] } { 'http://localhost:5000/0': [ 'http://localhost:5000/0?page=2' ] } undefined undefined { 'http://localhost:5000/0?predicate=http%3A%2F%2Fwww.w3.org%2F2004%2F02%2Fskos%2Fcore%23hasTopConcept': [ 'http://localhost:5000/0?predicate=http%3A%2F%2Fwww.w3.org%2F2004%2F02%2Fskos%2Fcore%23hasTopConcept&page=1' ] } { 'http://localhost:5000/0?predicate=http%3A%2F%2Fwww.w3.org%2F2004%2F02%2Fskos%2Fcore%23hasTopConcept': [ 'http://localhost:5000/0?predicate=http%3A%2F%2Fwww.w3.org%2F2004%2F02%2Fskos%2Fcore%23hasTopConcept&page=2' ] } undefined undefined ```

client console.log() for C# server

``` { 'http://10.0.2.2:49881/Dataset/0': [ 'http://10.0.2.2:49881/Dataset/0?pa=1' ] } { 'http://10.0.2.2:49881/Dataset/0': [ 'http://10.0.2.2:49881/Dataset/0?pa=2' ] } undefined undefined { 'http://10.0.2.2:49881/Dataset/0?p=http%3a%2f%2fwww.w3.org%2f2004%2f02%2fskos%2fcore%23hasTopConcept': [ 'http://10.0.2.2:49881/Dataset/0?p=http%3a%2f%2fwww.w3.org%2f2004%2f02%2fskos%2fcore%23hasTopConcept&pa=1' ] } { 'http://10.0.2.2:49881/Dataset/0?p=http%3a%2f%2fwww.w3.org%2f2004%2f02%2fskos%2fcore%23hasTopConcept': [ 'http://10.0.2.2:49881/Dataset/0?p=http%3a%2f%2fwww.w3.org%2f2004%2f02%2fskos%2fcore%23hasTopConcept&pa=2' ] } undefined undefined ```

I dont know javascript that well, but this leads me to believe that my server response is ok and that the links are correctly parsed? (Not sure where to poke next)

But according to tcpdump log, the client doesnt send another request to retrieve next page.

[RubenVerborgh]

I tried removing the ?offset=0, unfortunately it did not help.

Must be another issue then. This was certainly a problem.

I also tried changing offset parameter to paging.

That will note make any difference; the client just follows links, regardless of what they look like.

I tried poking in your clients code, adding console.log(links); in lib/extractors/ControlExtractor.js: Then I ran the client against your Node.js server and against my C# server with same dataset.

The links look good!

But according to tcpdump log, the client doesnt send another request to retrieve next page.

Can you give us the full server response, and the exact command to invoke the client? (Perhaps in a gist?)

[ghost]

I tried to mimick your nodejs server response as close as i could, still didnt help. I changed the pageSize in PageRouter.js to minimize clutter. Both servers are set for 1 triple per page and there are 3 result triples for the query below.

I run the client against your nodejs server with: ldf-client http://10.0.3.2:49881/0 queries/a.sparql

and against my c# server with: ldf-client http://10.0.2.2:49881/Dataset/0 queries/a.sparql

This is how the simple query from a.sparql file looks:

SELECT * WHERE {
  ?s <http://purl.org/dc/terms/created> ?o
}

This is the communication between client and nodejs server: https://gist.github.com/passingthru/a41464f082ec8ecbf2fc2611b47f00da

This is the communication between client and c# server: https://gist.github.com/passingthru/bad7877fc2bcc175986aa0cf206e66c2

Wireshark capture files: nodejs.pcap.gz csharp.pcap.gz

Side note: I noticed the nodejs server also puts hydra:next when the next page is empty, causing one unnecessary request?

[RubenVerborgh]

@rubensworks Can you check? If you don't see it, feel free to reassign to me.

[ghost]

Well I think I found the problem, its actually that the encoded characters are lowercase, eg. %2f instead of %2F

[rubensworks]

@passingthru Does making those characters uppercase actually fix the problem?

As the client does no URI component encoding itself (AFAIK), only URI decoding, it should be able to handle both uppercase and lowercase encodings.

Could you try adding a break-point in the client here?: https://github.com/LinkedDataFragments/Client.js/blob/86bc98b5b41760ca9be19382df33359389ac4125/lib/triple-pattern-fragments/FragmentsClient.js#L163 If nothing sensible comes out of that, check the status of controls after this block: https://github.com/LinkedDataFragments/Client.js/blob/e751f39f7e1406f0616fe29407bdcd229378b71b/lib/extractors/ControlsExtractor.js#L58-L64

[ghost]

Im not that good with javascript, all I was able to do is add console.log() in the ControlsExtractor.js like this:

console.log("\n");
    // Parse the links
    LINK_TYPES.forEach(function (property) {
      // check first the correct hydra:next, then the deprecated hydra:nextPage
      var links = controlData[property] || controlData[property + 'Page'],
          linkTargets = links && links[controls.fragment];
      if (linkTargets && linkTargets.length > 0)
        Object.defineProperty(controls, property, { value: linkTargets[0], enumerable: true });
console.log(links);
console.log(linkTargets);

Console output for lowercase: https://gist.github.com/passingthru/f9cd755102eef528c55a700b2583bd66

Console output for uppercase: https://gist.github.com/passingthru/77254c16ae9f351e01b833108a4184c8

If you compare the second block, line 15 you can see that the lowercase has undefined variable linkTargets, while uppercase has the correct value. Also the client does request the next page with uppercase.

[rubensworks]

@passingthru Could you print controls.fragment as well? It looks like that may be incorrect for some reason.

[ghost]

lowercase: https://gist.github.com/passingthru/30227bebc82ea9ac5a34cdc91b78f053

uppercase: https://gist.github.com/passingthru/5a507dc9da28f25aeeb601eca1020996

[rubensworks]

Ok, now it's clear what goes wrong.

The client performs the request using uppercase encodings. The C# server correctly interprets these encodings and handles the request. However, the C# server re-encodes the pattern using lowercase encodings to form its response, while it should in fact use the URL that has been requested by the client as-is (which could use uppercase encodings).

The URLs with lower and uppercase encodings may be semantically equivalent, but they are different URLs. AFAIK, these shouldn't not be seen as equal (@RubenVerborgh please correct me if you think this is wrong)

Alternatively, the server could perform a redirect to a URL that uses lowercase encodings (the client should be able to handle that). (This might be less efficient though)

[RubenVerborgh]

Indeed, @rubensworks nailed it. The IRIs should be exactly the same, that's how they're compared in the RDF model.