URIs returned for pagination are incorrect

[Gamebuster19901]

Special ASCII characters are encoded improperly in the server's response to pagination requests.

Fox example, the server will return

"uri": "https://www.speedrun.com/api/v1/runs?game\u003dm1zjxqm6\u0026offset\u003d20"

When it should instead return

"uri": "https://www.speedrun.com/api/v1/runs?game=m1zjxqm6&offset=20"

If you follow the first link, the next pagination URI returned will be:

https://www.speedrun.com/api/v1/runs?game%5Cu003dm1zjxqm6%5Cu0026offset%5Cu003d20=\u0026offset=20

[AnInternetTroll]

Running the following command seems to work fine for me.

$ curl https://www.speedrun.com/api/v1/runs?game=m1zjxqm6 | jq -r .pagination.links[0].uri
https://www.speedrun.com/api/v1/runs?game=m1zjxqm6&offset=20

The URL is fine and I can use it to request the next link

$ curl "$(curl https://www.speedrun.com/api/v1/runs?game=m1zjxqm6 | jq -r .pagination.links[0].uri)" | jq -r .pagination.links[1].uri
https://www.speedrun.com/api/v1/runs?game=m1zjxqm6&offset=40

[Gamebuster19901]

If you simply go to

https://www.speedrun.com/api/v1/runs?game=y65797de&offset=20

on any web browser the returned links are incorrect. Same happens if I make a HTTPS request in Java. I do not know why curl doesn't have this issue.

[AnInternetTroll]

On Sat, Mar 18, 2023 at 03:01:23PM -0700, Gamebuster wrote:

If you simply go to

https://www.speedrun.com/api/v1/runs?game=y65797de&offset=20

on any web browser the returned links are incorrect. Same happens if I make a HTTPS request in Java. I do not know why curl doesn't have this issue.

-- Reply to this email directly or view it on GitHub: https://github.com/speedruncomorg/api/issues/163#issuecomment-1475007150 You are receiving this because you commented.

Message ID: @.***> I am not familiar with java at all. Do you have a project I can take a look at? The URL does indeed look bad on chromium, but firefox displays it corectly in it's "JSON" view. -- Med vennlig hilsen, Luca Matei Pintilie

[Gamebuster19901]

It will be a while before the project is buildable. I'm essentially rewriting Speedrun4J because it was designed very poorly.

I did a little bit of research: the reason why it's displayed correctly in the JSON view though is because it's technically valid javascript. (JSON is just javascript) https://mathiasbynens.be/notes/javascript-escapes#unicode-code-point

The issue comes when you try to use/display the raw http response:

It really slows development when you can't just copy the links from the debug console output, you also have to manually replace the unicode literal sequences.

Using unicode literals in the json response isn't required for special characters. Any character that has special meaning in javascript already has a single character escape sequence: https://mathiasbynens.be/notes/javascript-escapes#single

So I guess technically the response isn't malformed, but I don't know why the unicode seqences are used when just using = and & would work just fine.

[AnInternetTroll]

I know absolutely nothing about java or your setup, but I have no problems copy pasting the "malformed" URL into a browser or curl

The link works and it returns the expected response as far as I can tell.

In JavaScript there is this function called decodeURIComponent1 which works well, maybe there is something similar in Java?

-- Med vennlig hilsen, Luca Matei Pintilie

[Gamebuster19901]

I know absolutely nothing about java or your setup, but I have no problems copy pasting the "malformed" URL into a browser or curl The link works and it returns the expected response as far as I can tell. In JavaScript there is this function called decodeURIComponent[1] which works well, maybe there is something similar in Java? [1]: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/decodeURIComponent -- Med vennlig hilsen, Luca Matei Pintilie

Sorry, I'll try to explain in more detail

The response returned is indeed incorrect, if go to https://www.speedrun.com/api/v1/runs?game=y65797de\u0026offset=40 the response returned says the next pagination is https://www.speedrun.com/api/v1/runs?game=y65797de%5Cu0026offset%3D40\u0026offset=20 when the offset should be offset=60 not offset%3D40\u0026offset=20 (which is offset=40offset=20 when fully decoded).

Using a URI decoder will not help, because special characters in URIs are preceded by percentage (%) symbols, not backslashes (\). Backslashes are never used for special characters in URIs.

Backslashes are used as special characters in JSON (JavaScript Object Notation). The issue is with how the data is being stored into the JSON string. The JSON string is technically valid, but the string represented will point to an invalid URL if it is not decoded through a JSON parser.

If I run curl https://www.speedrun.com/api/v1/runs?game=y65797de\u0026offset=40 on my ubuntu machine, it also returns the same invalid response:

Using the unicode escape sequences in JSON is unnecessary and is what causes these issues. The returned response can just use literal & and = and it would work for more platforms.

The issue has nothing to do with the Java programming language, but I will put something simple together when I get off work today.

[AnInternetTroll]

Yes I see, it seems I misunderstood the issue and I can replicate it indeed.

Regardless, JavaScript's decodeURIComponent seems to handle this appropriately

decodeURIComponent("https://www.speedrun.com/api/v1/runs?game=y65797de\u0026offset=20") "https://www.speedrun.com/api/v1/runs?game=y65797de&offset=20"

While this really is an issue with speedrun.com, I doubt it will be fixed any time soon My advice would be to look for a Java equvalent of decodeURIComponent

-- Med vennlig hilsen, Luca Matei Pintilie