Response.Data seems to be limited to ~16000 characters

[agpmilli]

After sending a GET request, we receive a response that contains ~25'000 characters. it comes with status code 200 but when fetching the data (response.Data), we obtain errors like these (it varies):

• Exception reported: Invalid JSON in line 1, column 16014: premature end of input
• Exception reported: Invalid JSON in line 1, column 16016: invalid escape

The column varies between 16014 and 16016 for this error to occur and the error message between the two above.

Is there a size limit to the JSON object? Can we bypass it? Any other idea on why it occurs?

The error seems to come from this and this lines in jansson/src/load.c

[agpmilli]

Just to give a bit more context. we have been using sm-ripext to read JSON from body string (Calls made using SteamWorks and body returned in string). It was working well when using FromString function. So it seems that there is not size limit there.

We felt it would be more optimal to use ripext completely, both to read the JSON and to do the calls and that is how we faced this size limit (?) issue.

[agpmilli]

Anyone else facing this issue? Any workaround?

Thanks in advance.

[Wend4r]

Over HTTPS and RiP on Linux, Response.Data works correctly for me with more than 86554 characters

[agpmilli]

Thanks. We are also using HTTPS but our test server is on Windows. This might be the issue. We will give it a try with RiP on Linux.

In the callback, we are doing: JSONObject data = view_as<JSONObject>(response.Data); and this is the line returning the error. Should we do it differently?

[Wend4r]

I do

            JSON hResponseData = bResponseIsSuccessful ? hResponse.Data : null;

Should we do it differently?

Your method of obtaining should be enough

[agpmilli]

Thanks for the input. It actually works perfectly well on Linux.

Seems like the issue is with RiP on Windows. Anything we can do to make it work on Windows as well?

[ErikMinekus]

Try the debug build posted here: https://forums.alliedmods.net/showthread.php?p=2749617#post2749617

And post your server console output here.

[agpmilli]

It seems to work with this build. I receive a 200 and it does not crash at:

JSONObject data = view_as<JSONObject>(response.Data);

Here is what I see in the server console: api_ripext.txt

Is it possible that this issue is fixed in this debug release and not in the latest windows release?

[ErikMinekus]

Is it possible that this issue is fixed in this debug release and not in the latest windows release?

No, the only difference between the 1.3.0 release and debug builds is that it prints those debug messages.

[agpmilli]

Can this be due to the fact that in the non-debug windows build the stream is not finished being read (as we don't print it) and it gets returned as an incomplete body that then fails to be JSONified?

[ErikMinekus]

It shouldn't, libcurl is supposed to read the response in 16384 byte chunks, and once the complete response is loaded into memory, it will call the callback. And as you saw, that works fine on Linux. So something must be going wrong in libcurl on Windows where it thinks the response has been completely received.

[agpmilli]

Ok. So, nothing we can do here? This is still weird that similar builds are not behaving the same.

For development purpose, I will keep this debug build (that works on Windows) and when deployed (on Linux machines), the Linux non-debug build seems to work.

[agpmilli]

It seems the issue is with libcurl on Windows only. Closing the ticket as it does not concern this code anymore.

For people that still needs to make it work with Windows, we have decided to use SteamWorks for the requests where the received data exceeds 16000 characters.

Thanks for support!

[rtldg]

Do you know if there's a curl issue/ticket regarding the Windows issue?