Closed rplati closed 3 years ago
We will be happy to fix if you can find the root cause of the problem. May by adding more verbose debugging into RestRserve app will help. Or if that doesn't work try to build Rserve with RSERV_DEBUG
defined ( adding PKG_CFLAGS += -DRSERV_DEBUG
).
I built Rserve with PKG_CFLAGS += -DRSERV_DEBUG like you suggested. This produces the following log for the failed request with HttpClient.
process request for 0x000001c99668b1e0
[attr = 2]
[attr = 42]
[attr = 4a]
process request for 0x000001c994b8b740
eval(try(.http.request('/echo'),silent=TRUE))
respond with 500 and content: Error in cpp_parse_multipart_body(body, boundary) :
Boundary string not found.
Rserve for some reason can’t use the boundary string used by HttpClient. I'll investigate this more next week and try to find examples of HttpClient’s raw requests that Rserve fails to process.
@rplati simply store the post body as rds
from the app and try to pass it to RestRserve:::cpp_parse_multipart_body
(with boundary string). Also you can post the body here.
I saved the requests with fiddler then tried parsing them with RestRserve:::cpp_parse_multipart_body
using the corresponding boundary from the header. It seems there are several differences between requests formed by RestSharp
and httpClient
. httpClient
submits requests like the following
POST http://127.0.0.1:6027/echo HTTP/1.1
Host: 127.0.0.1:6027
Content-Type: multipart/form-data; boundary="182e92b9-ebbb-4522-9205-a05c9e35191f"
Content-Length: 361
--182e92b9-ebbb-4522-9205-a05c9e35191f
Content-Disposition: form-data; name=part1; filename=part1.csv; filename*=utf-8''part1.csv
This is the first part.
--182e92b9-ebbb-4522-9205-a05c9e35191f
Content-Disposition: form-data; name=part2; filename=part2.csv; filename*=utf-8''part2.csv
This is the second part.
--182e92b9-ebbb-4522-9205-a05c9e35191f--
Characteristics of requests formed by httpClient
:
If I understand correctly, the error Boundary string not found
is caused by characteristic 1 - RestRserve:::cpp_parse_multipart_body
is supplied with the boundary in quotes of which it finds no instances in the body.
According to my testing, characteristics 2 through 4, while not resulting in an error, will result in RestRserve:::cpp_parse_multipart_body
returning an empty list.
It would be helpful if you post examples of the requests here. So we can debug a C++ code with this inputs.
Can you test on the boundary-parser-fix
branch?
remotes::install_github("rexyai/RestRserve@boundary-parser-fix")
Seems related SO question.
According to rfc7578 it seems reasonable to assume that boundary might come in quotes.
Can you test on the
boundary-parser-fix
branch?remotes::install_github("rexyai/RestRserve@boundary-parser-fix")
When testing the package from the fix branch, I get the error below for all requests. Googling the error message suggests I need to reinstall something. Do you know what package the object closeFD
might relate to?
eval(try(.http.request('/echo'),silent=TRUE))
respond with 500 and content: Error in get(name, envir = asNamespace(pkg), inherits = FALSE) :
object 'closeFD' not found
closeFD
likely comes from here https://github.com/rexyai/RestRserve/blob/f8e8f7def89a8e1a264735725a5184da7fc8d851/R/BackendRserve.R#L98. But I'm not sure why you are getting this error.
Windows issue may be?
Windows issue may be?
Seems so! and likely CI also fails because of the same issue. Need to call closeFD
conditionally on system type (call only on UNIX).
Indeed, this seems to be a Windows issue. The install from the fix branch works in a linux docker container.
The issue with the boundary string appearing in quotes seems to be resolved!
httpClient
’s multipart body is still not being parsed correctly, however, because of the other peculiarities of how httpClient
forms the requests. This is the example of the request body:
--182e92b9-ebbb-4522-9205-a05c9e35191f
Content-Disposition: form-data; name=part1; filename=part1.csv; filename*=utf-8''part1.csv
This is the first part.
--182e92b9-ebbb-4522-9205-a05c9e35191f
Content-Disposition: form-data; name=part2; filename=part2.csv; filename*=utf-8''part2.csv
This is the second part.
--182e92b9-ebbb-4522-9205-a05c9e35191f--
RestRserve:::cpp_parse_multipart_body
returns an empty list when trying to parse the above body. The following things seem to be causing the problem:
When testing, I had to change all of these to get cpp_parse_multipart_body
to correctly parse the body: I added double quotes to the name and filename, removed the utf-8 variant of the filename, and added a Content-Type row. In all other cases when I left one of these peculiarities uncorrected, cpp_parse_multipart_body
returned an empty list. It seems, therefore, that cpp_parse_multipart_body
would need to take into account the possibility of all of these peculiarities for httpClient
’s requests to parse correctly.
The omission of the Content-Type
is the most problematic of these, IMO. Would you say it would be reasonable to assume the file's Content-Type
is application/octet-stream
when the client does not provide one explicitly?
@rplati please checkout latest code from #162 - I believe I've fixed everything we've discussed here
Many thanks to both of you! httpClient
’s requests are now parsed correctly – tested on Windows and Linux container.
Could you say approx. when this will be available on cran?
Not sure. We need at least to add some basic tests to the PR. I'm quite busy these days, but can review PR if someone wants to do that.
Other than that there are no reasons to not push new version to cran.
We were able to get RestRserve working with .Net RestSharp client but multipart requests with the .Net core HttpClient are failing. A minimal example follows.
restrserve.r
This is started within a Docker container using the rexyai/restrserve:dev image.
Dockerfile
The following C# code uses .Net core 3.1
Program.cs
Requests submitted with the RestSharp client work fine and produce expected RestRserve log entries. Requests submitted with the HttpClient fail with the below error message and produce no RestRserve log entries.
Console output
Something about the way the .Net core HttpClient forms the multipart request causes problems for RestRserve. The HttpClient request should be ok as it works with the corresponding API implemented with plumber.
We would prefer to use the .Net core HttpClient over RestSharp because RestSharp uses unreasonable amounts of memory when the multipart body is large.