UTF-8 encoding and missing Accept-Charset header

[jfock]

I´m trying to send a POST request with UTF-8 accepted/encoded content and according to the documentation

request.charset = Charsets.UTF_8

should set the Accept-Charset header, but it´s not:

11:16:42.478 DEBUG o.a.h.headers - http-outgoing-0 >> POST /api/login HTTP/1.1
11:16:42.478 DEBUG o.a.h.headers - http-outgoing-0 >> X-CSRF-TOKEN: 17939d9e-be6e-42fe-871a-987a25d29842
11:16:42.478 DEBUG o.a.h.headers - http-outgoing-0 >> Accept: application/json
11:16:42.478 DEBUG o.a.h.headers - http-outgoing-0 >> Content-Type: application/json
11:16:42.478 DEBUG o.a.h.headers - http-outgoing-0 >> Cookie: X-CSRF-TOKEN=17939d9e-be6e-42fe-871a-987a25d29842; visited_beta="1";$Path="/";$Domain="test.org"; sessionid=32457a10-ed8a-4265-84e3-671f7c8770dd
11:16:42.478 DEBUG o.a.h.headers - http-outgoing-0 >> Transfer-Encoding: chunked
11:16:42.478 DEBUG o.a.h.headers - http-outgoing-0 >> Host: www.test.org
11:16:42.478 DEBUG o.a.h.headers - http-outgoing-0 >> Connection: Keep-Alive
11:16:42.478 DEBUG o.a.h.headers - http-outgoing-0 >> User-Agent: Apache-HttpClient/4.5.2 (Java/1.8.0_112)
11:16:42.478 DEBUG o.a.h.headers - http-outgoing-0 >> Cookie: X-CSRF-TOKEN=17939d9e-be6e-42fe-871a-987a25d29842; sessionid=32457a10-ed8a-4265-84e3-671f7c8770dd
11:16:42.479 DEBUG o.a.h.headers - http-outgoing-0 >> Accept-Encoding: gzip,deflate
11:16:42.499 DEBUG o.a.h.headers - http-outgoing-0 << HTTP/1.1 403 
11:16:42.499 DEBUG o.a.h.headers - http-outgoing-0 << X-Authenticated: false
11:16:42.499 DEBUG o.a.h.headers - http-outgoing-0 << X-Content-Type-Options: nosniff
11:16:42.499 DEBUG o.a.h.headers - http-outgoing-0 << X-XSS-Protection: 1; mode=block
11:16:42.499 DEBUG o.a.h.headers - http-outgoing-0 << Cache-Control: no-cache, no-store, max-age=0, must-revalidate
11:16:42.499 DEBUG o.a.h.headers - http-outgoing-0 << Pragma: no-cache
11:16:42.499 DEBUG o.a.h.headers - http-outgoing-0 << Expires: 0
11:16:42.499 DEBUG o.a.h.headers - http-outgoing-0 << Content-Type: application/json
11:16:42.499 DEBUG o.a.h.headers - http-outgoing-0 << Content-Length: 127
11:16:42.499 DEBUG o.a.h.headers - http-outgoing-0 << Date: Tue, 26 Sep 2017 12:44:38 GMT

Same result with the Java client.

Not sure if this is the proper way to send such an request, with the old http-builder it was sufficient to set header 'Content-Type: application/json;charset=UTF-8' but when doing it with this client I get an exception

java.lang.IllegalStateException: Could not find encoder for content-type (application/json;charset=UTF-8)

Thanks!

[cjstehno]

I will look into the Accept issue, but the exception is because there is no encoder installed for the content type (see Request Encoders).

Actually, the Accept-Charset header you would need to specify yourself. In the case of request.charset that is actually supposed to set the charset used in the Content-Type header, which it does not do currently, but I am fixing. I will also update the docs to reflect this.

Unless I am misunderstanding something the Accept-Content type header is not tied to the request content-type at all, but rather the response content-type.

[cjstehno]

I found some talk that the application/json content type does not allow charset to be specified or at least does not care; however, the fixed code (coming in the next release) will still handle it properly.

[cjstehno]

The missing charset in the content-type issue is resolved in the development branch. The missing Accept-Charset is not an issue.

[partparsh]

Hi @cjstehno I am trying to do a multipart/form-data request to one of our servers. There is an owasp modsecurity firewall in-front and it is rejecting our request because of the charset specified in Content-type header, with following message:

ModSecurity: Multipart parsing error (init): Multipart: Invalid boundary in C-T (characters)

Is it possible to not include charset at all in Content-Type header or it is always appended?

Btw HttpBuilder-ng is awesome, we have created a DSL for testing and it complements the HttpBuilder-ng.