search

mjaakko / NeoStumbler

New stumbler application for contributing data to Mozilla Location Services
MIT License
29 stars 5 forks source link

Upload content encoding #117

Open RealTux678 opened 2 weeks ago

RealTux678 commented 2 weeks ago

Hi,

I tested sending to my server by defining a custom full path, but the request generates a 403 error. Apache ModSecurity blocks this request. Looking at the error logs, it seems to come from the "Content-Encoding" header. So I tested the Android code by commenting out the line ".addHeader("Content-Encoding", "gzip")" and it works!

I'm not familiar with HTTP and it seems that the "Content-Encoding" header should rather be used in the server -> client direction and not the other way around; hence the blocking.

More fundamentally, your app compresses the data sent in gzip format. Was this a specification requested by MLS? If not, it would be simpler to transmit the json data uncompressed and use the header ("Content-Type", "application/json")

mjaakko commented 2 weeks ago

Hi,

MLS accepts gzipped requests, see https://github.com/mozilla/ichnaea/blob/efe73300296436f1b6a8d10db9739ffc3711ba94/ichnaea/api/views.py#L106

Ideally we want to compress the data submissions, because the amount of data can be quite large. I guess there could be an option to disable compression, but maybe you can configure your server to accept gzipped requests?

joelkoen commented 2 weeks ago

This content-encoding header is valid, the framework I'm using for my server automatically decompresses the data for me because of this header.

RealTux678 commented 2 weeks ago

It is not a problem to transmit the data in a compressed way; and indeed the header content-encoding will be useful on the server side for decoding.

So I modified the modsecurity detection rules to bypass the blocking.

For info, the extracts from the error log: Warning. String match within "/accept-charset/ /content-encoding/ /proxy/ /lock-token/ /content-range/ /if/" at TX:header_name_content-encoding. [file "/usr/share/modsecurity-crs/rules/REQUEST-920-PROTOCOL-ENFORCEMENT.conf"] [line "1128"] [id "920450"] [msg "HTTP header is restricted by policy (/content-encoding/)"]

Access denied with code 403 (phase 2). Operator GE matched 5 at TX:anomaly_score. [file "/usr/share/modsecurity-crs/rules/REQUEST-949-BLOCKING-EVALUATION.conf"] [line "94"] [id "949110"] [msg "Inbound Anomaly Score Exceeded (Total Score: 5)"] [severity "CRITICAL"] [ver "OWASP_CRS/3.3.4"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-generic"] [hostname