Chunked upload content-type is causing trouble with ModSecurity WAF

ne20002 commented 2 years ago

I've set up a ModSecurity Web Application Firewall to protect my Nextcloud instance. I now face a lot of errors in the WAF log (which would cause a lot af blocking once I disable detection only mode).

The cause for all errors listed can be boiled down to file upload requests with chunk files like this one: PUT /remote.php/dav/uploads/ag/23C6E9FC-E70D-4F88-89DF-47FD58AA44BF/22-03-18%2018-30-34%20F8EB.mov0000000000 HTTP/1.1 As far as I see, the encoding of the upload data is content-type: application/x-www-form-urlencoded which seems a bit strange for file uploads. I would expect to have it content-type: application/octet-stream.

The url-encoded body data is heavily causing trouble with the WAF when it tries to analyze the body data.

Steps to reproduce

Set up a ModSecurity WAF with nginx. Try to upload a file in chunks with iOS app.

Expected behaviour

File uploads, as it is binary data, shall be uploaded as content-type: application/octet-stream.

Actual behaviour

Uploaded Data is endoded as content-type: application/x-www-form-urlencoded and is confusing the web application firewall.

Logs

---6Y2eSSKW---A--
[31/Mar/2022:09:17:26 +0000] 1648718246 2a02:aa12:a682:880:6860:219d:34c7:e16b 0 10.0.2.100 80
---6Y2eSSKW---B--
PUT /remote.php/dav/uploads/ag/23C6E9FC-E70D-4F88-89DF-47FD58AA44BF/22-03-18%2018-30-34%20F8EB.mov0000000000 HTTP/1.1
accept: */*
Content-Length: 20000000
X-Real-IP: 2a02:aa12:a682:880:6860:219d:34c7:e16b
Host: xxxxxxxx
Connection: close
X-Forwarded-Proto: https
X-Forwarded-By: xxxxxxxxxxxxx
user-agent: Mozilla/5.0 (iOS) Nextcloud-iOS/4.3.1
content-type: application/x-www-form-urlencoded
X-Forwarded-Port: 443
X-Forwarded-For: 2a02:aa12:a682:880:6860:219d:34c7:e16b
Forwarded: for=2a02:aa12:a682:880:6860:219d:34c7:e16b; proto=https; by=xxxxxxx
accept-language: de-CH;q=1.0, en-CH;q=0.9, en-GB;q=0.8
authorization: Basic YWc6TWJDakQtS0tpSjgtWkdzUFktNnEzZEEtTmc0MkQ=
ocs-apirequest: true
accept-encoding: br;q=1.0, gzip;q=0.9, deflate;q=0.8

---6Y2eSSKW---F--
HTTP/1.1 200

---6Y2eSSKW---H--
ModSecurity: Warning. Matched "Operator `ValidateUrlEncoding' with parameter `' against variable `REQUEST_BODY' (Value: `\x00\x00\x00\x1cftypmp42\x00\x00\x00\x01isommp41mp42\x00\x00}\x91moov\x00\x00\x00lmvhd\x00\x00\x00\x (45429380 characters omitted)' ) [file "/etc/modsecurity.d/owasp-crs/rules/REQUEST-920-PROTOCOL-ENFORCEMENT.conf"] [line "375"] [id "920240"] [rev ""] [msg "URL Encoding Abuse Attack Attempt"] [data "\x00\x00\x00\x1cftypmp42\x00\x00\x00\x01isommp41mp42\x00\x00}\x91moov\x00\x00\x00lmvhd\x00\x00\x00\x00\xdeZrv\xdeZr~\x00\x00\x02X\x00\x00vg\x00\x01\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00@\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x03\x00\x00)\xdctrak\x00\x00\x00\tkhd\x00\x00\x00\x01\xdeZrv\xdeZr~\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00vf\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00 (15728440 characters omitted)"] [severity "4"] [ver "OWASP_CRS/3.3.2"] [maturity "0"] [accuracy "0"] [tag "modsecurity"] [tag "modsecurity"] [tag "modsecurity"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-protocol"] [tag "paranoia-level/1"] [tag "OWASP_CRS"] [tag "capec/1000/255/153/267/72"] [hostname "10.0.2.100"] [uri "/remote.php/dav/uploads/ag/23C6E9FC-E70D-4F88-89DF-47FD58AA44BF/22-03-18 18-30-34 F8EB.mov0000000000"] [unique_id "1648718246"] [ref "o0,33v489,33o979,1v1051,15728640o979,15728640v1051,15728640"]
ModSecurity: Warning. Matched "Operator `ValidateByteRange' with parameter `1-255' against variable `ARGS_NAMES' (Value: `\x8awU2'\xc5\x11\xd5W\xdf\xbaj\xe0vAEK\xfb\xb7\x92\x933@\x80L\xa8\xdfS\xfb\xab\xbb\xf0\xa6\xcdF\xc5\ (1978 characters omitted)' ) [file "/etc/modsecurity.d/owasp-crs/rules/REQUEST-920-PROTOCOL-ENFORCEMENT.conf"] [line "500"] [id "920270"] [rev ""] [msg "Invalid character in request (null character)"] [data "ARGS_NAMES=\x8awU2'\xc5\x11\xd5W\xdf\xbaj\xe0vAEK\xfb\xb7\x92\x933@\x80L\xa8\xdfS\xfb\xab\xbb\xf0\xa6\xcdF\xc5\x8e\x12,a\xbe\xfd\L-\xeb\xb9\xf5\xb6\xeb\x05Df\xa5\x8f\xbc \x00\x00\xecx\xa9\x0a#%\x8as\x84\x92\x84}Wu9\xcc \xf0\x17\xe0\x1eg\xed]9@\x8f\xf9\xde\xaf\x9d1\x87\xdb\xd0\xfc5zv\xf7\xbe\x85\x08d\xaf1\xa3\xae\xd0\xca\x8eC\x0c\xea\x9f.O\x0d\xc2Igs\xd9?\xcf\x9d\xb6C\x9e\xac\x98\xd1$\xa4{e\x92\x01\xeaI\x03R^X\xbe\xeb\x1f\x00\xee\xaf\x05cs\xea\x17)\x0c\x00\x01\x15\xa4k\x05\x85\x90\xc62\xae\xf7\xb2\x93L\xea\x09$\x8c0>\xce\xaa\x07\x84J\xd4Jd\x92o\x9e\xfe (515 characters omitted)"] [severity "2"] [ver "OWASP_CRS/3.3.2"] [maturity "0"] [accuracy "0"] [tag "modsecurity"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-protocol"] [tag "paranoia-level/1"] [tag "OWASP_CRS"] [tag "capec/1000/210/272"] [hostname "10.0.2.100"] [uri "/remote.php/dav/uploads/ag/23C6E9FC-E70D-4F88-89DF-47FD58AA44BF/22-03-18 18-30-34 F8EB.mov0000000000"] [unique_id "1648718246"] [ref

Reasoning or why should it be changed/implemented?

This makes using a web application firewall a pain (as it needs to disable lots of rules that are very valid in other cases).

Environment data

iOS version: e.g. iOS 15.4

Nextcloud iOS app version: 4.3.1

Server operating system: Debian Bullseye, Nextcloud docker image

Web server: Nginx

Database: Postgres

Nextcloud version: 22.2.6

ne20002 commented 2 years ago

Hi @thisIsTheFoxe Sorry to contact you directly. As I see you're working on a few tasks regarding uploads, may it be possible to have a look on this one as well?

thisIsTheFoxe commented 2 years ago

Hi @ne20002, Thanks for reaching out. Sadly, I'm not familiar with ModSecurity. Are you sure this is a problem specifically with the iOS app? Did you e.g. try to upload the files via the iOS Safari app, another web-browser, or client app? Also, if possible add the iOS logs to the ticket. You can find them in the app > More > Settings > Advanced

marinofaggiana commented 2 years ago

@tobiasKaminsky when you upload a chunk files uses a particular content-type ?

ne20002 commented 2 years ago

Hi @thisIsTheFoxe

Thank you for the fast answer.

I tested with Android client and Linux Mint client. Both do not cause any problems with ModSecurity. Only the iOS client when uploading files (or chunks).

For me it looks a bit suspicious to upload binary data with content-type: application/x-www-form-urlencoded. I doubt this is an efficient coding for binary data and it seems unusal to me (never seen this for binary data before). Also, assuming the body is form data makes the WAF looking into it. And this is not necessary and it is causing troubles.

I don't think one need knowledge in ModSecurity. A simple change of the body encoding for binary uploads on the iOS client should fix this issue.

There are no iOS logs as I don't have access to the iOS device (it's the phone of a friend of mine) and the iOS app just gets a 403 as response from the ModSecurity. I believe the Nextcloud server handles the body part according to its encoding und thus doesn't bother if it gets x-www-form-urlencoded or octet-stream. But for a web application firewall this is a huge difference.

According to RFC-2046 it says for mime-types:

application -- some other kind of data, typically either uninterpreted binary data or information to be processed by an application. The subtype "octet- stream" is to be used in the case of uninterpreted binary data, in which case the simplest recommended action is to offer to write the information into a file for the user.

tobiasKaminsky commented 2 years ago

@tobiasKaminsky when you upload a chunk files uses a particular content-type ?

It is a webdav PUT, like the not-chunked upload. Only difference is that it is not a complete file, but only a chunk.

ne20002 commented 2 years ago

Hi @tobiasKaminsky

It is a webdav PUT, like the not-chunked upload. Only difference is that it is not a complete file, but only a chunk.

And I'm pretty sure the content-type for the PUT is application/octet-stream, correct?

With the upload of the iOS app it is application/x-www-form-urlencoded which is the file content as one big string encoded as an url ...

tobiasKaminsky commented 2 years ago

I debugged it on Android:

PUT method has a request entity
This request entity has then a content-type, which is the mime-type of the file: https://github.com/nextcloud/android-library/blob/d4c0ac5a6079187292930bfb716624cd2ce33af3/library/src/main/java/com/owncloud/android/lib/resources/files/ChunkedFileUploadRemoteOperation.java#L300

So it seems that you cannot rely on content-type.

marinofaggiana commented 2 years ago

Seems to be correct to use MIME types, I read:

Browsers use the MIME type, not the file extension, to determine how to process a URL, so it's important that web servers send the correct MIME type in the response's Content-Type header. If this is not correctly configured, browsers are likely to misinterpret the contents of files, sites will not work correctly, and downloaded files may be mishandled...

ne20002 commented 2 years ago

To be honest, browsers/client as well as servers and all systems in between relay on the content-type. The content-type declares how the content shall be interpreted and/or processed. This applies for request and response.

This is, what happens here and causes problems with the web application firewall. While the Android client and the Linux client use a content-type, that is valid for binary data, the iOS client instead uses the content-type 'x-www-form-urlencoded' and also encodes the body as url (which also makes it much longer ...).

The WAF now interpretes the body as form data, because this is what the content header declares, and runs into lots of errors.

While the targeted system, the Nextcloud server, knows what to do with a request based on its implementation (in this case 'decode the body and write as file') and the url called and can maybe ignore the content-type as long as the body can successfulls decoded, the web application firewall in the transport fails due to the incorrect content-type.

So I suggest to either use a correct content-type based on the file (e.g image/jpg) or the generic content-type 'application/octet-stream' for binary content without further specification as this would also be sufficient.

I'm pretty sure that if the iOS client uses mime-type based on file type or application/octet-stream for binary data uploads the web application firewall will not complain anymore.

I also believe that using a file type mime type is not correct for a chunked upload as ist is just not that type of file but only a part of it and therefor generic binary is more suitable. But for the web application firewall i think either way would be ok.

ne20002 commented 2 years ago

No progress on this issue? I now disabled the Web Application Firewall for the uploads path and iOS user-agent. But for me this is an issue. content-type and content-encoding shall always be set correct and used correctly so that a WAF can scan the request/response and act on checks done.

mdkumargce commented 2 years ago

This issue still exists with Nextcloud IOS 4.4.0.

@ne20002 can you please tell me how exactly did you disable the WAF for a path and an user-agent? Thanks

ne20002 commented 2 years ago

Hi @mdkumargce those are the rules that worked for me:

# Disable iOS client blocking
SecRule REQUEST_HEADERS:User-Agent "@pm Nextcloud-iOS" \
   "id:900200,\
   phase:1,\
   nolog,\
   pass,\
   ctl:ruleRemoveById=949110"

#Exclude client file upload
SecRule REQUEST_FILENAME "@beginsWith /remote.php/dav/uploads" \
   "id:1005,\
   nolog,\
   allow,\
   ctl:ruleEngine=Off"

I actually don't like to disable rules or exclude requests from being processed by my WAF, but I haven't found any other solution as trying to disable only certain rules ended up in a mess of rules exclusions ... As far as the iOS client is using correct content-type I will give it another try.

Resmond-s commented 11 months ago

We have the same issues as described above. For example, to process some Request, an LDAP Injektion Attack Rule (211030) needs to be disabled - which should clearly be a no-go security wise. Example Rules that cause problems specifically with the iOS-App: 210380, 211030 (Comodo WAF).

As Web Application Firewalls are quite the standard for secure web services and the android/desktop/web clients of Nextcloud have no problems, i would like to push this issue.

Disabling rules here is a temporary alternative and for some use-cases actually the usual procedure, but Nextcloud iOS produces so many issues (in comparison with the other clients) that it could pose significant security issues for larger deployments.

Resmond-s commented 10 months ago

To add more backround information:

Limited (as tested so far) to the upload function through sharing dialogue (if one tries to upload files through the files app or from other applications). Uploading initialized through the plus button wihin the nextcloud app works.
This all boils down to the way Nextcloud ios handles file uploads given to it through the sharing dialogue (compared multipart/form-data in the ios app internal upload call and all upload variants in android, desktop and web).
this triggers (not wanted!) in-depth scans of uploaded file content by modsecurity which recognizes false-positives due to wrong interpretation of file-upload data as general header data (non-file data, which usually is smaller than 131 kB)
even if modsecurity rules are disabled (big security risk!), the size of the request header is enormous (due to files included there) and results in further problems when handling non-file data it security wise

As far as i can see, the problem lies with the declaration of files:

IS: The PUT Header includes the parameter "Content-Length: [big value]" and "content-type: application/x-www-form-urlencoded"
the size corresponds to the file, and the content-type suggests the file is uploaded without an corresponding type such as "audio/wav"
SHOULD: When uploading via the Nextcloud Web-Interface for example, the content-type is some multipart/form-data like "audio/wav" & there are additional parameters (not existing with ios) such as "Content-Disposition: attachment; filename="[some filename]"" -> this does not trigger problems when handling requests by modsecurity

In-Depth technical stuff:

Explanation of modsecurity file handling (file-upload data vs. non-file data and respective limits) (https://www.loadbalancer.org/blog/handling-large-requests-with-a-waf-while-avoiding-dos-attacks/
Sample error case (comparable to the one triggered by nextcloud ios) for the above situation: https://security.stackexchange.com/questions/267536/modsecurity-how-to-increase-secrequestbodylimit-for-specific-website-request-ur

Resmond-s commented 10 months ago

As this issue is quite relevant for admins of secured instances (and a long-standing issue further mentioned in #2363 and forums ), maybe a similar handling of external requests like in android or ios-app internal handling can be implemented? Although i am not knwledgable about the specific handling/and code snippets. @tobiasKaminsky

nextcloud / ios