Mac OS desktop client triggers `mod_security` violations where web interface, Finder's WebDAV client, and mobile app don't

posita commented 3 years ago

Actual behaviour

When attempting to create a new account from the latest client on Mac OS, I am faced with this:

screenshot

The following all (still) work for the exact same user and credentials:

✅ Web ✅ Mac OS Finder's WebDAV client (⌘K from Finder) ✅ Mobile app

Server configuration

Operating system: DreamHost (Ubuntu?)

Web server: Apache

Database: MySQL

PHP version: 7.4

ownCloud version: 10.8.0.4

Updated from an older ownCloud or fresh install: updated from 10.7.0

Where did you install ownCloud from: tar ball

Signing status (ownCloud 9.0 and above): ???

No errors have been found.

The content of config/config.php:

{
    "system": {
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "localhost",
            "…"
        ],
        "datadirectory": "\/home\/…\/data",
        "overwrite.cli.url": "http:\/\/localhost",
        "dbtype": "mysql",
        "version": "10.8.0.4",
        "dbname": "…",
        "dbhost": "…",
        "dbtableprefix": "oc_",
        "mysql.utf8mb4": true,
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "logtimezone": "UTC",
        "objectstore": {
            "class": "OCA\\Files_Primary_S3\\S3Storage",
            "arguments": {
                "bucket": "…",
                "options": {
                    "version": "2006-03-01",
                    "region": "us-west-1",
                    "credentials": {
                        "key": "***REMOVED SENSITIVE VALUE***",
                        "secret": "***REMOVED SENSITIVE VALUE***"
                    },
                    "endpoint": "http:\/\/s3.us-west-1.wasabisys.com\/",
                    "use_path_style_endpoint": true
                }
            }
        },
        "apps_paths": [
            {
                "path": "\/home\/…\/apps",
                "url": "\/apps",
                "writable": false
            },
            {
                "path": "\/home\/…\/apps-external",
                "url": "\/apps-external",
                "writable": true
            }
        ],
        "installed": true,
        "instanceid": "oct7apj43gaw",
        "maintenance": false,
        "loglevel": 2
    }
}

List of activated apps:

Enabled:
  - activity:
    - Version: 2.6.1
    - Path: .../apps/activity
  - calendar:
    - Version: 1.6.4
    - Path: .../apps-external/calendar
  - comments:
    - Version: 0.3.0
    - Path: .../apps/comments
  - configreport:
    - Version: 0.2.0
    - Path: .../apps/configreport
  - contacts:
    - Version: 1.5.5
    - Path: .../apps-external/contacts
  - dav:
    - Version: 0.6.0
    - Path: .../apps/dav
  - federatedfilesharing:
    - Version: 0.5.0
    - Path: .../apps/federatedfilesharing
  - federation:
    - Version: 0.1.0
    - Path: .../apps/federation
  - files:
    - Version: 1.5.2
    - Path: .../apps/files
  - files_external:
    - Version: 0.8.0
    - Path: .../apps/files_external
  - files_mediaviewer:
    - Version: 1.0.4
    - Path: .../apps/files_mediaviewer
  - files_pdfviewer:
    - Version: 0.12.2
    - Path: .../apps/files_pdfviewer
  - files_primary_s3:
    - Version: 1.1.2
    - Path: .../apps-external/files_primary_s3
  - files_sharing:
    - Version: 0.14.0
    - Path: .../apps/files_sharing
  - files_texteditor:
    - Version: 2.3.1
    - Path: .../apps/files_texteditor
  - files_trashbin:
    - Version: 0.9.1
    - Path: .../apps/files_trashbin
  - files_versions:
    - Version: 1.3.0
    - Path: .../apps/files_versions
  - firstrunwizard:
    - Version: 1.2.0
    - Path: .../apps/firstrunwizard
  - market:
    - Version: 0.6.1
    - Path: .../apps/market
  - notifications:
    - Version: 0.5.4
    - Path: .../apps/notifications
  - provisioning_api:
    - Version: 0.5.0
    - Path: .../apps/provisioning_api
  - systemtags:
    - Version: 0.3.0
    - Path: .../apps/systemtags
  - templateeditor:
    - Version: 0.4.0
    - Path: .../apps/templateeditor
  - updatenotification:
    - Version: 0.2.1
    - Path: .../apps/updatenotification
Disabled:
  - admin_audit:
    - Path: .../apps/admin_audit
  - announcementcenter:
    - Path: .../apps/announcementcenter
  - customgroups:
    - Path: .../apps/customgroups
  - encryption:
    - Path: .../apps/encryption
  - enterprise_key:
    - Path: .../apps/enterprise_key
  - external:
    - Path: .../apps/external
  - files_antivirus:
    - Path: .../apps/files_antivirus
  - files_classifier:
    - Path: .../apps/files_classifier
  - files_external_dropbox:
    - Path: .../apps/files_external_dropbox
  - files_external_ftp:
    - Path: .../apps/files_external_ftp
  - files_ldap_home:
    - Path: .../apps/files_ldap_home
  - files_lifecycle:
    - Path: .../apps/files_lifecycle
  - firewall:
    - Path: .../apps/firewall
  - graphapi:
    - Path: .../apps/graphapi
  - guests:
    - Path: .../apps/guests
  - metrics:
    - Path: .../apps/metrics
  - oauth2:
    - Path: .../apps/oauth2
  - openidconnect:
    - Path: .../apps/openidconnect
  - password_policy:
    - Path: .../apps/password_policy
  - ransomware_protection:
    - Path: .../apps/ransomware_protection
  - sharepoint:
    - Path: .../apps/sharepoint
  - systemtags_management:
    - Path: .../apps/systemtags_management
  - theme-enterprise:
    - Path: .../apps/theme-enterprise
  - user_external:
    - Path: .../apps/user_external
  - user_ldap:
    - Path: .../apps/user_ldap
  - user_shibboleth:
    - Path: .../apps/user_shibboleth
  - web:
    - Path: .../apps/web
  - windows_network_drive:
    - Path: .../apps/windows_network_drive
  - wopi:
    - Path: .../apps/wopi
  - workflow:
    - Path: .../apps/workflow

Are you using external storage, if yes which one: local + S3

Are you using encryption: no

cortho commented 3 years ago

I guess it is big help for the devs when you provide your owncloud.log around the time when the 5xx error appeared

posita commented 3 years ago

There is no corresponding owncloud.log entry to either the client's attempt to log me in or my attempt to replicate that with curl:

% curl --request PROPFIND --verbose --user <USERNAME> 'https://<HOST>/remote.php/webdav/'
Enter host password for user '<USERNAME>':
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 2606:4700:3030::6815:5903:443...
* Immediate connect fail for 2606:4700:3030::6815:5903: No route to host
*   Trying 2606:4700:3030::ac43:9b35:443...
* Immediate connect fail for 2606:4700:3030::ac43:9b35: No route to host
*   Trying 172.67.155.53:443...
* Connected to <HOST> (172.67.155.53) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /opt/local/macports-20200907/share/curl/curl-ca-bundle.crt
*  CApath: none
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [122 bytes data]
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
{ [19 bytes data]
* TLSv1.3 (IN), TLS handshake, Certificate (11):
{ [2324 bytes data]
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
{ [80 bytes data]
* TLSv1.3 (IN), TLS handshake, Finished (20):
{ [52 bytes data]
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.3 (OUT), TLS handshake, Finished (20):
} [52 bytes data]
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=US; ST=California; L=San Francisco; O=Cloudflare, Inc.; CN=sni.cloudflaressl.com
*  start date: Apr 16 00:00:00 2021 GMT
*  expire date: Apr 15 23:59:59 2022 GMT
*  subjectAltName: host "<HOST>" matched cert's "<HOST>"
*  issuer: C=US; O=Cloudflare, Inc.; CN=Cloudflare Inc ECC CA-3
*  SSL certificate verify ok.
* Using HTTP2, server supports multiplexing
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Server auth using Basic with user '<USERNAME>'
* Using Stream ID: 1 (easy handle 0x7faf1500a600)
} [5 bytes data]
> PROPFIND /remote.php/webdav/ HTTP/2
> Host: <HOST>
> authorization: Basic …=
> user-agent: curl/7.78.0
> accept: */*
> 
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [230 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [230 bytes data]
* old SSL session ID is stale, removing
{ [5 bytes data]
* Connection state changed (MAX_CONCURRENT_STREAMS == 256)!
} [5 bytes data]
< HTTP/2 207 
< date: Fri, 13 Aug 2021 17:37:14 GMT
< content-type: application/xml; charset=utf-8
< cache-control: no-store, no-cache, must-revalidate
< cf-railgun: direct (waiting for pending WAN connection)
< content-security-policy: default-src 'none';
< dav: 1, 3, extended-mkcol, 2
< expires: Thu, 19 Nov 1981 08:52:00 GMT
< pragma: no-cache
< set-cookie: oc_sessionPassphrase=…; path=/; secure; HttpOnly; SameSite=Strict
< vary: Brief,Prefer,User-Agent
< x-content-type-options: nosniff
< x-download-options: noopen
< x-frame-options: SAMEORIGIN
< x-permitted-cross-domain-policies: none
< x-robots-tag: none
< x-xss-protection: 0
< cf-cache-status: DYNAMIC
< set-cookie: oct7apj43gaw=…; path=/; secure; HttpOnly; SameSite=Strict
< set-cookie: cookie_test=test; expires=Fri, 13-Aug-2021 18:37:14 GMT; Max-Age=3600
< expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
< report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=…}],"group":"cf-nel","max_age":604800}
< nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
< server: cloudflare
< cf-ray: …
< alt-svc: h3-27=":443"; ma=86400, h3-28=":443"; ma=86400, h3-29=":443"; ma=86400, h3=":443"; ma=86400
< 
{ [228 bytes data]

100  5870    0  5870    0     0   5879      0 --:--:-- --:--:-- --:--:--  5881
100  5870    0  5870    0     0   5878      0 --:--:-- --:--:-- --:--:--  5875
* Connection #0 to host <HOST> left intact
<?xml version="1.0"?>
<d:multistatus xmlns:d="DAV:" xmlns:s="http://sabredav.org/ns" xmlns:oc="http://owncloud.org/ns"><d:response><d:href>/remote.php/webdav/</d:href><d:propstat><d:prop><d:getlastmodified>Thu, 12 Aug 2021 23:15:23 GMT</d:getlastmodified><d:resourcetype><d:collection/></d:resourcetype><d:quota-used-bytes>5815349573</d:quota-used-bytes><d:quota-available-bytes>-3</d:quota-available-bytes><d:getetag>&quot;6115ab8b2faec&quot;</d:getetag></d:prop><d:status>HTTP/1.1 200 OK</d:status></d:propstat></d:response><d:response><d:href>/remote.php/webdav/.DS_Store</d:href><d:propstat><d:prop><d:getlastmodified>Thu, 12 Aug 2021 23:15:23 GMT</d:getlastmodified><d:getcontentlength>6148</d:getcontentlength><d:resourcetype/><d:getetag>&quot;6115ab8b11a68&quot;</d:getetag><d:getcontenttype>application/octet-stream</d:getcontenttype></d:prop><d:status>HTTP/1.1 200 OK</d:status></d:propstat><d:propstat><d:prop><d:quota-used-bytes/><d:quota-available-bytes/></d:prop><d:status>HTTP/1.1 404 Not Found</d:status></d:propstat></d:response><d:response><d:href>/remote.php/webdav/._.DS_Store</d:href><d:propstat><d:prop><d:getlastmodified>Thu, 12 Aug 2021 22:48:24 GMT</d:getlastmodified><d:getcontentlength>4096</d:getcontentlength><d:resourcetype/><d:getetag>&quot;6115a538f230e&quot;</d:getetag><d:getcontenttype>application/octet-stream</d:getcontenttype></d:prop><d:status>HTTP/1.1 200 OK</d:status></d:propstat><d:propstat><d:prop><d:quota-used-bytes/><d:quota-available-bytes/></d:prop><d:status>HTTP/1.1 404 Not Found</d:status></d:propstat></d:response><d:response><d:href>/remote.php/webdav/Data.sparsebundle/</d:href><d:propstat><d:prop><d:getlastmodified>Fri, 14 May 2021 13:26:50 GMT</d:getlastmodified><d:resourcetype><d:collection/></d:resourcetype><d:quota-used-bytes>30321128</d:quota-used-bytes><d:quota-available-bytes>-3</d:quota-available-bytes><d:getetag>&quot;609e7aabd190f&quot;</d:getetag></d:prop><d:status>HTTP/1.1 200 OK</d:status></d:propstat><d:propstat><d:prop><d:getcontentlength/><d:getcontenttype/></d:prop><d:status>HTTP/1.1 404 Not Found</d:status></d:propstat></d:response><d:response><d:href>/remote.php/webdav/Documents/</d:href><d:propstat><d:prop><d:getlastmodified>Sun, 09 May 2021 01:24:32 GMT</d:getlastmodified><d:resourcetype><d:collection/></d:resourcetype><d:quota-used-bytes>36227</d:quota-used-bytes><d:quota-available-bytes>-3</d:quota-available-bytes><d:getetag>&quot;609739d047575&quot;</d:getetag></d:prop><d:status>HTTP/1.1 200 OK</d:status></d:propstat><d:propstat><d:prop><d:getcontentlength/><d:getcontenttype/></d:prop><d:status>HTTP/1.1 404 Not Found</d:status></d:propstat></d:response><d:response><d:href>/remote.php/webdav/Photo-2021-07-24-10-51-22_5907.JPG</d:href><d:propstat><d:prop><d:getlastmodified>Sat, 24 Jul 2021 15:51:22 GMT</d:getlastmodified><d:getcontentlength>2255706</d:getcontentlength><d:resourcetype/><d:getetag>&quot;61042daca32f4&quot;</d:getetag><d:getcontenttype>image/jpeg</d:getcontenttype></d:prop><d:status>HTTP/1.1 200 OK</d:status></d:propstat><d:propstat><d:prop><d:quota-used-bytes/><d:quota-available-bytes/></d:prop><d:status>HTTP/1.1 404 Not Found</d:status></d:propstat></d:response><d:response><d:href>/remote.php/webdav/Photo-2021-07-24-10-51-32_5908.JPG</d:href><d:propstat><d:prop><d:getlastmodified>Sat, 24 Jul 2021 15:51:32 GMT</d:getlastmodified><d:getcontentlength>1871025</d:getcontentlength><d:resourcetype/><d:getetag>&quot;61042dbd67cb8&quot;</d:getetag><d:getcontenttype>image/jpeg</d:getcontenttype></d:prop><d:status>HTTP/1.1 200 OK</d:status></d:propstat><d:propstat><d:prop><d:quota-used-bytes/><d:quota-available-bytes/></d:prop><d:status>HTTP/1.1 404 Not Found</d:status></d:propstat></d:response><d:response><d:href>/remote.php/webdav/Photos/</d:href><d:propstat><d:prop><d:getlastmodified>Sun, 09 May 2021 01:24:31 GMT</d:getlastmodified><d:resourcetype><d:collection/></d:resourcetype><d:quota-used-bytes>1011464</d:quota-used-bytes><d:quota-available-bytes>-3</d:quota-available-bytes><d:getetag>&quot;609739cfe9452&quot;</d:getetag></d:prop><d:status>HTTP/1.1 200 OK</d:status></d:propstat><d:propstat><d:prop><d:getcontentlength/><d:getcontenttype/></d:prop><d:status>HTTP/1.1 404 Not Found</d:status></d:propstat></d:response><d:response><d:href>/remote.php/webdav/media/</d:href><d:propstat><d:prop><d:getlastmodified>Fri, 14 May 2021 13:27:06 GMT</d:getlastmodified><d:resourcetype><d:collection/></d:resourcetype><d:quota-used-bytes>4423853555</d:quota-used-bytes><d:quota-available-bytes>-3</d:quota-available-bytes><d:getetag>&quot;609e81ce687b1&quot;</d:getetag></d:prop><d:status>HTTP/1.1 200 OK</d:status></d:propstat><d:propstat><d:prop><d:getcontentlength/><d:getcontenttype/></d:prop><d:status>HTTP/1.1 404 Not Found</d:status></d:propstat></d:response><d:response><d:href>/remote.php/webdav/misc/</d:href><d:propstat><d:prop><d:getlastmodified>Sat, 10 Jul 2021 23:03:48 GMT</d:getlastmodified><d:resourcetype><d:collection/></d:resourcetype><d:quota-used-bytes>1349491568</d:quota-used-bytes><d:quota-available-bytes>-3</d:quota-available-bytes><d:getetag>&quot;60ea275c2d164&quot;</d:getetag></d:prop><d:status>HTTP/1.1 200 OK</d:status></d:propstat><d:propstat><d:prop><d:getcontentlength/><d:getcontenttype/></d:prop><d:status>HTTP/1.1 404 Not Found</d:status></d:propstat></d:response><d:response><d:href>/remote.php/webdav/ownCloud%20Manual.pdf</d:href><d:propstat><d:prop><d:getlastmodified>Sun, 09 May 2021 01:24:32 GMT</d:getlastmodified><d:getcontentlength>6498656</d:getcontentlength><d:resourcetype/><d:getetag>&quot;609739d0ce448&quot;</d:getetag><d:getcontenttype>application/pdf</d:getcontenttype></d:prop><d:status>HTTP/1.1 200 OK</d:status></d:propstat><d:propstat><d:prop><d:quota-used-bytes/><d:quota-available-bytes/></d:prop><d:status>HTTP/1.1 404 Not Found</d:status></d:propstat></d:response></d:multistatus>

posita commented 3 years ago

It turns out this was being blocked by mod_security in Apache. But what's weird is that it was only blocked for the desktop client, not for other avenues (like curl and Finder's Connect to Server… functionality). Unsetting the option in DreamHost removed the symptom.

posita commented 3 years ago

Hmmm…this one's going to be hard to push forward. I filed coreruleset/coreruleset#2176, but I'm not hopeful, since this will likely require coordination with my hosting provider. My guess is that no party involved will be excited to help. 🤷 We'll see….

~Is there anything the ownCloud desktop client could be doing to make things harder for itself? (Missing or present headers, user-agent, etc.?)~ _Hmmm…maybe not. See, e.g., SpiderLabs/ModSecurity#1566. This is unfortunate, since I use a hosting provider where I don't have control over the mod_security configuration._

fzipi commented 3 years ago

Hi @ho4ho @posita!

While this ends being blocked by CRS, the problem looks very similar to https://github.com/coreruleset/coreruleset/issues/1838#issuecomment-678771144, so I've filed https://github.com/nextcloud/server/issues/25250, but the user didn't followed.. IMHO, the client is not sending the proper content-type (should be application/xml or similar), so it ends not being parsed properly as XML.

Can you try to add -H "Content-Type : application/xml" to your curl?

posita commented 3 years ago

@fzipi, thanks for the links. I'm in the process of checking now and will report back. To be clear, because I don't have control over the mod_security configuration (since this is hosted on DreamHost), I am re-enabling mod_security via DreamHost's "Extra Web Security" feature, waiting for that to propagate, then trying my curl command again with the added header. I will report back with what I find.

posita commented 3 years ago

@ho4ho, do you have contributor access to the client repo? Is this issue a good candidate to use GitHub's inter-repository issue transfer feature?

@fzipi, if I understand your comment correctly, you also believe this may be a client issue?

fzipi commented 3 years ago

Yes, I think the client is not adding the proper content-type headers, and it is hitting the web server with text/plain by default. Because owncloud/nextcloud will process this in their standard way, it always work (I mean, it will try to read XML).

But then the client should be sending the content-type: application/xml according to its content.

posita commented 3 years ago

@fzipi, FWIW, my curl isn't sending a payload with PROPFIND (although it was not sending the Depth: … header, either). Maybe you're thinking of Accept: …?

In any event, the following work (as above), even with mod_security enabled:

curl --request PROPFIND --header 'Accept: application/xml' --header 'Content-type: application/xml' --header 'Depth: 0' --header 'User-Agent: Mozilla/5.0 (Macintosh) mirall/2.8.2 (build 4246) (ownCloud, osx-19.6.0 ClientArchitecture: x86_64 OsArchitecture: x86_64)' --verbose --user <USERNAME> 'https://…/remote.php/webdav/'
curl --request PROPFIND --header 'Accept: text/plain' --header 'Content-type: text/plain' --header 'Depth: 0' --header 'User-Agent: Mozilla/5.0 (Macintosh) mirall/2.8.2 (build 4246) (ownCloud, osx-19.6.0 ClientArchitecture: x86_64 OsArchitecture: x86_64)' --verbose --user <USERNAME> 'https://…/remote.php/webdav/'

fzipi commented 3 years ago

🤔 if propfind doesn't have payload, then this is not a problem with payload 😄 .

Taking a third look at this specific one, maybe you just need to enable exceptions in the domain. Meaning the CRS has specific exceptions for (own|next)cloud, but they need to be enabled. Maybe you can verify the logs looking for the error 418 there and see if it is blocking the PROPFIND directive first, or what rule it is matching?

fzipi commented 3 years ago

Ok, sorry about this ☝️ . I didn't checked the url and of course this is no place for this. Shall we continue this in the linked coreruleset issue?

fzipi commented 3 years ago

Well, in the linked ticket the problem is in a PROPFIND within REQUEST_BODY. So I think it is still the same problem?

posita commented 2 years ago

github-actions[bot] commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 10 days if no further activity occurs. Thank you for your contributions.

github-actions[bot] commented 2 years ago

This issue has been automatically closed.

owncloud / core

Mac OS desktop client triggers `mod_security` violations where web interface, Finder's WebDAV client, and mobile app don't #39109

Actual behaviour

Server configuration