nextcloud / nextcloudpi

📦 Build code for NextcloudPi: Raspberry Pi, Odroid, Rock64, curl installer...
https://nextcloudpi.com
2.58k stars 298 forks source link

Modsecurity WAF stopping syncing of large files from desktop client #992

Closed Iolaum closed 5 years ago

Iolaum commented 5 years ago

Problem description

Syncing files larger than 5mb results in sync errors most of the time. Disabling modsecurity fixes the problem.

How to replicate the problem.

Enable modsecurity waf on NCP. Then copy a folder containing ~10 video files 200-500mb in size inside a folder that syncs from the laptop to ncp. When modsecurity is active sync will fail with errors and no files will be uploaded to ncp from the laptop. With modsecurity disabled the files will sync - a force sync may be needed for the client to notice the changes faster than later - after that the files will sync correctly.

System information

NextCloudPi diagnostics ``` NextCloudPi version v1.16.1 NextCloudPi image NextCloudPi_03-09-19 distribution Raspbian GNU/Linux 10 \n \l automount yes USB devices sda datadir /media/USBdrive/ncdata data in SD no data filesystem ext2/ext3 data disk usage 260G/916G rootfs usage 2.0G/15G swapfile /media/ncphd/swap dbdir /media/USBdrive/ncdb2 Nextcloud check ok Nextcloud version 16.0.4.1 HTTPD service up PHP service up MariaDB service up Redis service up Postfix service up internet check ok port check 80 open port check 443 open IP ***REMOVED SENSITIVE VALUE*** gateway ***REMOVED SENSITIVE VALUE*** interface eth0 certificates ***REMOVED SENSITIVE VALUE*** NAT loopback yes uptime 3:13 ```
Nextcloud configuration ``` { "system": { "passwordsalt": "***REMOVED SENSITIVE VALUE***", "secret": "***REMOVED SENSITIVE VALUE***", "trusted_domains": { "0": "localhost", "5": "nextcloudpi.local", "7": "nextcloudpi", "8": "nextcloudpi.lan", "11": "$GLOBAL_IP", "1": "$LOCAL_IP", "20": "$OLD_DOMAIN_NAME", "21": "_", "22": "_", "4": "$CURRENT_DOMAIN", "3": "$CURRENT_DOMAIN" }, "datadirectory": "***REMOVED SENSITIVE VALUE***", "dbtype": "mysql", "version": "16.0.4.1", "overwrite.cli.url": "https:\/\/$CURRENT_DOMAIN\/", "dbname": "***REMOVED SENSITIVE VALUE***", "dbhost": "***REMOVED SENSITIVE VALUE***", "dbport": "", "dbtableprefix": "oc_", "mysql.utf8mb4": true, "dbuser": "***REMOVED SENSITIVE VALUE***", "dbpassword": "***REMOVED SENSITIVE VALUE***", "installed": true, "instanceid": "***REMOVED SENSITIVE VALUE***", "memcache.local": "\\OC\\Memcache\\Redis", "memcache.locking": "\\OC\\Memcache\\Redis", "redis": { "host": "***REMOVED SENSITIVE VALUE***", "port": 0, "timeout": 0, "password": "***REMOVED SENSITIVE VALUE***" }, "tempdirectory": "\/media\/USBdrive\/ncdata\/tmp", "mail_smtpmode": "sendmail", "mail_smtpauthtype": "LOGIN", "mail_from_address": "***REMOVED SENSITIVE VALUE***", "mail_domain": "***REMOVED SENSITIVE VALUE***", "overwriteprotocol": "https", "maintenance": false, "logfile": "\/media\/USBdrive\/ncdata\/nextcloud.log", "loglevel": "2", "log_type": "file", "jpeg_quality": "60", "preview_max_x": "2048", "preview_max_y": "2048", "theme": "" } } ```
HTTPd logs ``` [Sat Sep 07 00:00:04.680226 2019] [mpm_event:notice] [pid 11042:tid 1996227088] AH00489: Apache/2.4.38 (Raspbian) OpenSSL/1.1.1c configured -- resuming normal operations [Sat Sep 07 00:00:04.680273 2019] [core:notice] [pid 11042:tid 1996227088] AH00094: Command line: '/usr/sbin/apache2' [Sat Sep 07 17:22:32.598159 2019] [mpm_event:notice] [pid 11042:tid 1996227088] AH00493: SIGUSR1 received. Doing graceful restart [Sat Sep 07 17:22:32.715569 2019] [mpm_event:notice] [pid 11042:tid 1996227088] AH00489: Apache/2.4.38 (Raspbian) OpenSSL/1.1.1c configured -- resuming normal operations [Sat Sep 07 17:22:32.715619 2019] [core:notice] [pid 11042:tid 1996227088] AH00094: Command line: '/usr/sbin/apache2' [Sat Sep 07 17:22:38.260280 2019] [mpm_event:notice] [pid 11042:tid 1996227088] AH00491: caught SIGTERM, shutting down [Sat Sep 07 17:23:17.762948 2019] [mpm_event:notice] [pid 780:tid 1996284432] AH00489: Apache/2.4.38 (Raspbian) OpenSSL/1.1.1c configured -- resuming normal operations [Sat Sep 07 17:23:17.763321 2019] [core:notice] [pid 780:tid 1996284432] AH00094: Command line: '/usr/sbin/apache2' ```
Database logs ``` 2019-09-07 17:23:38 0 [Note] InnoDB: Compressed tables use zlib 1.2.11 2019-09-07 17:23:38 0 [Note] InnoDB: Number of pools: 1 2019-09-07 17:23:38 0 [Note] InnoDB: Using generic crc32 instructions 2019-09-07 17:23:38 0 [Note] InnoDB: Initializing buffer pool, total size = 384M, instances = 1, chunk size = 128M 2019-09-07 17:23:38 0 [Note] InnoDB: Completed initialization of buffer pool 2019-09-07 17:23:38 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 2019-09-07 17:23:38 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2019-09-07 17:23:38 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2019-09-07 17:23:38 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2019-09-07 17:23:38 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB. 2019-09-07 17:23:38 0 [Note] InnoDB: Waiting for purge to start 2019-09-07 17:23:38 0 [Note] InnoDB: 10.3.15 started; log sequence number 1609206609; transaction id 8388357 2019-09-07 17:23:38 0 [Note] InnoDB: Loading buffer pool(s) from /media/ncphd/ncdb2/ib_buffer_pool 2019-09-07 17:23:38 0 [Note] Plugin 'FEEDBACK' is disabled. 2019-09-07 17:23:39 0 [Note] Server socket created on IP: '127.0.0.1'. 2019-09-07 17:23:39 0 [Note] Reading of all Master_info entries succeded 2019-09-07 17:23:39 0 [Note] Added new Master_info '' to hash table 2019-09-07 17:23:39 0 [Note] /usr/sbin/mysqld: ready for connections. Version: '10.3.15-MariaDB-1' socket: '/run/mysqld/mysqld.sock' port: 3306 Raspbian testing-staging 2019-09-07 17:23:53 0 [Note] InnoDB: Buffer pool(s) load completed at 190907 17:23:53 ```
Nextcloud logs ``` {"reqId":"94qyaZkhuIkk99FSKbUs","level":1,"time":"2019-09-07T05:32:01+00:00","remoteAddr":"","user":"--","app":"updater","method":"","url":"--","message":"\\OC\\Repair::info: Repair info: SCSS cache cleared","userAgent":"--","version":"16.0.3.0"} {"reqId":"94qyaZkhuIkk99FSKbUs","level":1,"time":"2019-09-07T05:32:02+00:00","remoteAddr":"","user":"--","app":"updater","method":"","url":"--","message":"\\OC\\Repair::info: Repair info: JS cache cleared","userAgent":"--","version":"16.0.3.0"} {"reqId":"94qyaZkhuIkk99FSKbUs","level":1,"time":"2019-09-07T05:32:02+00:00","remoteAddr":"","user":"--","app":"updater","method":"","url":"--","message":"\\OC\\Repair::step: Repair step: Clear every generated avatar on major updates","userAgent":"--","version":"16.0.3.0"} {"reqId":"94qyaZkhuIkk99FSKbUs","level":1,"time":"2019-09-07T05:32:02+00:00","remoteAddr":"","user":"--","app":"updater","method":"","url":"--","message":"\\OC\\Repair::step: Repair step: Add preview background cleanup job","userAgent":"--","version":"16.0.3.0"} {"reqId":"94qyaZkhuIkk99FSKbUs","level":1,"time":"2019-09-07T05:32:02+00:00","remoteAddr":"","user":"--","app":"updater","method":"","url":"--","message":"\\OC\\Repair::step: Repair step: Queue a one-time job to cleanup old backups of the updater","userAgent":"--","version":"16.0.3.0"} {"reqId":"94qyaZkhuIkk99FSKbUs","level":1,"time":"2019-09-07T05:32:02+00:00","remoteAddr":"","user":"--","app":"updater","method":"","url":"--","message":"\\OC\\Repair::step: Repair step: Repair pending cron jobs","userAgent":"--","version":"16.0.3.0"} {"reqId":"94qyaZkhuIkk99FSKbUs","level":1,"time":"2019-09-07T05:32:02+00:00","remoteAddr":"","user":"--","app":"updater","method":"","url":"--","message":"\\OC\\Repair::info: Repair info: No need to repair pending cron jobs.","userAgent":"--","version":"16.0.3.0"} {"reqId":"94qyaZkhuIkk99FSKbUs","level":1,"time":"2019-09-07T05:32:02+00:00","remoteAddr":"","user":"--","app":"updater","method":"","url":"--","message":"\\OC\\Repair::step: Repair step: Extract the vcard uid and store it in the db","userAgent":"--","version":"16.0.3.0"} {"reqId":"94qyaZkhuIkk99FSKbUs","level":1,"time":"2019-09-07T05:32:02+00:00","remoteAddr":"","user":"--","app":"updater","method":"","url":"--","message":"\\OC\\Repair::step: Repair step: Cleanup invalid photocache files for carddav","userAgent":"--","version":"16.0.3.0"} {"reqId":"94qyaZkhuIkk99FSKbUs","level":1,"time":"2019-09-07T05:32:02+00:00","remoteAddr":"","user":"--","app":"updater","method":"","url":"--","message":"\\OC\\Repair::step: Repair step: Add background job to cleanup login flow v2 tokens","userAgent":"--","version":"16.0.3.0"} {"reqId":"94qyaZkhuIkk99FSKbUs","level":1,"time":"2019-09-07T05:32:02+00:00","remoteAddr":"","user":"--","app":"updater","method":"","url":"--","message":"\\OC\\Repair::step: Repair step: Remove potentially over exposing share links","userAgent":"--","version":"16.0.3.0"} {"reqId":"94qyaZkhuIkk99FSKbUs","level":1,"time":"2019-09-07T05:32:02+00:00","remoteAddr":"","user":"--","app":"updater","method":"","url":"--","message":"\\OC\\Repair::info: Repair info: No need to remove link shares.","userAgent":"--","version":"16.0.3.0"} {"reqId":"94qyaZkhuIkk99FSKbUs","level":1,"time":"2019-09-07T05:32:02+00:00","remoteAddr":"","user":"--","app":"updater","method":"","url":"--","message":"\\OC\\Repair::step: Repair step: Cleanup cypress files from viewer app","userAgent":"--","version":"16.0.3.0"} {"reqId":"94qyaZkhuIkk99FSKbUs","level":1,"time":"2019-09-07T05:32:28+00:00","remoteAddr":"","user":"--","app":"updater","method":"","url":"--","message":"\\OC\\Updater::startCheckCodeIntegrity: Starting code integrity check...","userAgent":"--","version":"16.0.3.0"} {"reqId":"XXNA-zueEsJaieJCYsLBQAAAAAI","level":0,"time":"2019-09-07T05:32:49+00:00","remoteAddr":"88.98.252.2","user":"--","app":"no app in context","method":"GET","url":"\/index.php\/204","message":"JSCombiner: successfully cached: merged-template-prepend.js","userAgent":"Mozilla\/5.0 (Android) Nextcloud-android\/3.7.2","version":"16.0.3.0"} {"reqId":"XXNBABdQKzL3etb32YJeugAAAEU","level":0,"time":"2019-09-07T05:32:49+00:00","remoteAddr":"88.98.252.2","user":"--","app":"no app in context","method":"GET","url":"\/index.php\/204","message":"JSCombiner: successfully cached: merged-template-prepend.js","userAgent":"Mozilla\/5.0 (Android) Nextcloud-android\/3.7.2","version":"16.0.3.0"} {"reqId":"94qyaZkhuIkk99FSKbUs","level":1,"time":"2019-09-07T05:32:56+00:00","remoteAddr":"","user":"--","app":"updater","method":"","url":"--","message":"\\OC\\Updater::finishedCheckCodeIntegrity: Finished code integrity check","userAgent":"--","version":"16.0.3.0"} {"reqId":"94qyaZkhuIkk99FSKbUs","level":1,"time":"2019-09-07T05:32:56+00:00","remoteAddr":"","user":"--","app":"updater","method":"","url":"--","message":"\\OC\\Updater::updateEnd: Update successful","userAgent":"--","version":"16.0.4.1"} {"reqId":"94qyaZkhuIkk99FSKbUs","level":1,"time":"2019-09-07T05:32:56+00:00","remoteAddr":"","user":"--","app":"updater","method":"","url":"--","message":"\\OC\\Updater::maintenanceDisabled: Turned off maintenance mode","userAgent":"--","version":"16.0.4.1"} {"reqId":"94qyaZkhuIkk99FSKbUs","level":1,"time":"2019-09-07T05:32:56+00:00","remoteAddr":"","user":"--","app":"updater","method":"","url":"--","message":"\\OC\\Updater::resetLogLevel: Reset log level to Warning(2)","userAgent":"--","version":"16.0.4.1"} ```
Desktop System Information Fedora MATE 30 running nextcloud-client-2.5.2-2.fc30 package.

Given the information mentioned about modsecurity here I also looked for modsec_audit.log However the file was empty!:

ls -la /var/log/apache2/ | grep modsec
-rw-r-----  1 root root      0 Jul 18 02:16 modsec_audit.log
# file has 0 size !!
Iolaum commented 5 years ago

Hm just saw that there is already a bug about this :o