[Bug]: Web UI for all users freezes when downloading any file with IDM or XDM (Multithreads downloader)

afzl-wtu commented 1 year ago

⚠️ This issue respects the following points: ⚠️

[X] This is a bug, not a question or a configuration/webserver/proxy issue.
[X] This issue is not already reported on Github OR Nextcloud Community Forum (I've searched it).
[X] Nextcloud Server is up to date. See Maintenance and Release Schedule for supported versions.
[X] I agree to follow Nextcloud's Code of Conduct.

Bug description

As discussed by someone else here nextcloud gui freezes when I download any file hosted on my nextcloud server with XDM (Multithreaded downloader) or IDM. It freezes web ui for all other users too. nextcloud.log

Steps to reproduce

Download any file with XDM or IDM. And while it is downloading you can not access your web ui. My server has 24 GB RAM and 4cores of arm processor on Oracle Cloud. Also it does not have other apps that cause server in load.

Expected behavior

It should work as normal.

Installation method

Community Web installer on a VPS or web space

Nextcloud Server version

27

Operating system

Debian/Ubuntu

PHP engine version

PHP 8.2

Web server

Other

Database engine version

PostgreSQL

Is this bug present after an update or on a fresh install?

Fresh Nextcloud Server install

Are you using the Nextcloud Server Encryption module?

Encryption is Disabled

What user-backends are you using?

[X] Default user-backend (database)
[ ] LDAP/ Active Directory
[ ] SSO - SAML
[ ] Other

Configuration report

Warning: Failed to set memory limit to 0 bytes (Current memory usage is 2097152 bytes) in Unknown on line 0
The current PHP memory limit is below the recommended value of 512MB.
{
    "system": {
        "memcache.local": "\\OC\\Memcache\\APCu",
        "apps_paths": [
            {
                "path": "\/var\/www\/html\/apps",
                "url": "\/apps",
                "writable": false
            },
            {
                "path": "\/var\/www\/html\/custom_apps",
                "url": "\/custom_apps",
                "writable": true
            }
        ],
        "memcache.distributed": "\\OC\\Memcache\\Redis",
        "memcache.locking": "\\OC\\Memcache\\Redis",
        "redis": {
            "host": "***REMOVED SENSITIVE VALUE***",
            "password": "***REMOVED SENSITIVE VALUE***",
            "port": 6379
        },
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "default_phone_region": "PK",
        "trusted_domains": [
            "nc.afzal.website"
        ],
        "trusted_proxies": "***REMOVED SENSITIVE VALUE***",
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "dbtype": "pgsql",
        "version": "27.0.1.2",
        "overwrite.cli.url": "https:\/\/nc.afzal.website",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbport": "",
        "dbtableprefix": "oc_",
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpmode": "smtp",
        "mail_sendmailmode": "smtp",
        "mail_from_address": "***REMOVED SENSITIVE VALUE***",
        "mail_domain": "***REMOVED SENSITIVE VALUE***",
        "mail_smtphost": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpport": "465",
        "mail_smtpauth": 1,
        "mail_smtpname": "***REMOVED SENSITIVE VALUE***",
        "mail_smtppassword": "***REMOVED SENSITIVE VALUE***",
        "htaccess.RewriteBase": "\/nextcloud"
    }
}

List of activated Apps

Warning: Failed to set memory limit to 0 bytes (Current memory usage is 2097152 bytes) in Unknown on line 0
The current PHP memory limit is below the recommended value of 512MB.
Enabled:
  - activity: 2.19.0
  - cfg_share_links: 4.1.0
  - checksum: 1.2.2
  - cloud_federation_api: 1.10.0
  - comments: 1.17.0
  - contactsinteraction: 1.8.0
  - cospend: 1.5.10
  - dashboard: 7.7.0
  - dav: 1.27.0
  - extract: 1.3.6
  - federatedfilesharing: 1.17.0
  - federation: 1.17.0
  - files: 1.22.0
  - files_external: 1.19.0
  - files_pdfviewer: 2.8.0
  - files_rightclick: 1.6.0
  - files_sharing: 1.19.0
  - files_trashbin: 1.17.0
  - files_versions: 1.20.0
  - files_zip: 1.4.0
  - firstrunwizard: 2.16.0
  - impersonate: 1.14.0
  - logreader: 2.12.0
  - lookup_server_connector: 1.15.0
  - mail: 3.2.4
  - metadata: 0.19.0
  - money: 0.20.0
  - nextcloud_announcements: 1.16.0
  - notes: 4.8.1
  - notifications: 2.15.0
  - oauth2: 1.15.1
  - passman: 2.4.7
  - password_policy: 1.17.0
  - passwords: 2023.8.30
  - phonetrack: 0.7.6
  - photos: 2.3.0
  - privacy: 1.11.0
  - provisioning_api: 1.17.0
  - quicknotes: 0.8.10
  - recommendations: 1.6.0
  - related_resources: 1.2.0
  - serverinfo: 1.17.0
  - settings: 1.9.0
  - sharebymail: 1.17.0
  - support: 1.10.0
  - survey_client: 1.15.0
  - systemtags: 1.17.0
  - text: 3.8.0
  - theming: 2.2.0
  - timemanager: 0.3.8
  - twofactor_backupcodes: 1.16.0
  - unsplash: 2.2.1
  - updatenotification: 1.17.0
  - user_status: 1.7.0
  - viewer: 2.1.0
  - weather_status: 1.7.0
  - workflowengine: 2.9.0
Disabled:
  - admin_audit: 1.17.0
  - bruteforcesettings: 2.7.0
  - circles: 27.0.1 (installed 27.0.1)
  - encryption: 2.15.0
  - suspicious_login: 5.0.0
  - twofactor_totp: 9.0.0
  - user_ldap: 1.17.0

Nextcloud Signing status

No errors have been found.

Nextcloud Logs

Log file is included into Bug Description as it exceeds github charcter limit.

Additional info

I am using nextcloud fpm docker version behind and with Caddy Webserver. Memory limit in Nextcloud web ui is 1024 MB. I do not know why only in logs I provide to you it reports 0 Byte.

solracsf commented 1 year ago

I don't think this is a bug but rather a setup issue; example: in your htop screenshots we can see your system hang on smbd daemon or mount.ntfs (the D letter) telling that your system is I/O waiting (and blocking) your other requests.

Also, if you download manager exhausts server connection slots (either kernel or apache) other connection will wait (and block) until other are freed.

There is nothing server component can do about that IMO.

afzl-wtu commented 1 year ago

I don't think this is a bug but rather a setup issue; example: in your htop screenshots we can see your system hang on smbd daemon or mount.ntfs (the D letter) telling that your system is I/O waiting (and blocking) your other requests.

Also, if you download manager exhausts server connection slots (either kernel or apache) other connection will wait (and block) until other are freed.

There is nothing server component can do about that IMO.

LOL I mentioned other person post a year ago in the link above. But issue is still there. That is why I created this issue and as a refernce I mentioned his post too.

solracsf commented 1 year ago

My answer stills. This is not a Nextcloud issue, but a server/setup issue.

joshtrichards commented 1 year ago

LOL I mentioned other person post a year ago in the link above. But issue is still there. That is why I created this issue and as a refernce I mentioned his post too.

Well, you referenced that other person's post as if it might be your own + it has more readily actionable information in it than your report so that's what we have to go on. :)

I saw you provided a recent Nextcloud log (thanks). What was a specific time window where you saw this behavior? This will help identify which log entries might be relevant (since it's over a week's worth of entries and many no doubt irrelevant).

I'd also be interested in:

your FPM logs (during the time this behavior occurs)
your Caddy logs (also only during the time this behavior occurs)
htop or similar output (same)

From what I did gather from a quick glance: you're using multiple External Storage mounts of various types. Were some/all of the source folders(s) for these download transactions located on mounts provisioned through External Storage?

Memory limit in Nextcloud web ui is 1024 MB. I do not know why only in logs I provide to you it reports 0 Byte.

Because your CLI-mode PHP configuration is not setup properly for Nextcloud. PHP has a dedicated php.ini (or loaded config directory) for each run mode (e.g. cli, apache, fpm). Presumably your FPM one is set to 1024, but not the others.

afzl-wtu commented 1 year ago

LOL I mentioned other person post a year ago in the link above. But issue is still there. That is why I created this issue and as a refernce I mentioned his post too.

Well, you referenced that other person's post as if it might be your own + it has more readily actionable information in it than your report so that's what we have to go on. :)

I saw you provided a recent Nextcloud log (thanks). What was a specific time window where you saw this behavior? This will help identify which log entries might be relevant (since it's over a week's worth of entries and many no doubt irrelevant).

I'd also be interested in:

your FPM logs (during the time this behavior occurs)

your Caddy logs (also only during the time this behavior occurs)

htop or similar output (same)

From what I did gather from a quick glance: you're using multiple External Storage mounts of various types. Were some/all of the source folders(s) for these download transactions located on mounts provisioned through External Storage?

Memory limit in Nextcloud web ui is 1024 MB. I do not know why only in logs I provide to you it reports 0 Byte.

Because your CLI-mode PHP configuration is not setup properly for Nextcloud. PHP has a dedicated php.ini (or loaded config directory) for each run mode (e.g. cli, apache, fpm). Presumably your FPM one is set to 1024, but not the others.

PHP FPM log when error occurs:

WARNING: [pool www] server reached pm.max_children setting (5), consider raising it

afzl-wtu commented 1 year ago

And Caddy log when error happens:

{"level":"error","ts":1691751556.7215335,"logger":"http.handlers.reverse_proxy","msg":"aborting with incomplete response","upstream":"127.0.0.1:9000","duration":9.865105177,"request":{"remote_ip":"138.2.138.227","remote_port":"39724","client_ip":"138.2.138.227","proto":"HTTP/1.1","method":"GET","host":"nc.afzal.website","uri":"/remote.php?downloadStartSecret=xmo3ir904","headers":{"Cookie":[],"Range":["bytes=638460174-658932226"],"Accept-Encoding":["gzip, deflate, br"],"X-Forwarded-For":["138.2.138.227"],"X-Forwarded-Proto":["https"],"X-Forwarded-Host":["nc.afzal.website"],"User-Agent":["Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/115.0.0.0 Safari/537.36"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"","server_name":"nc.afzal.website"}},"error":"writing: write tcp 10.0.0.10:443->138.2.138.227:39724: write: broken pipe"}

szaimen commented 1 year ago

WARNING: [pool www] server reached pm.max_children setting (5), consider raising it

Moving to the docker repo

joshtrichards commented 1 year ago

WARNING: [pool www] server reached pm.max_children setting (5), consider raising it

See #1982, #1977, and others.

Unfortunately PHP's own default for this parameter remains extremely low to do this day (although I understand why changing the default isn't desirable since it could cause problems in environments relying on the current default). See:

https://docs.nextcloud.com/server/latest/admin_manual/installation/server_tuning.html?highlight=fpm#tune-php-fpm

At the moment, the community Docker NC image has taken the approach of trying to expose the pm parameters in a semi-standard way upstream (in the official PHP Docker image used as the base for this image in this repository). The alternative of creating a new NC Docker image specific way of adjusting a PHP parameter isn't as preferable. Unfortunately, so far, it's been rejected upstream - i.e. https://github.com/docker-library/php/pull/1328

Fortunately, all pm settings can be overridden and set to an appropriate parameter for your local environment. See the above referenced issues, check the Nextcloud Help Forum, and do some search engine checks. It's fairly routine FPM parameter optimizing (and there are several other relevant pm parameters that often make sense in various environments.)

Memory limit in Nextcloud web ui is 1024 MB. I do not know why only in logs I provide to you it reports 0 Byte.

If you're using the community Docker image this memory matter shouldn't be happening. Can you provide your Docker Compose file and, specifically, where you've overriden PHP_MEMORY_LIMIT to be 1024M?

afzl-wtu commented 1 year ago

See #1982, #1977, and others.

Thanks I already had fixed pm issue on my local instance by searching above issues in this repo.

If you're using the community Docker image this memory matter shouldn't be happening. Can you provide your Docker Compose file and, specifically, where you've overriden PHP_MEMORY_LIMIT to be 1024M? Here is my docker compose file:

version: "3"
services:
app:
container_name: nextcloud
network_mode: host
restart: always
image: nextcloud:fpm-alpine
volumes:
#    - ./data:/var/www/html/data
#     - ./config:/var/www/html/config
#      - ./themes:/var/www/html/themes
- ./web:/var/www/html
- '/home/ubuntu/Downloads:/@Downloads'
- '/home/ubuntu/@OneDrive/Rclone/OneDrive:/@OneDrive/Root'
- '/home/ubuntu/@OneDrive/Rclone/Chunker:/@OneDrive/Chunker'
- '/home/ubuntu/@OneDrive/Encryption:/@OneDrive/Encryption'
environment:
#      - SQLITE_DATABASE=***
#      - MYSQL_PASSWORD=*****
#      - MYSQL_DATABASE=*****
#      - MYSQL_USER=*****
#      - NEXTCLOUD_ADMIN_USER=****
#      - NEXTCLOUD_ADMIN_PASSWORD=******
#      - MYSQL_HOST=127.0.0.1
- POSTGRES_DB=*****
- POSTGRES_USER=*****
- POSTGRES_PASSWORD=*****
- POSTGRES_HOST=127.0.0.1:5432
- REDIS_HOST=127.0.0.1
- REDIS_HOST_PORT=6379
#      - REDIS_HOST_PASSWORD
- NEXTCLOUD_INIT_HTACCESS=true
- PHP_MEMORY_LIMIT=1024M
- PHP_UPLOAD_LIMIT=17G

joshtrichards commented 4 months ago

Warning: Failed to set memory limit to 0 bytes (Current memory usage is 2097152 bytes) in Unknown on line 0 The current PHP memory limit is below the recommended value

My guess is you're running occ via sudo or something unconventional for the Docker image. That breaks things because it abandons the environment variables. Use docker compose exec -u33 {bash, xxx, etc} or similar:

https://github.com/nextcloud/docker/?tab=readme-ov-file#using-the-nextcloud-command-line-interface

nextcloud / docker