search

nextcloud / server

☁️ Nextcloud server, a safe home for all your data
https://nextcloud.com
GNU Affero General Public License v3.0
26.9k stars 4.01k forks source link

SFTP Performance slow and Connection buggy #5668

Closed ManOki closed 2 months ago

ManOki commented 7 years ago
### Steps to reproduce 1. Mount a SFTP share via files_external (on another pc in same gigabit network) 2. Upload a big file 3. Download it via WebBrowser or DesktopClient 4. Cancel download after a while ### Expected behaviour fast download speed (about 40mb/s with commandline tool sftp), server stops downloading file from external sftp share, when user cancels file download ### Actual behaviour slow download speed (about 3.5mb/s), server does not stop downloading file from external sftp share, when user canceled file download ### Server configuration **Operating system:** Debian Stretch **Web server:** Apache 2.4.25 **Database:** Mariadb 10.1.23 **PHP version:** Php 7.0.19 **Nextcloud version:** 12.0.0 **Updated from an older Nextcloud/ownCloud or fresh install:** Updated from Owncloud 9 to Nextcloud **Where did you install Nextcloud from:** directly from website **Signing status:**
Signing status No errors have been found.
**List of activated apps:**
App list ``` Enabled: - activity: 2.5.2 - admin_audit: 1.2.0 - audioplayer: 2.0.2 - calendar: 1.5.3 - checksum: 0.3.4 - circles: 0.11.0 - comments: 1.2.0 - contacts: 1.5.3 - dav: 1.3.0 - external: 2.0.3 - federatedfilesharing: 1.2.0 - federation: 1.2.0 - files: 1.7.2 - files_external: 1.3.0 - files_pdfviewer: 1.1.1 - files_retention: 1.1.2 - files_sharing: 1.4.0 - files_texteditor: 2.4.1 - files_versions: 1.5.0 - files_videoplayer: 1.1.0 - gallery: 17.0.0 - groupfolders: 1.1.0 - impersonate: 1.0.1 - keeweb: 0.4.0 - logreader: 2.0.0 - lookup_server_connector: 1.0.0 - notifications: 2.0.0 - oauth2: 1.0.5 - password_policy: 1.2.2 - previewgenerator: 1.0.6 - provisioning_api: 1.2.0 - serverinfo: 1.2.0 - sharebymail: 1.2.0 - spreed: 2.0.1 - survey_client: 1.0.0 - systemtags: 1.2.0 - tasks: 0.9.5 - theming: 1.3.0 - twofactor_backupcodes: 1.1.1 - updatenotification: 1.2.0 - user_external: 0.4 - workflowengine: 1.2.0 Disabled: - bruteforcesettings - encryption - files_trashbin - firstrunwizard - nextcloud_announcements - user_ldap ```
**Nextcloud configuration:**
Config report ``` { "system": { "instanceid": "***REMOVED SENSITIVE VALUE***", "passwordsalt": "***REMOVED SENSITIVE VALUE***", "datadirectory": "***REMOVED SENSITIVE VALUE***", "dbtype": "mysql", "version": "12.0.0.29", "updatechecker": false, "dbname": "owncloud", "dbhost": "localhost", "dbtableprefix": "oc_", "dbuser": "***REMOVED SENSITIVE VALUE***", "dbpassword": "***REMOVED SENSITIVE VALUE***", "installed": true, "forcessl": true, "trusted_domains": [ "***REMOVED SENSITIVE VALUE***", "***REMOVED SENSITIVE VALUE***", "***REMOVED SENSITIVE VALUE***" ], "loglevel": 3, "logtimezone": "UTC", "secret": "***REMOVED SENSITIVE VALUE***", "overwrite.cli.url": "***REMOVED SENSITIVE VALUE***", "trashbin_retention_obligation": "auto", "memcache.local": "\\OC\\Memcache\\Redis", "redis": { "host": "localhost", "port": 6379 }, "memcache.locking": "\\OC\\Memcache\\Redis", "enable_previews": true, "preview_max_filesize_image": -1, "preview_libreoffice_path": "\/usr\/bin\/libreoffice", "preview_office_cl_parameters": " --headless --nologo --nofirststartwizard --invisible --norestore -convert-to pdf -outdir ", "enabledPreviewProviders": [ "OC\\Preview\\PNG", "OC\\Preview\\JPEG", "OC\\Preview\\GIF", "OC\\Preview\\BMP", "OC\\Preview\\XBitmap", "OC\\Preview\\MP3", "OC\\Preview\\TXT", "OC\\Preview\\MarkDown", "OC\\Preview\\Illustrator", "OC\\Preview\\Movie", "OC\\Preview\\MSOffice2003", "OC\\Preview\\MSOffice2007", "OC\\Preview\\MSOfficeDoc", "OC\\Preview\\OpenDocument", "OC\\Preview\\PDF", "OC\\Preview\\Photoshop", "OC\\Preview\\Postscript", "OC\\Preview\\StarOffice", "OC\\Preview\\SVG", "OC\\Preview\\TIFF", "OC\\Preview\\Font" ], "mail_smtpmode": "smtp", "mail_smtpsecure": "tls", "mail_from_address": "***REMOVED SENSITIVE VALUE***", "mail_domain": "openmailbox.org", "mail_smtpauth": 1, "mail_smtphost": "smtp.openmailbox.org", "mail_smtpport": "587", "mail_smtpname": "***REMOVED SENSITIVE VALUE***", "mail_smtppassword": "***REMOVED SENSITIVE VALUE***", "mail_smtpauthtype": "LOGIN", "maintenance": false, "theme": "", "htaccess.RewriteBase": "\/nextcloud", "overwritewebroot": "\/nextcloud", "updater.release.channel": "stable", "auth.bruteforce.protection.enabled": false } } ```
**Are you using external storage, if yes which one:** sftp **Are you using encryption:** no **Are you using an external user-backend, if yes which one:** no This issue was already discussed in the forum, but no issue was raised. https://help.nextcloud.com/t/incredibly-slow-performance-with-sftp/4218/6
ManOki commented 7 years ago

Performance issue seems to be a generic php problem: apps/files_external/3rdparty/icewind/streams and icewind1991/Streams

php's internal buffer size (8192 on default)

otih commented 6 years ago

is there any update available?

kescherCode commented 5 years ago

Able to reproduce this exact behavior, including speeds and the download that doesn't stop when cancelled in browser.

Server configuration

Operating system: Debian Stretch (9.8) Web server: nginx 1.10.3 Database: PostgreSQL 9.6.11 PHP version: 7.3.3 Nextcloud version: 15.0.5 Updated from an older Nextcloud/ownCloud or fresh install: Updated from NC 14 Where did you install Nextcloud from: Nextcloud's website Signing status:

Signing status `No errors have been found.`

List of activated apps:

App list ``` Enabled: - accessibility: 1.1.0 - activity: 2.8.2 - admin_audit: 1.5.0 - apporder: 0.6.0 - audioplayer: 2.6.1 - auto_mail_accounts: 0.1.3 - calendar: 1.6.4 - camerarawpreviews: 0.6.5 - checksum: 0.4.2 - cloud_federation_api: 0.1.0 - comments: 1.5.0 - contacts: 3.0.5 - dav: 1.8.1 - federatedfilesharing: 1.5.0 - federation: 1.5.0 - files: 1.10.0 - files_downloadactivity: 1.4.0 - files_external: 1.6.0 - files_markdown: 2.0.5 - files_mindmap: 0.0.11 - files_pdfviewer: 1.4.0 - files_rightclick: 0.11.1 - files_sharing: 1.7.0 - files_texteditor: 2.7.0 - files_trackdownloads: 1.4.0 - files_trashbin: 1.5.0 - files_versions: 1.8.0 - files_videoplayer: 1.4.0 - firstrunwizard: 2.4.0 - flowupload: 0.0.9 - gallery: 18.2.0 - group_everyone: 0.1.1 - logreader: 2.0.0 - lookup_server_connector: 1.3.0 - mail: 0.11.0 - metadata: 0.9.0 - nextcloud_announcements: 1.4.0 - notes: 2.5.1 - notifications: 2.3.0 - oauth2: 1.3.0 - password_policy: 1.5.0 - previewgenerator: 2.1.0 - provisioning_api: 1.5.0 - quota_warning: 1.4.0 - radio: 0.6.4 - serverinfo: 1.5.0 - sharebymail: 1.5.0 - sharerenamer: 2.5.0 - spreed: 5.0.2 - support: 1.0.0 - survey_client: 1.3.0 - systemtags: 1.5.0 - theming: 1.6.0 - twofactor_backupcodes: 1.4.1 - twofactor_totp: 2.1.2 - updatenotification: 1.5.0 - user_backend_sql_raw: 1.0.6 - w2g2: 2.2.4 - workflowengine: 1.5.0 Disabled: - encryption - user_ldap ```

Nextcloud configuration:

Config report ``` { "system": { "passwordsalt": "***REMOVED SENSITIVE VALUE***", "secret": "***REMOVED SENSITIVE VALUE***", "trusted_domains": [ ***REMOVED SENSITIVE VALUE***, ***REMOVED SENSITIVE VALUE*** ], "datadirectory": "***REMOVED SENSITIVE VALUE***", "dbtype": "pgsql", "version": "15.0.5.3", "dbname": "***REMOVED SENSITIVE VALUE***", "dbhost": "***REMOVED SENSITIVE VALUE***", "dbport": "", "dbtableprefix": "oc_", "dbuser": "***REMOVED SENSITIVE VALUE***", "dbpassword": "***REMOVED SENSITIVE VALUE***", "installed": true, "instanceid": "***REMOVED SENSITIVE VALUE***", "overwrite.cli.url": "***REMOVED SENSITIVE VALUE***", "log_type": "syslog", "syslog_tag": "nextcloud", "logtimezone": "Europe\/Vienna", "memcache.local": "\\OC\\Memcache\\APCu", "filelocking.enabled": true, "memcache.locking": "\\OC\\Memcache\\Redis", "redis": { "host": "***REMOVED SENSITIVE VALUE***", "port": 0, "timeout": 0 }, "user_backend_sql_raw": { "db_name": "***REMOVED SENSITIVE VALUE***", "db_user": "***REMOVED SENSITIVE VALUE***", "db_password": "***REMOVED SENSITIVE VALUE***", "queries": { "get_password_hash_for_user": "SELECT password_hash FROM users_fqda WHERE fqda = :username", "user_exists": "SELECT EXISTS(SELECT 1 FROM users_fqda WHERE fqda = :username)", "get_users": "SELECT fqda FROM users_fqda WHERE (fqda ILIKE :search) OR (display_name ILIKE :search)", "set_password_hash_for_user": "UPDATE users SET password_hash = :new_password_hash WHERE local = split_part(:username, '@', 1) AND domain = split_part(:username, '@', 2)", "delete_user": "DELETE FROM users WHERE local = split_part(:username, '@', 1) AND domain = split_part(:username, '@', 2)", "get_display_name": "SELECT display_name FROM users WHERE local = split_part(:username, '@', 1) AND domain = split_part(:username, '@', 2)", "set_display_name": "UPDATE users SET display_name = :new_display_name WHERE local = split_part(:username, '@', 1) AND domain = split_part(:username, '@', 2)", "count_users": "SELECT COUNT (*) FROM users", "create_user": "INSERT INTO users (local, domain, password_hash) VALUES (split_part(:username, '@', 1), split_part(:username, '@', 2), :password_hash)" }, "hash_algorithm_for_new_passwords": "sha512" }, "simpleSignUpLink.shown": false, "versions_retention_obligation": "auto, 30", "mail_smtpmode": "sendmail", "mail_from_address": "***REMOVED SENSITIVE VALUE***", "mail_domain": "***REMOVED SENSITIVE VALUE***", "updater.release.channel": "production", "preview_max_scale_factor": 1, "preview_max_x": 1024, "preview_max_y": 1024, "preview_max_filesize_image": 70, "enabledPreviewProviders": [ "OC\\Preview\\PNG", "OC\\Preview\\JPEG", "OC\\Preview\\GIF", "OC\\Preview\\HEIC", "OC\\Preview\\BMP", "OC\\Preview\\XBitmap", "OC\\Preview\\MP3", "OC\\Preview\\TXT", "OC\\Preview\\MarkDown" ], "maintenance": false } } ```

Are you using external storage, if yes which one: sftp

Are you using encryption: no

Are you using an external user-backend, if yes which one: Raw SQL User Backend. Shouldn't matter for this issue. It's merely used for having a Nextcloud account for each user of my mail server, which does not interfere with SFTP speeds at all.

UltimateByte commented 5 years ago

Experiencing this as well when mouting a remote SFTP... There seem to be no usable secure remote conneciton at the moment, this issue's priority should be increased. FTP won't show any files, and FTPS isn't able to ignore TLS certificates. Last resort for me as a workaround will be VPN + SMB, or VPN + NFS, or maybe SSHFS but then it'll be a pain for permissions...

I'm connecting from my dedicated web server to my home NAS (and I've got gigabit fiber at home). I am observing around 3mbit/s download from Nextcloud on mounted SFTP files. If running multiple downloads at the same time, then I'm getting multiple times 3mbit/s, so it's a "per instance" limit.

Server configuration Operating system: CentOS 7.6.1810 (running Plesk 17.8) Web server: nginx Apache/2.4.6 Database: Distrib 10.1.40-MariaDB PHP version: 7.3.5 Nextcloud version: 16.0.1 Where did you install Nextcloud from: Installed via Plesk Extension

Server configuration detail

Operating system: Linux 3.10.0-957.5.1.el7.x86_64 #1 SMP Fri Feb 1 14:54:57 UTC 2019 x86_64

Webserver: Apache (fpm-fcgi)

Database: mysql 10.1.40

PHP version: 7.3.5

Modules loaded: Core, date, libxml, openssl, pcre, zlib, bz2, calendar, ctype, hash, filter, ftp, gettext, gmp, SPL, iconv, Reflection, session, standard, SimpleXML, sockets, mbstring, tokenizer, xml, cgi-fcgi, mysqlnd, bcmath, curl, dba, dom, enchant, fileinfo, gd, imagick, imap, intl, json, ldap, exif, mysqli, odbc, PDO, pdo_mysql, PDO_ODBC, pdo_pgsql, pdo_sqlite, pgsql, Phar, posix, pspell, redis, soap, sodium, sqlite3, sysvmsg, sysvsem, sysvshm, tidy, xmlreader, xmlrpc, xmlwriter, xsl, zip, ionCube Loader, Zend OPcache

Nextcloud version: 16.0.1 - 16.0.1.1

Updated from an older Nextcloud/ownCloud or fresh install:

Where did you install Nextcloud from: unknown

Signing status Array ( [files_rightclick] => Array ( [EXTRA_FILE] => Array ( [README.md] => Array ( [expected] => [current] => cf73849388838de5037624e53303618579b164ec69fd55834ce4c5332ae33f778839e36298cbcc8ede90620927e076dbb8883479754e4c09b0491b15bf7653f3 ) ) ) )
List of activated apps ``` Enabled: - accessibility: 1.2.0 - activity: 2.9.1 - admin_audit: 1.6.0 - bookmarks: 1.0.5 - calendar: 1.7.0 - carnet: 0.15.6 - cloud_federation_api: 0.2.0 - comments: 1.6.0 - contacts: 3.1.1 - dav: 1.9.2 - federatedfilesharing: 1.6.0 - federation: 1.6.0 - files: 1.11.0 - files_external: 1.7.0 - files_pdfviewer: 1.5.0 - files_sharing: 1.8.0 - files_texteditor: 2.8.0 - files_trashbin: 1.6.0 - files_versions: 1.9.0 - files_videoplayer: 1.5.0 - firstrunwizard: 2.5.0 - gallery: 18.3.0 - logreader: 2.1.0 - lookup_server_connector: 1.4.0 - metadata: 0.9.0 - music: 0.9.5 - nextcloud_announcements: 1.5.0 - notifications: 2.4.1 - oauth2: 1.4.2 - passman: 2.3.1 - password_policy: 1.6.0 - phonetrack: 0.5.1 - privacy: 1.0.0 - provisioning_api: 1.6.0 - recommendations: 0.4.0 - serverinfo: 1.6.0 - sharebymail: 1.6.0 - support: 1.0.0 - survey_client: 1.4.0 - systemtags: 1.6.0 - theming: 1.7.0 - twofactor_backupcodes: 1.5.0 - updatenotification: 1.6.0 - viewer: 1.0.0 - workflowengine: 1.6.0 Disabled: - encryption - files_rightclick - user_ldap ```
Configuration (config/config.php) ``` { "passwordsalt": "***REMOVED SENSITIVE VALUE***", "secret": "***REMOVED SENSITIVE VALUE***", "trusted_domains": [ "localhost", "cloud.lrob.fr" ], "datadirectory": "***REMOVED SENSITIVE VALUE***", "dbtype": "mysql", "version": "16.0.1.1", "overwrite.cli.url": "http:\/\/localhost", "dbname": "***REMOVED SENSITIVE VALUE***", "dbhost": "***REMOVED SENSITIVE VALUE***", "dbport": "3306", "dbtableprefix": "oc_", "dbuser": "***REMOVED SENSITIVE VALUE***", "dbpassword": "***REMOVED SENSITIVE VALUE***", "installed": true, "instanceid": "***REMOVED SENSITIVE VALUE***", "mail_from_address": "***REMOVED SENSITIVE VALUE***", "mail_smtpmode": "smtp", "mail_sendmailmode": "smtp", "mail_domain": "***REMOVED SENSITIVE VALUE***", "mail_smtpsecure": "ssl", "mail_smtpauthtype": "LOGIN", "mail_smtpauth": 1, "mail_smtpport": "465", "mail_smtphost": "***REMOVED SENSITIVE VALUE***", "mail_smtpname": "***REMOVED SENSITIVE VALUE***", "mail_smtppassword": "***REMOVED SENSITIVE VALUE***", "has_rebuilt_cache": true, "maintenance": false, "theme": "", "loglevel": 2 } ```

External storages: yes

External storage configuration ``` +----------+-------------+---------+---------------------+-------------------------------------------------------------------------------------------------------------------------+-----------------+------------------+-------------------+-------+ | Mount ID | Mount Point | Storage | Authentication Type | Configuration | Options | Applicable Users | Applicable Groups | Type | +----------+-------------+---------+---------------------+-------------------------------------------------------------------------------------------------------------------------+-----------------+------------------+-------------------+-------+ | 2 | /Multimedia | SFTP | RSA public key | host: "nas.lrob.fr:22", root: "\/sharedfolders\/Media", user: "UltimateByte", public_key: "***", private_key: "***" | readonly: false | admin | admin | Admin | | 3 | /Storage | SFTP | RSA public key | host: "nas.lrob.fr:22", root: "\/sharedfolders\/Storage", user: "UltimateByte", public_key: "***", private_key: "***" | readonly: false | admin | admin | Admin | +----------+-------------+---------+---------------------+-------------------------------------------------------------------------------------------------------------------------+-----------------+------------------+-------------------+-------+ ```

Encryption: no

User-backends:

Browser: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:67.0) Gecko/20100101 Firefox/67.0

skjnldsv commented 4 years ago

As the version of the software you've reported this for has reached end of life, I will close this ticket. If this is still happening after an upgrade to the latest version, feel free to reopen

kescherCode commented 4 years ago

This issue is definitely still happening, and I am still resorting to mounting filesystems via sshfs and then mounting them as local filesystem (as admin) in order to have file listings not be slowed down to a crawl.

skjnldsv commented 4 years ago

Thank you @kescherCode :)

Ludo444 commented 4 years ago

definitely can confirm this is still a problem

SimplyCorbett commented 3 years ago

Issue still occurs in 20.0.2. SFTP is not usable for me because of it.

kescherCode commented 3 years ago

@SimplyCorbett I've given up hope on this issue ever being resolved, and instead created on-demand sshfs mounts for "local" external storage for each user.

SimplyCorbett commented 3 years ago

@SimplyCorbett I've given up hope on this issue ever being resolved, and instead created on-demand sshfs mounts for "local" external storage for each user.

sshfs is also my current solution.

szaimen commented 3 years ago

Lets track this in https://github.com/nextcloud/viewer/issues/894 (is probably the same underlaying issue)

kescherCode commented 3 years ago

No. Entirely separate issue. Not the same at all.

szaimen commented 3 years ago

Yes, it is the same since the filesystem doesn't stop the loading from the storage when the connection is closed.

kescherCode commented 3 years ago

@szaimen Still not even close to being the same issue. The performance of downloading a single file via an SFTP external share is capped at 128 kbps for me. Without any previews being generated or other nonsense. If I do the same using sshfs and add it as a local external share, the performance is in the Gbps range as it should.

kescherCode commented 3 years ago

That is something that is tracked in this issue.

szaimen commented 3 years ago

Ah okay, Sry must have misread the issue.

joshtrichards commented 1 year ago

I'm unable to reproduce the performance issue with NC26 today. A fair number of things have changed, including in two third-party modules that would relate to this.

If you're no longer experiencing this issue, please comment so we know.

And if you're still experiencing the performance issue can you provide:

joshtrichards commented 2 months ago

Closing due to no further updates/reports received >1 year.