Open GAS85 opened 2 years ago
Also I can't open e.g. Videos from the external Storages via UI any more...
/tmp
Video files are there:
ls -lah /tmp | grep "www-data"
-rw------- 1 www-data www-data 800M Dec 16 08:52 oc_tmp_uLpKpT-.avi
-rw------- 1 www-data www-data 50M Dec 16 08:53 php0NKva5
-rw------- 1 www-data www-data 100M Dec 16 08:53 phpYRlgBW
-rw------- 1 www-data www-data 152M Dec 16 08:53 phptIAg9A
-rw------- 1 www-data www-data 200M Dec 16 08:53 phpwbuY14
Add a data point.
I upgraded from 21.0.4 to 22.2.3.
The problem didn't happen in 21.0.4.
With 22.2.3 it's a new behavior that every time when user try to download a file from the webdav external storage, the file needs to be downloadeded to /tmp from the server side, then the web download could start.
If the server download could not finish within 30 seconds, the error happens.
A related, but might be different problem, the webdav external storage speed is degraded in 22 comparing to 21.0.4.
When do a full scan (occ scan
This problem is still in 22.2.5. When the user access a file and the WebDAV client is involved, Nextcloud would copy the whole file from remote to local /tmp folder. If the file is large, the delay could be very long and trigger curl time out. This applies to the external webdav storage, and to the federation sharing.
Problem still occurs in 23.0.2 External samba storage, folders with files larger than 512mb doesn't sync on entering them and shows as empty. Command occ fieles:scan doesn't sync them as well.
I'm pretty sure I've tracked down the issue. In https://github.com/nextcloud/server/pull/26266 the way Guzzle HTTP clients are constructed was changed from the default, which chose the handler based on the environment, to always use the curl handler. Just using the curl handler doesn't seem to support the "stream"
argument like the default handler stack does - you need to somehow directly pass in the sink
option, otherwise the response is written to a temporary file.
The following patch, on top of 23.0.3.2 works around this issue, although it also reverts the security improvements from said PR:
diff -ruN ./1/lib/private/Http/Client/ClientService.php ./0/lib/private/Http/Client/ClientService.php
--- ./1/lib/private/Http/Client/ClientService.php 2022-03-21 13:02:41.000000000 +0000
+++ ./0/lib/private/Http/Client/ClientService.php 2022-06-02 19:31:07.114812663 +0000
@@ -68,11 +68,7 @@
* @return Client
*/
public function newClient(): IClient {
- $handler = new CurlHandler();
- $stack = HandlerStack::create($handler);
- $stack->push($this->dnsPinMiddleware->addDnsPinning());
-
- $client = new GuzzleClient(['handler' => $stack]);
+ $client = new GuzzleClient();
return new Client(
$this->config,
@LukasReschke @rullzer could you have a look?
Good finding :+1:
php-curl is a hard requirement for guzzle.
But you are right we are always using CurlHandler
. The default implementation does a bit more: https://github.com/nextcloud/3rdparty/blob/ce3a3cedfd36dd914ec103a62b8a0f47e950a0e1/guzzlehttp/guzzle/src/Utils.php#L77-L106
Index: lib/private/Http/Client/ClientService.php
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/lib/private/Http/Client/ClientService.php b/lib/private/Http/Client/ClientService.php
--- a/lib/private/Http/Client/ClientService.php (revision e5309201f7479be1464fe10c71e4a039dddd26b9)
+++ b/lib/private/Http/Client/ClientService.php (date 1654617117944)
@@ -63,8 +63,7 @@
* @return Client
*/
public function newClient(): IClient {
- $handler = new CurlHandler();
- $stack = HandlerStack::create($handler);
+ $stack = HandlerStack::create();
$stack->push($this->dnsPinMiddleware->addDnsPinning());
$client = new GuzzleClient(['handler' => $stack]);
Does this work too?
@kesselb Confirmed to work! :+1:
Mind to sent a pull request?
@kesselb This fixes the issue for Nextcloud 24.0.2 aswell :+1:
I was previously experiencing timeouts when syncing larger files such as videos from/to an external WebDAV storage.
If no one wants to create the PR, shall I do it?
done
Not fixed in the stable25 Branch: https://github.com/nextcloud/server/pull/34749
Hey, I do the test and it is not really fixed. There are to parts:
1) Requests from external storage were not streamed anymore was fixed by #33087. 2) The operation time issue was addressed by #38292.
Please update to Nextcloud 27, set a higher timeout for dav storage, and create a new issue if the issue is still there.
Thanks! NC 27.0.1 seems not mature enough for me now to make Prod update. Nevertheless, when I look on a https://github.com/nextcloud/server/pull/33087 it was successfully backported to NC23 and NC24, but seems failed on NC25 - https://github.com/nextcloud/server/pull/34749 So from my point of view bug is still valid here and should not be closed till this time. Not sure if this fits to your process.
Oh, well spotted.
I don't understand the partial revert :shrug:
Feel free to do another attempt on this with the same fix against master. Last time I tried the tests have been failing and I had no idea how to resolve them.
Hi, we still have the issue on 29.0.4. How can we help ?
At least it is still an issue in 28.0.9.
How to use GitHub
Steps to reproduce
Seems it is related to #30262 and #29969 with #26071
Expected behaviour
Do not download files (as per #30262) when folder simply opened, use cache instead. Do not download files when properties only needed (last modification date, or size)
Actual behaviour
External Storage could not be synchronized, even via Desktop Client it simply does not open and not synced any more. All files downloaded to
/tmp
or RAM and causing timeout.Server configuration
Operating system: Ubuntu 20.04
Web server: Apache/2.4.51
Database: 10.3.32-MariaDB
PHP version: 7.4.26
Nextcloud version: 22.2.3.0
Updated from an older Nextcloud/ownCloud or fresh install: updated
Where did you install Nextcloud from: Official
Signing status:
Signing status
``` No errors have been found. ```List of activated apps:
App list
``` Enabled: - accessibility: 1.8.0 - activity: 2.15.0 - admin_audit: 1.12.0 - audioplayer: 3.2.2 - bruteforcesettings: 2.2.0 - calendar: 3.0.0 - camerarawpreviews: 0.7.15 - checksum: 1.1.2 - cloud_federation_api: 1.5.0 - comments: 1.12.0 - contacts: 4.0.6 - contactsinteraction: 1.3.0 - cospend: 1.4.3 - dashboard: 7.2.0 - data_request: 1.9.0 - dav: 1.19.0 - deck: 1.5.5 - drawio: 1.0.1 - external: 3.9.0 - extract: 1.3.2 - facerecognition: 0.8.5 - federatedfilesharing: 1.12.0 - files: 1.17.0 - files_automatedtagging: 1.12.0 - files_downloadactivity: 1.11.1 - files_external: 1.13.0 - files_mindmap: 0.0.25 - files_pdfviewer: 2.3.1 - files_retention: 1.11.1 - files_rightclick: 1.1.0 - files_sharing: 1.14.0 - files_trashbin: 1.12.0 - files_versions: 1.15.0 - files_videoplayer: 1.11.0 - firstrunwizard: 2.11.0 - forms: 2.4.0 - gpxedit: 0.0.14 - gpxpod: 4.3.0 - integration_github: 1.0.2 - integration_gitlab: 1.0.3 - keeweb: 0.6.8 - logreader: 2.7.0 - lookup_server_connector: 1.10.0 - mail: 1.11.0 - maps: 0.1.9 - nextcloud_announcements: 1.11.0 - notes: 4.2.0 - notifications: 2.10.1 - oauth2: 1.10.0 - ocdownloader: 1.7.12 - password_policy: 1.12.0 - phonetrack: 0.6.9 - photos: 1.4.0 - polls: 3.3.0 - previewgenerator: 3.3.0 - privacy: 1.6.0 - provisioning_api: 1.12.0 - recommendations: 1.1.0 - serverinfo: 1.12.0 - settings: 1.4.0 - sharebymail: 1.12.0 - side_menu: 2.0.1 - spreed: 12.1.2 - support: 1.5.0 - survey_client: 1.10.0 - systemtags: 1.12.0 - text: 3.3.0 - theming: 1.13.0 - twofactor_backupcodes: 1.11.0 - twofactor_totp: 6.1.0 - unsplash: 1.2.4 - updatenotification: 1.12.0 - user_status: 1.2.0 - video_converter: 1.0.3 - viewer: 1.6.0 - weather_status: 1.2.0 - workflowengine: 2.4.0 Disabled: - circles - encryption - federation - files_accesscontrol - flowupload - gpxmotion - impersonate - integration_jira - radio - recognize - user_ldap - weather ```Nextcloud configuration:
Config report
``` { "system": { "instanceid": "***REMOVED SENSITIVE VALUE***", "passwordsalt": "***REMOVED SENSITIVE VALUE***", "secret": "***REMOVED SENSITIVE VALUE***", "trusted_domains": [ "***REMOVED SENSITIVE VALUE***", "***REMOVED SENSITIVE VALUE***", "***REMOVED SENSITIVE VALUE***" ], "datadirectory": "***REMOVED SENSITIVE VALUE***", "overwrite.cli.url": "***REMOVED SENSITIVE VALUE***", "dbtype": "mysql", "version": "22.2.3.0", "dbname": "***REMOVED SENSITIVE VALUE***", "dbhost": "***REMOVED SENSITIVE VALUE***", "dbport": "", "dbtableprefix": "oc_", "mysql.utf8mb4": true, "dbuser": "***REMOVED SENSITIVE VALUE***", "dbpassword": "***REMOVED SENSITIVE VALUE***", "installed": true, "filesystem_check_changes": 0, "filelocking.enabled": true, "memcache.local": "\\OC\\Memcache\\APCu", "memcache.locking": "\\OC\\Memcache\\Redis", "memcache.distributed": "\\OC\\Memcache\\Redis", "redis": { "host": "***REMOVED SENSITIVE VALUE***", "port": 0, "timeout": 1.5 }, "default_phone_region": "DE", "mail_smtpmode": "smtp", "mail_smtpauthtype": "LOGIN", "mail_from_address": "***REMOVED SENSITIVE VALUE***", "mail_domain": "***REMOVED SENSITIVE VALUE***", "maintenance": false, "theme": "", "logfile": "\/var\/nextcloud\/data\/nextcloud.log", "loglevel": 1, "trashbin_retention_obligation": "14, auto", "versions_retention_obligation": "14, auto", "data-fingerprint": "7be51475e95ea13b254cd880319d98ec", "enable_previews": true, "enabledPreviewProviders": [ "OC\\Preview\\PNG", "OC\\Preview\\JPEG", "OC\\Preview\\GIF", "OC\\Preview\\BMP", "OC\\Preview\\XBitmap", "OC\\Preview\\Movie", "OC\\Preview\\PDF", "OC\\Preview\\MP3", "OC\\Preview\\TXT", "OC\\Preview\\MarkDown" ], "preview_max_x": 1920, "preview_max_y": 1080, "jpeg_quality": 90, "auth.bruteforce.protection.enabled": true, "simpleSignUpLink.shown": false, "mail_smtpsecure": "tls", "mail_smtpauth": 1, "mail_smtphost": "***REMOVED SENSITIVE VALUE***", "mail_smtpport": "587", "mail_smtpname": "***REMOVED SENSITIVE VALUE***", "mail_smtppassword": "***REMOVED SENSITIVE VALUE***", "twofactor_enforced": true, "twofactor_enforced_groups": [ "admin" ], "twofactor_enforced_excluded_groups": [], "has_rebuilt_cache": true, "updater.release.channel": "stable", "mail_sendmailmode": "smtp" } } ```Are you using external storage, if yes which one: local/webdav
Are you using encryption: no
Are you using an external user-backend, if yes which one: no
Logs
Nextcloud log (data/nextcloud.log)
Nextcloud log
``` {"reqId":"3bdICeP14G6vPYqfCKqQ","level":3,"time":"2021-12-15T09:30:33+00:00","remoteAddr":"","user":"--","app":"files","method":"","url":"--","message":"cURL error 28: Operation timed out after 30000 milliseconds with 379564427 out of 838860800 bytes received (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for https://magentacloud.de/remote.php/webdav/System/NewTea.avi","userAgent":"--","version":"22.2.3.0","exception":{"Exception":"GuzzleHttp\\Exception\\ConnectException","Message":"cURL error 28: Operation timed out after 30000 milliseconds with 379564427 out of 838860800 bytes received (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for https://magentacloud.de/remote.php/webdav/System/NewTea.avi","Code":0,"Trace":[{"file":"/var/www/nextcloud/3rdparty/guzzlehttp/guzzle/src/Handler/CurlFactory.php","line":158,"function":"createRejection","class":"GuzzleHttp\\Handler\\CurlFactory","type":"::"},{"file":"/var/www/nextcloud/3rdparty/guzzlehttp/guzzle/src/Handler/CurlFactory.php","line":110,"function":"finishError","class":"GuzzleHttp\\Handler\\CurlFactory","type":"::"},{"file":"/var/www/nextcloud/3rdparty/guzzlehttp/guzzle/src/Handler/CurlHandler.php","line":47,"function":"finish","class":"GuzzleHttp\\Handler\\CurlFactory","type":"::"},{"file":"/var/www/nextcloud/lib/private/Http/Client/DnsPinMiddleware.php","line":146,"function":"__invoke","class":"GuzzleHttp\\Handler\\CurlHandler","type":"->"},{"file":"/var/www/nextcloud/3rdparty/guzzlehttp/guzzle/src/PrepareBodyMiddleware.php","line":35,"function":"OC\\Http\\Client\\{closure}","class":"OC\\Http\\Client\\DnsPinMiddleware","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/nextcloud/3rdparty/guzzlehttp/guzzle/src/Middleware.php","line":31,"function":"__invoke","class":"GuzzleHttp\\PrepareBodyMiddleware","type":"->"},{"file":"/var/www/nextcloud/3rdparty/guzzlehttp/guzzle/src/RedirectMiddleware.php","line":71,"function":"GuzzleHttp\\{closure}","class":"GuzzleHttp\\Middleware","type":"::","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/nextcloud/3rdparty/guzzlehttp/guzzle/src/Middleware.php","line":63,"function":"__invoke","class":"GuzzleHttp\\RedirectMiddleware","type":"->"},{"file":"/var/www/nextcloud/3rdparty/guzzlehttp/guzzle/src/HandlerStack.php","line":75,"function":"GuzzleHttp\\{closure}","class":"GuzzleHttp\\Middleware","type":"::","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/nextcloud/3rdparty/guzzlehttp/guzzle/src/Client.php","line":331,"function":"__invoke","class":"GuzzleHttp\\HandlerStack","type":"->"},{"file":"/var/www/nextcloud/3rdparty/guzzlehttp/guzzle/src/Client.php","line":168,"function":"transfer","class":"GuzzleHttp\\Client","type":"->"},{"file":"/var/www/nextcloud/3rdparty/guzzlehttp/guzzle/src/Client.php","line":187,"function":"requestAsync","class":"GuzzleHttp\\Client","type":"->"},{"file":"/var/www/nextcloud/lib/private/Http/Client/Client.php","line":223,"function":"request","class":"GuzzleHttp\\Client","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Storage/DAV.php","line":356,"function":"get","class":"OC\\Http\\Client\\Client","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Storage/LocalTempFileTrait.php","line":65,"function":"fopen","class":"OC\\Files\\Storage\\DAV","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Storage/LocalTempFileTrait.php","line":48,"function":"toTmpFile","class":"OC\\Files\\Storage\\Common","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Storage/Common.php","line":269,"function":"getCachedFile","class":"OC\\Files\\Storage\\Common","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Storage/Wrapper/Wrapper.php","line":367,"function":"getLocalFile","class":"OC\\Files\\Storage\\Common","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Storage/Wrapper/Availability.php","line":374,"function":"getLocalFile","class":"OC\\Files\\Storage\\Wrapper\\Wrapper","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Storage/Wrapper/Wrapper.php","line":367,"function":"getLocalFile","class":"OC\\Files\\Storage\\Wrapper\\Availability","type":"->"},{"file":"/var/www/nextcloud/apps/workflowengine/lib/Check/FileMimeType.php","line":131,"function":"getLocalFile","class":"OC\\Files\\Storage\\Wrapper\\Wrapper","type":"->"},{"file":"/var/www/nextcloud/apps/workflowengine/lib/Check/FileMimeType.php","line":109,"function":"getActualValue","class":"OCA\\WorkflowEngine\\Check\\FileMimeType","type":"->"},{"file":"/var/www/nextcloud/apps/workflowengine/lib/Service/RuleMatcher.php","line":245,"function":"executeCheck","class":"OCA\\WorkflowEngine\\Check\\FileMimeType","type":"->"},{"file":"/var/www/nextcloud/apps/workflowengine/lib/Service/RuleMatcher.php","line":180,"function":"check","class":"OCA\\WorkflowEngine\\Service\\RuleMatcher","type":"->"},{"file":"/var/www/nextcloud/apps/workflowengine/lib/Service/RuleMatcher.php","line":128,"function":"getMatchingOperations","class":"OCA\\WorkflowEngine\\Service\\RuleMatcher","type":"->"},{"file":"/var/www/nextcloud/apps/files_automatedtagging/lib/Operation.php","line":95,"function":"getFlows","class":"OCA\\WorkflowEngine\\Service\\RuleMatcher","type":"->"},{"file":"/var/www/nextcloud/apps/files_automatedtagging/lib/Listener/CacheListener.php","line":43,"function":"checkOperations","class":"OCA\\FilesAutomatedTagging\\Operation","type":"->"},{"file":"/var/www/nextcloud/lib/private/EventDispatcher/ServiceEventListener.php","line":75,"function":"handle","class":"OCA\\FilesAutomatedTagging\\Listener\\CacheListener","type":"->"},{"file":"/var/www/nextcloud/3rdparty/symfony/event-dispatcher/EventDispatcher.php","line":251,"function":"__invoke","class":"OC\\EventDispatcher\\ServiceEventListener","type":"->"},{"file":"/var/www/nextcloud/3rdparty/symfony/event-dispatcher/EventDispatcher.php","line":73,"function":"callListeners","class":"Symfony\\Component\\EventDispatcher\\EventDispatcher","type":"->"},{"file":"/var/www/nextcloud/lib/private/EventDispatcher/EventDispatcher.php","line":88,"function":"dispatch","class":"Symfony\\Component\\EventDispatcher\\EventDispatcher","type":"->"},{"file":"/var/www/nextcloud/lib/private/EventDispatcher/EventDispatcher.php","line":100,"function":"dispatch","class":"OC\\EventDispatcher\\EventDispatcher","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Cache/Cache.php","line":324,"function":"dispatchTyped","class":"OC\\EventDispatcher\\EventDispatcher","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Cache/Scanner.php","line":294,"function":"insert","class":"OC\\Files\\Cache\\Cache","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Cache/Scanner.php","line":224,"function":"addToCache","class":"OC\\Files\\Cache\\Scanner","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Cache/Scanner.php","line":427,"function":"scanFile","class":"OC\\Files\\Cache\\Scanner","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Cache/Scanner.php","line":388,"function":"handleChildren","class":"OC\\Files\\Cache\\Scanner","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Cache/Scanner.php","line":340,"function":"scanChildren","class":"OC\\Files\\Cache\\Scanner","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Cache/Scanner.php","line":510,"function":"scan","class":"OC\\Files\\Cache\\Scanner","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Cache/Scanner.php","line":521,"function":"OC\\Files\\Cache\\{closure}","class":"OC\\Files\\Cache\\Scanner","type":"->","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/nextcloud/lib/private/Files/Cache/Scanner.php","line":509,"function":"runBackgroundScanJob","class":"OC\\Files\\Cache\\Scanner","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Utils/Scanner.php","line":185,"function":"backgroundScan","class":"OC\\Files\\Cache\\Scanner","type":"->"},{"file":"/var/www/nextcloud/apps/files/lib/BackgroundJob/ScanFiles.php","line":89,"function":"backgroundScan","class":"OC\\Files\\Utils\\Scanner","type":"->"},{"file":"/var/www/nextcloud/apps/files/lib/BackgroundJob/ScanFiles.php","line":125,"function":"runScanner","class":"OCA\\Files\\BackgroundJob\\ScanFiles","type":"->"},{"file":"/var/www/nextcloud/lib/private/BackgroundJob/Job.php","line":51,"function":"run","class":"OCA\\Files\\BackgroundJob\\ScanFiles","type":"->"},{"file":"/var/www/nextcloud/lib/private/BackgroundJob/TimedJob.php","line":58,"function":"execute","class":"OC\\BackgroundJob\\Job","type":"->"},{"file":"/var/www/nextcloud/cron.php","line":127,"function":"execute","class":"OC\\BackgroundJob\\TimedJob","type":"->"}],"File":"/var/www/nextcloud/3rdparty/guzzlehttp/guzzle/src/Handler/CurlFactory.php","Line":210,"CustomMessage":"--"},"id":"61b9f1e0b44fa"} ```Nextcloud Desktop Client log
Nextcloud Client log
``` ||MagentaCloud|2|2|1638616226|_invalid_|0|10006743xxxxxxxxxxx|4||0|0|0|| #=#=#=# Syncrun finished 2021-12-15T13:59:38Z (last step: 157 msec, total: 2134 msec) #=#=#=# Syncrun started 2021-12-15T13:59:40Z #=#=#=#=# Propagation starts 2021-12-15T14:00:24Z (last step: 43733 msec, total: 43733 msec) #=#=#=# Syncrun finished 2021-12-15T14:00:24Z (last step: 24 msec, total: 43757 msec) #=#=#=# Syncrun started 2021-12-15T14:01:03Z #=#=#=#=# Propagation starts 2021-12-15T14:01:03Z (last step: 371 msec, total: 371 msec) ||MagentaCloud|8|2|1638616226|61ab4ca2b0b9c|0|10006743xxxxxxxxxxx|4||0|0|0|| #=#=#=# Syncrun finished 2021-12-15T14:01:03Z (last step: 28 msec, total: 399 msec) ```