nextcloud / server

☁️ Nextcloud server, a safe home for all your data
https://nextcloud.com
GNU Affero General Public License v3.0
26.34k stars 3.97k forks source link

Files on SMB-share corrupted while up- and downloading #26457

Closed ShinjiLE closed 3 years ago

ShinjiLE commented 3 years ago

Steps to reproduce

Using SMB Share

Expected behaviour

Files uploaded and downloaded without corruption

Actual behaviour

Datablocks are truncated to 8192 Bytes

Server configuration detail

Operating system: Linux 5.11.6-1-default #1 SMP Thu Mar 11 16:11:36 UTC 2021 (7358b30) x86_64

Webserver: Apache (fpm-fcgi)

Database: mysql 10.5.9

PHP version:

7.4.16 Modules loaded: Core, date, libxml, pcre, filter, hash, Reflection, SPL, session, SimpleXML, standard, xml, cgi-fcgi, mysqlnd, apcu, bcmath, bz2, ctype, curl, dom, enchant, mbstring, fileinfo, ftp, gd, gettext, gmp, ice, iconv, imagick, intl, json, lzf, exif, memcached, mysqli, openssl, pcntl, PDO, pdo_mysql, pdo_sqlite, zlib, posix, readline, redis, smbclient, sockets, sqlite3, tidy, tokenizer, uuid, xmlreader, xmlwriter, zip, Phar, libsmbclient, Zend OPcache

Nextcloud version: 21.0.1 RC1 - 21.0.1.0

Updated from an older Nextcloud/ownCloud or fresh install:

Where did you install Nextcloud from: unknown

Signing status Array ( )
List of activated apps ``` Enabled: - activity: 2.14.3 - admin_audit: 1.11.0 - announcementcenter: 5.0.0 - bookmarks: 4.1.0 - calendar: 2.2.0 - checksum: 1.1.2 - cloud_federation_api: 1.4.0 - comments: 1.11.0 - contacts: 3.5.1 - contactsinteraction: 1.2.0 - dashboard: 7.1.0 - dav: 1.17.1 - deck: 1.3.2 - duplicatefinder: 0.0.6 - event_update_notification: 1.2.0 - extract: 1.3.1 - federatedfilesharing: 1.11.0 - federation: 1.11.0 - files: 1.16.0 - files_external: 1.12.0 - files_fulltextsearch: 21.0.0 - files_fulltextsearch_tesseract: 20.0.1 - files_markdown: 2.3.3 - files_pdfviewer: 2.1.0 - files_rightclick: 1.0.0 - files_sharing: 1.13.1 - files_texteditor: 2.14.0 - files_trashbin: 1.11.0 - files_versions: 1.14.0 - files_videoplayer: 1.10.0 - firstrunwizard: 2.10.0 - fulltextsearch: 21.0.0 - fulltextsearch_elasticsearch: 21.0.0 - groupfolders: 9.0.0 - impersonate: 1.8.0 - integration_github: 1.0.0 - integration_mastodon: 1.0.0 - integration_twitter: 1.0.0 - issuetemplate: 0.7.0 - logreader: 2.6.0 - lookup_server_connector: 1.9.0 - maps: 0.1.8 - metadata: 0.13.0 - music: 1.1.0 - news: 15.3.2 - nextcloud_announcements: 1.10.0 - notes: 4.0.4 - notifications: 2.9.0 - notify_push: 0.1.6 - oauth2: 1.9.0 - passman: 2.3.7 - password_policy: 1.11.0 - phonetrack: 0.6.7 - photos: 1.3.0 - polls: 1.8.1 - previewgenerator: 3.1.1 - pride: 0.0.3 - privacy: 1.5.0 - provisioning_api: 1.11.0 - recommendations: 1.0.0 - serverinfo: 1.11.0 - settings: 1.3.0 - sharebymail: 1.11.0 - side_menu: 1.23.1 - social: 0.4.2 - spreed: 11.1.1 - support: 1.4.0 - survey_client: 1.9.0 - systemtags: 1.11.0 - tasks: 0.13.6 - telephoneprovider: 1.0.3 - text: 3.2.0 - theming: 1.12.0 - twofactor_backupcodes: 1.10.0 - updatenotification: 1.11.0 - user_status: 1.1.1 - viewer: 1.5.0 - weather_status: 1.1.0 - workflowengine: 2.3.0 Disabled: - accessibility - apporder - bruteforcesettings - circles - encryption - epubreader - files_linkeditor - files_trackdownloads - flowupload - integration_discourse - nextbackup - suspicious_login - talk_matterbridge - talk_simple_poll - user_ldap ```
Configuration (config/config.php) ``` { "instanceid": "***REMOVED SENSITIVE VALUE***", "passwordsalt": "***REMOVED SENSITIVE VALUE***", "secret": "***REMOVED SENSITIVE VALUE***", "trusted_domains": [ "nextcloud.mybase.selfhost.de", "cloudnext.mybase.selfhost.de" ], "trusted_proxies": "***REMOVED SENSITIVE VALUE***", "default_phone_region": "DE", "datadirectory": "***REMOVED SENSITIVE VALUE***", "tempdirectory": "\/var\/tmp\/nextcloudtemp", "overwrite.cli.url": "https:\/\/nextcloud.mybase.selfhost.de", "knowledgebaseenabled": true, "dbtype": "mysql", "version": "21.0.1.0", "dbname": "***REMOVED SENSITIVE VALUE***", "dbhost": "***REMOVED SENSITIVE VALUE***", "dbport": "", "dbtableprefix": "oc_", "dbuser": "***REMOVED SENSITIVE VALUE***", "dbpassword": "***REMOVED SENSITIVE VALUE***", "installed": true, "updater.release.channel": "beta", "appstore.experimental.enabled": true, "maintenance": false, "loglevel": 3, "debug": false, "htaccess.RewriteBase": "\/", "activity_use_cached_mountpoints": true, "session_keepalive": true, "session_lifetime": 86400, "versions_retention_obligation": "auto", "trashbin_retention_obligation": "auto", "enable_previews": true, "preview_max_x": 2560, "preview_max_y": 1600, "preview_max_filesize_image": 200, "preview_libreoffice_path": "\/usr\/bin\/libreoffice", "preview_office_cl_parameters": " --headless --nologo --nofirststartwizard --invisible --norestore --convert-to pdf --outdir ", "enabledPreviewProviders": [ "OC\\Preview\\PNG", "OC\\Preview\\JPEG", "OC\\Preview\\GIF", "OC\\Preview\\BMP", "OC\\Preview\\XBitmap", "OC\\Preview\\MP3", "OC\\Preview\\TXT", "OC\\Preview\\MarkDown", "OC\\Preview\\TIFF", "OC\\Preview\\SVG", "OC\\Preview\\Postscript", "OC\\Preview\\OpenDocument", "OC\\Preview\\PDF", "OC\\Preview\\Font", "OC\\Preview\\Epub", "OC\\Preview\\StarOffice", "OC\\Preview\\HEIC", "OC\\Preview\\KRITA" ], "mail_from_address": "***REMOVED SENSITIVE VALUE***", "mail_smtpmode": "smtp", "mail_smtpauthtype": "LOGIN", "mail_domain": "***REMOVED SENSITIVE VALUE***", "log_type": "syslog", "logfile": "\/var\/log\/nextcloud.log", "logfilemode": 416, "log_rotate_size": 104857600, "syslog_tag": "Nextcloud", "memcache.distributed": "\\OC\\Memcache\\Memcached", "memcache.local": "\\OC\\Memcache\\Redis", "filelocking.enabled": "true", "filelocking.ttl": 3600, "memcache.locking": "\\OC\\Memcache\\Redis", "redis": { "host": "***REMOVED SENSITIVE VALUE***", "port": 6379, "timeout": 0, "dbindex": 0, "password": "***REMOVED SENSITIVE VALUE***" }, "memcached_servers": [ [ "192.168.1.10", 11211 ], [ "192.168.1.50", 11211 ] ], "data-fingerprint": "790e9a77b6ee76b2af214c8f58d75dfd", "mail_smtphost": "***REMOVED SENSITIVE VALUE***", "mail_smtpauth": 1, "mail_smtpname": "***REMOVED SENSITIVE VALUE***", "mail_smtppassword": "***REMOVED SENSITIVE VALUE***", "mail_smtpsecure": "tls", "mysql.utf8mb4": true, "allow_user_to_change_display_name": true, "auth.bruteforce.protection.enabled": true, "ldapIgnoreNamingRules": false, "ldapProviderFactory": "OCA\\User_LDAP\\LDAPProviderFactory", "theme": "", "app_install_overwrite": [ "fulltextsearch", "fulltextsearch_elasticsearch", "files_fulltextsearch", "files_fulltextsearch_tesseract", "passman", "pride", "checksum", "files_markdown", "telephoneprovider", "circles", "camerarawpreviews", "files_external_gdrive", "music", "facerecognition", "groupfolders", "side_menu", "metadata", "issuetemplate" ], "updater.secret": "***REMOVED SENSITIVE VALUE***" } ```

Are you using external storage, if yes which one: smb

Are you using encryption:

Are you using an external user-backend, if yes which one: LDAP/ActiveDirectory/Webdav/...

Client configuration

Browser: Mozilla/5.0 (X11; Linux x86_64; rv:86.0) Gecko/20100101 Firefox/86.0

Operating system:

Logs

Web server error log ``` Insert your web server log here ```
Nextcloud log ``` Apr 08 10:37:33 rin libsmbclient[2213]: {"reqId":"ipoKR7FMx5eRhV7BUHwF","level":3,"time":"2021-04-08T08:37:33+00:00","remoteAddr":"84.134.23.233","user":"shinji","app":"PHP","method":"GET","url":"/index.php/apps/files/api/v1/thumbnail/256/256/Bilder/Photos/2021/04/IMG_20210408_103713.jpg","message":"{\"Exception\":\"Error\",\"Message\":\"fread(): Icewind\\\\SMB\\\\Native\\\\NativeReadStream::stream_read - read 253952 bytes more data than requested (262144 read, 8192 max) - excess data will be lost at /srv/www/vhosts/nextcloud/apps/files_external/3rdparty/icewind/streams/src/Wrapper.php#55\",\"Code\":0,\"Trace\":[{\"function\":\"onError\",\"class\":\"OC\\\\Log\\\\ErrorHandler\",\"type\":\"::\",\"args\":[2,\"fread(): Icewind\\\\SMB\\\\Native\\\\NativeReadStream::stream_read - read 253952 bytes more data than requested (262144 read, 8192 max) - excess data will be lost\",\"/srv/www/vhosts/nextcloud/apps/files_external/3rdparty/icewind/streams/src/Wrapper.php\",55,{\"count\":8192}]},{\"file\":\"/srv/www/vhosts/nextcloud/apps/files_external/3rdparty/icewind/streams/src/Wrapper.php\",\"line\":55,\"function\":\"fread\",\"args\":[null,8192]},{\"file\":\"/srv/www/vhosts/nextcloud/apps/files_external/3rdparty/icewind/streams/src/CallbackWrapper.php\",\"line\":96,\"function\":\"stream_read\",\"class\":\"Icewind\\\\Streams\\\\Wrapper\",\"type\":\"->\",\"args\":[8192]},{\"function\":\"stream_read\",\"class\":\"Icewind\\\\Streams\\\\CallbackWrapper\",\"type\":\"->\",\"args\":[8192]},{\"file\":\"/srv/www/vhosts/nextcloud/lib/private/Preview/ProviderV2.php\",\"line\":92,\"function\":\"file_put_contents\",\"args\":[\"/tmp/oc_tmp_XQQFAM\",null]},{\"file\":\"/srv/www/vhosts/nextcloud/lib/private/Preview/Image.php\",\"line\":50,\"function\":\"getLocalFile\",\"class\":\"OC\\\\Preview\\\\ProviderV2\",\"type\":\"->\",\"args\":[{\"__class__\":\"OC\\\\Files\\\\Node\\\\File\"}]},{\"file\":\"/srv/www/vhosts/nextcloud/lib/private/Preview/GeneratorHelper.php\",\"line\":63,\"function\":\"getThumbnail\",\"class\":\"OC\\\\Preview\\\\Image\",\"type\":\"->\",\"args\":[{\"__class__\":\"OC\\\\Files\\\\Node\\\\File\"},2560,1600]},{\"file\":\"/srv/www/vhosts/nextcloud/lib/private/Preview/Generator.php\",\"line\":244,\"function\":\"getThumbnail\",\"class\":\"OC\\\\Preview\\\\GeneratorHelper\",\"type\":\"->\",\"args\":[{\"__class__\":\"OC\\\\Preview\\\\JPEG\"},{\"__class__\":\"OC\\\\Files\\\\Node\\\\File\"},2560,1600]},{\"file\":\"/srv/www/vhosts/nextcloud/lib/private/Preview/Generator.php\",\"line\":140,\"function\":\"getMaxPreview\",\"class\":\"OC\\\\Preview\\\\Generator\",\"type\":\"->\",\"args\":[{\"__class__\":\"OC\\\\Files\\\\SimpleFS\\\\SimpleFolder\"},{\"__class__\":\"OC\\\\Files\\\\Node\\\\File\"},\"image/jpeg\",\"\"]},{\"file\":\"/srv/www/vhosts/nextcloud/lib/private/Preview/Generator.php\",\"line\":109,\"function\":\"generatePreviews\",\"class\":\"OC\\\\Preview\\\\Generator\",\"type\":\"->\",\"args\":[{\"__class__\":\"OC\\\\Files\\\\Node\\\\File\"},[{\"width\":256,\"height\":256,\"crop\":true,\"mode\":\"fill\"}],\"image/jpeg\"]},{\"file\":\"/srv/www/vhosts/nextcloud/lib/private/PreviewManager.php\",\"line\":190,\"function\":\"getPreview\",\"class\":\"OC\\\\Preview\\\\Generator\",\"type\":\"->\",\"args\":[{\"__class__\":\"OC\\\\Files\\\\Node\\\\File\"},256,256,true,\"fill\",null]},{\"file\":\"/srv/www/vhosts/nextcloud/apps/files/lib/Controller/ApiController.php\",\"line\":130,\"function\":\"getPreview\",\"class\":\"OC\\\\PreviewManager\",\"type\":\"->\",\"args\":[{\"__class__\":\"OC\\\\Files\\\\Node\\\\File\"},256,256,true]},{\"file\":\"/srv/www/vhosts/nextcloud/lib/private/AppFramework/Http/Dispatcher.php\",\"line\":218,\"function\":\"getThumbnail\",\"class\":\"OCA\\\\Files\\\\Controller\\\\ApiController\",\"type\":\"->\",\"args\":[256,256,{\"__class__\":\"OC\\\\Files\\\\Node\\\\File\"}]},{\"file\":\"/srv/www/vhosts/nextcloud/lib/private/AppFramework/Http/Dispatcher.php\",\"line\":127,\"function\":\"executeController\",\"class\":\"OC\\\\AppFramework\\\\Http\\\\Dispatcher\",\"type\":\"->\",\"args\":[{\"__class__\":\"OCA\\\\Files\\\\Controller\\\\ApiController\"},\"getThumbnail\"]},{\"file\":\"/srv/www/vhosts/nextcloud/lib/private/AppFramework/App.php\",\"line\":157,\"function\":\"dispatch\",\"class\":\"OC\\\\AppFramework\\\\Http\\\\Dispatcher\",\"type\":\"->\",\"args\":[{\"__class__\":\"OCA\\\\Files\\\\Controller\\\\ApiController\"},\"getThumbnail\"]},{\"file\":\"/srv/www/vhosts/nextcloud/lib/private/Route/Router.php\",\"line\":302,\"function\":\"main\",\"class\":\"OC\\\\AppFramework\\\\App\",\"type\":\"::\",\"args\":[\"APIController\",\"getThumbnail\",{\"__class__\":\"OC\\\\AppFramework\\\\DependencyInjection\\\\DIContainer\"},{\"x\":\"256\",\"y\":\"256\",\"file\":\"Bilder/Photos/2021/04/IMG_20210408_103713.jpg\",\"_route\":\"files.API.getThumbnail\"}]},{\"file\":\"/srv/www/vhosts/nextcloud/lib/base.php\",\"line\":993,\"function\":\"match\",\"class\":\"OC\\\\Route\\\\Router\",\"type\":\"->\",\"args\":[\"/apps/files/api/v1/thumbnail/256/256/Bilder/Photos/2021/04/IMG_20210408_103713.jpg\"]},{\"file\":\"/srv/www/vhosts/nextcloud/index.php\",\"line\":37,\"function\":\"handleRequest\",\"class\":\"OC\",\"type\":\"::\",\"args\":[]}],\"File\":\"/srv/www/vhosts/nextcloud/lib/private/Log/ErrorHandler.php\",\"Line\":92,\"CustomMessage\":\"--\"}","userAgent":"Mozilla/5.0 (Android) Nextcloud-android/3.15.1","version":"21.0.1.0"} ```
Browser log Insert your browser log here, this could for example include: a) The javascript console log b) The network log c) ...

720-1455-max

tsueri commented 3 years ago

There's also a problem with PDFs, if you try to open them on a SMB-Share:

image

Opening files from an SMB-Share also resluts in this error with onlyoffice:

image

This is the related log entry:

[PHP] Error: Error: stream_get_contents(): Icewind\SMB\Native\NativeReadStream::stream_read - read 6762 bytes more data than requested (14954 read, 8192 max) - excess data will be lost at /usr/local/www/nginx-dist/nextcloud/lib/private/Files/Storage/Common.php#200 at <<closure>>

 0. <<closure>>
    OC\Log\ErrorHandler::onError(2, "stream_get_cont ... t", "/usr/local/www/ ... p", 200, {path: "06 Abtei ... l})
 1. /usr/local/www/nginx-dist/nextcloud/lib/private/Files/Storage/Common.php line 200
    stream_get_contents(null)
 2. /usr/local/www/nginx-dist/nextcloud/lib/private/Files/Storage/Wrapper/Wrapper.php line 246
    OC\Files\Storage\Common->file_get_contents("06 Abteilung Ba ... x")
 3. /usr/local/www/nginx-dist/nextcloud/lib/private/Files/Storage/Wrapper/Wrapper.php line 246
    OC\Files\Storage\Wrapper\Wrapper->file_get_contents("06 Abteilung Ba ... x")
 4. /usr/local/www/nginx-dist/nextcloud/lib/private/Files/Storage/Wrapper/Availability.php line 265
    OC\Files\Storage\Wrapper\Wrapper->file_get_contents("06 Abteilung Ba ... x")
 5. /usr/local/www/nginx-dist/nextcloud/lib/private/Files/Storage/Wrapper/Wrapper.php line 246
    OC\Files\Storage\Wrapper\Availability->file_get_contents("06 Abteilung Ba ... x")
 6. /usr/local/www/nginx-dist/nextcloud/lib/private/Files/Storage/Wrapper/Wrapper.php line 246
    OC\Files\Storage\Wrapper\Wrapper->file_get_contents("06 Abteilung Ba ... x")
 7. /usr/local/www/nginx-dist/nextcloud/lib/private/Files/View.php line 1167
    OC\Files\Storage\Wrapper\Wrapper->file_get_contents("06 Abteilung Ba ... x")
 8. /usr/local/www/nginx-dist/nextcloud/lib/private/Files/View.php line 597
    OC\Files\View->basicOperation("file_get_contents", "/NCuser ... x", ["read"])
 9. /usr/local/www/nginx-dist/nextcloud/lib/private/Files/Node/File.php line 57
    OC\Files\View->file_get_contents("/NCuser ... x")
10. /usr/local/www/nginx-dist/nextcloud/apps/onlyoffice/controller/callbackcontroller.php line 295
    OC\Files\Node\File->getContent()
11. /usr/local/www/nginx-dist/nextcloud/lib/private/AppFramework/Http/Dispatcher.php line 169
    OCA\Onlyoffice\Controller\CallbackController->download("eyJ0eXAiOiJKV1Q ... o")
12. /usr/local/www/nginx-dist/nextcloud/lib/private/AppFramework/Http/Dispatcher.php line 100
    OC\AppFramework\Http\Dispatcher->executeController(OCA\Onlyoffice\C ... {}, "download")
13. /usr/local/www/nginx-dist/nextcloud/lib/private/AppFramework/App.php line 152
    OC\AppFramework\Http\Dispatcher->dispatch(OCA\Onlyoffice\C ... {}, "download")
14. /usr/local/www/nginx-dist/nextcloud/lib/private/Route/Router.php line 309
    OC\AppFramework\App::main("OCA\\Onlyoffice ... r", "download", OC\AppFramework\ ... {}, {_route: "onlyoffice.callback.download"})
15. /usr/local/www/nginx-dist/nextcloud/lib/base.php line 1008
    OC\Route\Router->match("/apps/onlyoffice/download")
16. /usr/local/www/nginx-dist/nextcloud/index.php line 37
    OC::handleRequest()

GET /apps/onlyoffice/download?doc=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJhY3Rpb24iOiJkb3dubG9hZCIsImZpbGVJZCI6MTAzMjE3MzksInVzZXJJZCI6ImpvbmFzLnp1ZXJjaGVyIn0.krSgL_a7mKFKNIOkO3AyJAO0sVZp5SjWzVHRuoZkJ0o
from xxx.xxx.xxx.xxx by NCuser at 2021-04-09T10:34:25+00:00
tfischer77 commented 3 years ago

Exact same problem here since the upgrade of nextcloud to 21.0.1.1 (Debian 10 buster). It used to work on nextcloud 20.0 {"reqId":"UFEfRIRM1StPanG5TPc9","level":3,"time":"2021-04-09T11:17:30+00:00","remoteAddr":"172.29.245.218","user":"0DDE4180-D98F-4A42-B092-9EBCF09A70FB","app":"PHP","method":"GET","url":"/nextcloud/remote.php/webdav/05_Automation//01_Backbone/INX_C.vsdx","message":{"Exception":"Error","Message":"fread(): Icewind\\SMB\\Native\\NativeReadStream::stream_read - read 229376 bytes more data than requested (237568 read, 8192 max) - excess data will be lost at /srv/nextcloud/apps/files_external/3rdparty/icewind/streams/src/Wrapper.php#55","Code":0,"Trace":[{"function":"onError","class":"OC\\Log\\ErrorHandler","type":"::","args":[2,"fread(): Icewind\\SMB\\Native\\NativeReadStream::stream_read - read 229376 bytes more data than requested (237568 read, 8192 max) - excess data will be lost","/srv/nextcloud/apps/files_external/3rdparty/icewind/streams/src/Wrapper.php",55,{"count":8192}]},{"file":"/srv/nextcloud/apps/files_external/3rdparty/icewind/streams/src/Wrapper.php","line":55,"function":"fread","args":[null,8192]},{"file":"/srv/nextcloud/apps/files_external/3rdparty/icewind/streams/src/CallbackWrapper.php","line":96,"function":"stream_read","class":"Icewind\\Streams\\Wrapper","type":"->","args":[8192]},{"function":"stream_read","class":"Icewind\\Streams\\CallbackWrapper","type":"->","args":[8192]},{"file":"/srv/nextcloud/3rdparty/sabre/http/lib/Sapi.php","line":112,"function":"stream_copy_to_stream","args":[null,null,3635960]},{"file":"/srv/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php","line":490,"function":"sendResponse","class":"Sabre\\HTTP\\Sapi","type":"::","args":[{"__class__":"Sabre\\HTTP\\Response"}]},{"file":"/srv/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php","line":253,"function":"invokeMethod","class":"Sabre\\DAV\\Server","type":"->","args":[{"__class__":"Sabre\\HTTP\\Request"},{"__class__":"Sabre\\HTTP\\Response"}]},{"file":"/srv/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php","line":321,"function":"start","class":"Sabre\\DAV\\Server","type":"->","args":[]},{"file":"/srv/nextcloud/apps/dav/appinfo/v1/webdav.php","line":84,"function":"exec","class":"Sabre\\DAV\\Server","type":"->","args":[]},{"file":"/srv/nextcloud/remote.php","line":167,"args":["/srv/nextcloud/apps/dav/appinfo/v1/webdav.php"],"function":"require_once"}],"File":"/srv/nextcloud/lib/private/Log/ErrorHandler.php","Line":92,"CustomMessage":"--"},"userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.114 Safari/537.36","version":"21.0.1.1"}

muchachagrande commented 3 years ago

Same problem here

EDIT 1: I can confirm that it was working fine on 21.0.0

I'm using Nextcloud official VM from hanssonit.se

EDIT 2: May be related to this update: https://github.com/nextcloud/server/pull/26263

cheneraie commented 3 years ago

Same problem after update 20.0.8 to 20.0.9. I had to restore 20.0.8. Debian 10, Apache 2.4, PHP 7.3

snetat commented 3 years ago

Same problem after update 20.0.8.1 to 21.0.1.1 Debian 10, Apache 2.4, PHP 7.3

@muchahagrande, good hint! I have /var/www/apps/files_external/3rdparty/icewind (only icewind dir) restored from version 20.0.8.1 and now it works again

CacheMinimal commented 3 years ago

Same issue here with a 20.0.8.1 to 21.0.1.1 upgrade, as released today.

I am running on Ubuntu Server 20.04.2 LTS. The issue manifests as a whole series of massive buffer overflows in the fread() function when accessing External Storage via SMB:

fread(): Icewind\\SMB\\Native\\NativeReadStream::stream_read - read 8192 bytes more data than requested (16384 read, 8192 max) - excess data will be lost at /srv/www/nextcloud/apps/files_external/3rdparty/icewind/streams/src/Wrapper.php#55

I have narrowed the issue down to only happening when php-smbclient is installed (in this case from php7.4-smbclient_1.0.6-1+ubuntu20.04.1+deb.sury.org+1_amd64.deb via https://launchpad.net/~ondrej/+archive/ubuntu/php due to the package having been pulled from Debian repositories due to licensing issues).

I have another identical server running Nextcloud 20.0.8.1 with the same php7.4-smbclient_1.0.6-1+ubuntu20.04.1+deb.sury.org+1_amd64.deb installed. This does not exhibit any such issues.

My workaround is to remove php-smbclient and install just smbclient instead. After a bounce of the web server (Apache, in this case) access to the External Storage via SMB returns without corruption. Sadly the notify of changes implemented via php-smbclient is lost and I regularly need to scan the external storage for changes.

A quick patch would be much appreciated. Maybe @icewind1991 has already seen this?

jfdelaquis commented 3 years ago

Could also be related with #26303. It is mentioned that the problem is with SFTP, although I experience the same issue with the SMB mounts as outlined here. Opening the files directly from the server, there is no corruption, but opening from withing the WebUI I experience the same as mentioned above; pictures open as seen in the first post or not at all and pdf files don't work. Audio files skip in the playback. After checking the logs, it is related with fread capping at 8192 bytes as mentioned above.

The problem also occurred after updating from 20.0.8 to 21.0.1

muchachagrande commented 3 years ago

Same problem after update 20.0.8.1 to 21.0.1.1 Debian 10, Apache 2.4, PHP 7.3

@muchahagrande, good hint! I have /var/www/apps/files_external/3rdparty/icewind (only icewind dir) restored from version 20.0.8.1 and now it works again

@Aejoh, I've just tried your workaround and I can confirm that is working again. Thank you.

KekcuHa commented 3 years ago

I have /var/www/apps/files_external/3rdparty/icewind (only icewind dir) restored from version 20.0.8.1 and now it works again

Same issue after upgrade 21.0.0 -> 21.0.1, same workaround (replace icewing with previous version).

manjman commented 3 years ago

Same issue on a fresh 21.0.1.1 docker install (latest fpm)

efelon commented 3 years ago

I have installed NC manually and can confirm the workaround as well. For me the path is:

/var/www/nextcloud/apps/files_external/3rdparty/icewind

scharel commented 3 years ago

I have the same issue after updating from 20.0.8 to 20.0.9. Restoring icewind from the updater-backup as described above resolved the issue.

fracklaus commented 3 years ago

Same here, after upgrading from 20.0.8 to 21.0.1. Upload seems to work for me, downloads are broken. Edit: Workaround (replacing apps/files_external/3rdparty/icewind with content from version 21.0.0 seems to be working for me as well).

helmut72 commented 3 years ago

Same for me. Replacing it with the older version works.

alex-c3 commented 3 years ago

Had the same issue after upgrading from 20.0.2 to 21.0.1. Note that that last backup didn't contain the "files_external" forlder in apps. I guess because the update failed a few times before succeeding. Restoring 'icewind' from official 20.0.9 download didn't work, but from the previous backup from november 2020 worked.

klodner commented 3 years ago

Had the same problem after upgrade from 20 to 21, the files were corrupted at 16KB I can confirm the old Icewind replacement works, found it in this related bug https://github.com/nextcloud/server/issues/20622

RisedSky commented 3 years ago

Hello, i have the same problem, SMB doesn't work and i have this excess data will be lost at /var/www/nextcloud/apps/files_external/3rdparty/icewind/streams/src/Wrapper.php#55 at <<closure>>

small1 commented 3 years ago

Confirming about restoring files from an older files_external version. This will affect a few of my installation if i'm upgrading them

RisedSky commented 3 years ago

Same problem after update 20.0.8.1 to 21.0.1.1 Debian 10, Apache 2.4, PHP 7.3 @muchahagrande, good hint! I have /var/www/apps/files_external/3rdparty/icewind (only icewind dir) restored from version 20.0.8.1 and now it works again

@Aejoh, I've just tried your workaround and I can confirm that is working again. Thank you.

can you provide a way to do this pls ?

rroseselavy42 commented 3 years ago

able to reproduce, Debian 10, Apache 2.4, PHP 7.4 with php-smbclient

workaround of copying old icewind directory works.

muchachagrande commented 3 years ago

@RisedSky, the steps to do it may differe a bit from system to system, but essentially: 1 - locate "icewind" directory in your Nextcloud installation may be something like /var/www/apps/files_external/3rdparty/icewind or /var/www/nextcloud/apps/files_external/3rdparty/icewind 2 - Take the content of "icewind" directory of the last backup, the one before the Nextcloud update 3 - Remove the contents of the actual "icewind" directory 4 - Put the backed up "icewind" content in to the actual directory

EDIT: I've done this with Nextcloud in maintenance mode.

RisedSky commented 3 years ago

@RisedSky, the steps to do it may differe a bit from system to system, but essentially: 1 - locate "icewind" directory in your Nextcloud installation may be something like /var/www/apps/files_external/3rdparty/icewind or /var/www/nextcloud/apps/files_external/3rdparty/icewind 2 - Take the content of "icewind" directory of the last backup, the one before the Nextcloud update 3 - Remove the contents of the actual "icewind" directory 4 - Put the backed up "icewind" content in to the actual directory

EDIT: I've done this with Nextcloud in maintenance mode.

Thanks !!!!!!!

Mith-09 commented 3 years ago

@RisedSky, the steps to do it may differe a bit from system to system, but essentially: 1 - locate "icewind" directory in your Nextcloud installation may be something like /var/www/apps/files_external/3rdparty/icewind or /var/www/nextcloud/apps/files_external/3rdparty/icewind 2 - Take the content of "icewind" directory of the last backup, the one before the Nextcloud update 3 - Remove the contents of the actual "icewind" directory 4 - Put the backed up "icewind" content in to the actual directory

EDIT: I've done this with Nextcloud in maintenance mode.

Dit this but had mutiples issues with my server after getting back the old icewind folder from my NC 20.8 :(

muchachagrande commented 3 years ago

Just for help @RisedSky . You don't need to extract icewind from an old backup. Instead, you may download the previous Nextcloud version from the web page: https://download.nextcloud.com/server/releases/nextcloud-21.0.0.tar.bz2 Extract icewind from this archive.

snetat commented 3 years ago

@RisedSky, the steps to do it may differe a bit from system to system, but essentially: 1 - locate "icewind" directory in your Nextcloud installation may be something like /var/www/apps/files_external/3rdparty/icewind or /var/www/nextcloud/apps/files_external/3rdparty/icewind 2 - Take the content of "icewind" directory of the last backup, the one before the Nextcloud update 3 - Remove the contents of the actual "icewind" directory 4 - Put the backed up "icewind" content in to the actual directory EDIT: I've done this with Nextcloud in maintenance mode.

Dit this but had mutiples issues with my server after getting back the old icewind folder from my NC 20.8 :(

which errors, so that we can understand them

Mith-09 commented 3 years ago

@RisedSky, the steps to do it may differe a bit from system to system, but essentially: 1 - locate "icewind" directory in your Nextcloud installation may be something like /var/www/apps/files_external/3rdparty/icewind or /var/www/nextcloud/apps/files_external/3rdparty/icewind 2 - Take the content of "icewind" directory of the last backup, the one before the Nextcloud update 3 - Remove the contents of the actual "icewind" directory 4 - Put the backed up "icewind" content in to the actual directory EDIT: I've done this with Nextcloud in maintenance mode.

Dit this but had mutiples issues with my server after getting back the old icewind folder from my NC 20.8 :(

which errors, so that we can understand them

When I logged as admin, my users disappeared, also files. Some apps aren't working anymore (Passwords, for example. Couldn't connect to the base of my credentials). Didn't explore the other errors, sorry, was my production environment. Had to copy back the nc21.0.1 icewind folder to make it work again.

muchachagrande commented 3 years ago

If you are restarting from 21.0.1, then you may try with the other workaround that @CacheMinimal did upstairs.

My workaround is to remove php-smbclient and install just smbclient instead. After a bounce of the web server (Apache, in this case) access to the External Storage via SMB returns without corruption. Sadly the notify of changes implemented via php-smbclient is lost and I regularly need to scan the external storage for changes.

snetat commented 3 years ago

@RisedSky, the steps to do it may differe a bit from system to system, but essentially: 1 - locate "icewind" directory in your Nextcloud installation may be something like /var/www/apps/files_external/3rdparty/icewind or /var/www/nextcloud/apps/files_external/3rdparty/icewind 2 - Take the content of "icewind" directory of the last backup, the one before the Nextcloud update 3 - Remove the contents of the actual "icewind" directory 4 - Put the backed up "icewind" content in to the actual directory EDIT: I've done this with Nextcloud in maintenance mode.

Dit this but had mutiples issues with my server after getting back the old icewind folder from my NC 20.8 :(

which errors, so that we can understand them

When I logged as admin, my users disappeared, also files. Some apps aren't working anymore (Passwords, for example. Couldn't connect to the base of my credentials). Didn't explore the other errors, sorry, was my production environment. Had to copy back the nc21.0.1 icewind folder to make it work again.

are you sure you have used the correct directory ? icewind can be found in /var/www//apps/files_external/3rdparty as well as /var/www/3rdparty

Mith-09 commented 3 years ago

@RisedSky, the steps to do it may differe a bit from system to system, but essentially: 1 - locate "icewind" directory in your Nextcloud installation may be something like /var/www/apps/files_external/3rdparty/icewind or /var/www/nextcloud/apps/files_external/3rdparty/icewind 2 - Take the content of "icewind" directory of the last backup, the one before the Nextcloud update 3 - Remove the contents of the actual "icewind" directory 4 - Put the backed up "icewind" content in to the actual directory EDIT: I've done this with Nextcloud in maintenance mode.

Dit this but had mutiples issues with my server after getting back the old icewind folder from my NC 20.8 :(

which errors, so that we can understand them

When I logged as admin, my users disappeared, also files. Some apps aren't working anymore (Passwords, for example. Couldn't connect to the base of my credentials). Didn't explore the other errors, sorry, was my production environment. Had to copy back the nc21.0.1 icewind folder to make it work again.

are you sure you have used the correct directory ? icewind can be found in /var/www//apps/files_external/3rdparty as well as /var/www/3rdparty

Yes sir. It was the good one. Now I downloaded the icewind from a fresh 21.0.0, and it works. The xslx I was trying to open during my test still doesn't open in onyloffice. But I can download it without errors. Trying on multiple other documents but don't see any errors.

Edit for my case: The error with Onlyoffice is not directly linked with the icewind one. It appears only on files I tried to open several times during the icewind bug. Just have to download the file, erase it from the cloud, then reupload it. Onlyoffice can open the "new" one.

PnjDbq commented 3 years ago

Confirmed with 21.0.1 using docker container linuxserver/nextcloud

Jednadvacet commented 3 years ago

@RisedSky, the steps to do it may differe a bit from system to system, but essentially: 1 - locate "icewind" directory in your Nextcloud installation may be something like /var/www/apps/files_external/3rdparty/icewind or /var/www/nextcloud/apps/files_external/3rdparty/icewind 2 - Take the content of "icewind" directory of the last backup, the one before the Nextcloud update 3 - Remove the contents of the actual "icewind" directory 4 - Put the backed up "icewind" content in to the actual directory

EDIT: I've done this with Nextcloud in maintenance mode.

You made my day!!! Thanks a lot!

Kevin1357 commented 3 years ago

I fix it with the manual from muchahagrande.

Do Anyone know if the files on the SMB Storage have been physicaly damaged?

PnjDbq commented 3 years ago

Do Anyone know if the files on the SMB Storage have been physicaly damaged?

That depends (on some variable I can't quite determine).

I send PDF's from my scanner to the SMB share on my Synology, mount the external storage in Nextcloud, and copy files from the share into Nextcloud.

Some of my PDF's were damaged. Some were not.

Good news (for me) is that the originals were sitting on the scanner...I scanned them again and loaded them into Nextcloud without using the external storage.

Snousmoumryk commented 3 years ago

Do Anyone know if the files on the SMB Storage have been physicaly damaged?

SMB share files remain intact

Jednadvacet commented 3 years ago

I fix it with the manual from muchahagrande.

Do Anyone know if the files on the SMB Storage have been physicaly damaged?

Don't think so. My external files seem healthy. By now.

NetBaron78 commented 3 years ago

The fix didnt' work by me. I had someone replace the file for me with the one in the version downloaded here https://download.nextcloud.com/server/releases/nextcloud-21.0.0.tar.bz2 but the files won't open and in the logs i still see:

  Error: fread(): Icewind\SMB\Native\NativeReadStream::stream_read - read 18071 bytes more data than requested (26263 read, 8192 max) - excess data will be lost at /var/www/nextcloud/apps/files_external/3rdparty/icewind/streams/src/Wrapper.php#55

Will this be fixed soon in an update?

fracklaus commented 3 years ago

The fix didnt' work by me. I had someone replace the file for me with the one in the version downloaded here https://download.nextcloud.com/server/releases/nextcloud-21.0.0.tar.bz2 but the files won't open

If you are using php-fpm, did you restart it?

NetBaron78 commented 3 years ago

The fix didnt' work by me. I had someone replace the file for me with the one in the version downloaded here https://download.nextcloud.com/server/releases/nextcloud-21.0.0.tar.bz2 but the files won't open

If you are using php-fpm, did you restart it?

I restarted the whole server...

muchachagrande commented 3 years ago

Check the owner and group. Check permissions. If you downloaded the files from https://download.nextcloud.com/server/releases/nextcloud-21.0.0.tar.bz2 you shoud, at least, change the ownership and group to grant the web server access to the new files. Check the owner and group of the files inside /var/www/nextcloud and use them in your recovered files from the archive.

NetBaron78 commented 3 years ago

Check the owner and group. Check permissions. If you downloaded the files from https://download.nextcloud.com/server/releases/nextcloud-21.0.0.tar.bz2 you shoud, at least, change the ownership and group to grant the web server access to the new files. Check the owner and group of the files inside /var/www/nextcloud and use them in your recovered files from the archive.

Oh yeah we did that. At first with the "new" Icewind we could not even login and that was because the directory SMB was not present. So we copied that too and we could log back in but the problem stayed.

image

Kevin1357 commented 3 years ago

Did you change permissions rekursiv? Have you activated maintenance mode while making changes?

Maybe occ files:scan --all can help?

NetBaron78 commented 3 years ago

Did you change permissions rekursiv? Have you activated maintenance mode while making changes?

Maybe occ files:scan --all can help?

recursive: yes, maintenance mode: no and will try the file scan

NetBaron78 commented 3 years ago

occ files:scan --all

what am i doing wrong for scanning?

image

RisedSky commented 3 years ago

occ files:scan --all

what am i doing wrong for scanning?

image

you need to put the /var/www/nextcloud/occ occ is a program by nextcloud, you need to put the path then the command For me it's sudo -u www-data php /var/www/nextcloud/occ maintenance:mode --on in exemple

Helipil0t commented 3 years ago

So I had the same problem and replaced the icewind directory with an older version, made the appropriate modifications to owner / permissions. From a web browser I can now download files from my SMB shares. But I'm getting the same errors when trying to download from the Android App. :(

Error PHP Error: fread(): Icewind\SMB\Native\NativeReadStream::stream_read - read 19511 bytes more data than requested (27703 read, 8192 max) - excess data will be lost at /var/www/html/nextcloud/apps/files_external/3rdparty/icewind/streams/src/Wrapper.php#55<>OC\Log\ErrorHandler::onError(2, "fread(): Ic ... t", "/var/www/ht ... p", 55, { count: 8192})/var/www/html/nextcloud/apps/files_external/3rdparty/icewind/streams/src/Wrapper.php - line 55: fread(null, 8192)/var/www/html/nextcloud/apps/files_external/3rdparty/icewind/streams/src/CallbackWrapper.php - line 96: Icewind\Streams\Wrapper->stream_read(8192)<>Icewind\Streams\CallbackWrapper->stream_read(8192)/var/www/html/nextcloud/3rdparty/sabre/http/lib/Sapi.php - line 112: stream_copy_to_stream(null, null, 281655)/var/www/html/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php - line 492: Sabre\HTTP\Sapi::sendResponse(Sabre\HTTP\Response {})/var/www/html/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php - line 251: Sabre\DAV\Server->invokeMethod(Sabre\HTTP\Request {}, Sabre\HTTP\Response {})/var/www/html/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php - line 319: Sabre\DAV\Server->start()/var/www/html/nextcloud/apps/dav/lib/Server.php - line 332: Sabre\DAV\Server->exec()/var/www/html/nextcloud/apps/dav/appinfo/v2/remote.php - line 35: OCA\DAV\Server->exec()/var/www/html/nextcloud/remote.php - line 167: require_once("/var/www/ht ... p")

I tried Version 21.0.0, 20.0.9 and 20.0.8 all with the same issues using the APP and Linux Client.

Anyone else having issues with Android App or any other client for that matter?

Helipil0t commented 3 years ago

Just a follow up. I started with a fresh install of version 19.0.10 and everything is working perfectly. Will now try to upgrade again from there to 21 and downgrade icewind from version 19 backup. Will report back later.

wwebers commented 3 years ago

Same problem here. Am running version 21.0.1 and tried to replace icewind with the version bundled in 20.0.8. Did not help. The things I observed are:

  1. all file > 10KB are corrupted after downloading via WebDAV (alternative via web client). Smaller files, like simple text file are fine.
  2. Larger files take ages to download and timeout after a few percent downloaded. Resuming them did not help

Question is if downgrade is possible (any database changes involved as I'm running an external postgres database).

mariomaurer commented 3 years ago

if you use a samba share you should change the following setting in /etc/apache2/apache2.conf EnableMMAP on >> EnableMMAP off

https://httpd.apache.org/docs/2.4/en/mod/core.html

wwebers commented 3 years ago

if you use a samba share you should change the following setting in /etc/apache2/apache2.conf EnableMMAP on >> EnableMMAP off

https://httpd.apache.org/docs/2.4/en/mod/core.html

Did that already. Doesn't solve the problem:

  1. files are corrupted
  2. download speed is unbelievable slow (>1min for a file of around 100KB, larger files simply timeout)

Would say: external storage module is completely broken and useless. Temporary workaround: mount external storage outside Nextcloud. Pitfall: no convenient way to handle user credentials is not setting up Kerberos for SSO...

snetat commented 3 years ago

i have traced the error to a change in apps/files_external/3rdparty/icewind/smb from 3.3.1 to 3.4 limited. https://github.com/icewind1991/SMB/releases