Mass periodical external Storage files download after 21.0.5 upgrade.

GAS85 commented 3 years ago

How to use GitHub

Please use the 👍 reaction to show that you are affected by the same issue.
Please don't comment if you have no relevant information to add. It's just extra noise for everyone subscribed to this issue.
Subscribe to receive notifications on status change and new comments.

Steps to reproduce

Use external Storages via WebDav
Upgrade to 21.0.5
From some point periodical mass scan will appear:

Check tmp Folder and it is full of files from the external storage (was checked by download few of tmp files), seems for preview generation (?):

ls -la /tmp/oc_*
-rw------- 1 www-data www-data         0 Nov 17 09:19 /tmp/oc_tmp_0ABNip
-rw------- 1 www-data www-data   6519718 Nov 17 09:19 /tmp/oc_tmp_0ABNip-.JPG
-rw------- 1 www-data www-data         0 Nov 17 09:17 /tmp/oc_tmp_0BASfm
-rw------- 1 www-data www-data      4757 Nov 17 09:17 /tmp/oc_tmp_0BASfm-.png
-rw------- 1 www-data www-data         0 Nov 17 09:16 /tmp/oc_tmp_0JPaKn
-rw------- 1 www-data www-data     14135 Nov 17 09:16 /tmp/oc_tmp_0JPaKn-.png
-rw------- 1 www-data www-data         0 Nov 17 09:16 /tmp/oc_tmp_0Ytsul
-rw------- 1 www-data www-data    400761 Nov 17 09:16 /tmp/oc_tmp_0Ytsul-.png
-rw------- 1 www-data www-data         0 Nov 17 09:16 /tmp/oc_tmp_0iJTEp
-rw------- 1 www-data www-data      8528 Nov 17 09:16 /tmp/oc_tmp_0iJTEp-.png
-rw------- 1 www-data www-data         0 Nov 17 09:17 /tmp/oc_tmp_0xJSqp
-rw------- 1 www-data www-data      3928 Nov 17 09:17 /tmp/oc_tmp_0xJSqp-.png
-rw------- 1 www-data www-data         0 Nov 17 09:16 /tmp/oc_tmp_11tZKo
-rw------- 1 www-data www-data     37630 Nov 17 09:16 /tmp/oc_tmp_11tZKo-.png
-rw------- 1 www-data www-data         0 Nov 17 09:16 /tmp/oc_tmp_1ML1Nl
-rw------- 1 www-data www-data     33559 Nov 17 09:16 /tmp/oc_tmp_1ML1Nl-.png
-rw------- 1 www-data www-data    139075 Nov 17 09:16 /tmp/oc_tmp_1UwNMo
-rw------- 1 www-data www-data         0 Nov 17 09:16 /tmp/oc_tmp_1VtGYl
-rw------- 1 www-data www-data      7168 Nov 17 09:16 /tmp/oc_tmp_1VtGYl-.xls
-rw------- 1 www-data www-data         0 Nov 17 09:17 /tmp/oc_tmp_1YnGln
-rw------- 1 www-data www-data      6147 Nov 17 09:17 /tmp/oc_tmp_1YnGln-.png
-rw------- 1 www-data www-data         0 Nov 17 09:16 /tmp/oc_tmp_1dvkdo
-rw------- 1 www-data www-data   3394241 Nov 17 09:16 /tmp/oc_tmp_1dvkdo-.pdf
-rw------- 1 www-data www-data         0 Nov 17 09:18 /tmp/oc_tmp_2DeKSp
-rw------- 1 www-data www-data  48512299 Nov 17 09:18 /tmp/oc_tmp_2DeKSp-.mp4
....

Seems a lot of data downloaded (around 5 MB/s for 10 Min = around 3 GB downloaded). External Storage is around 3 GB big.
Perform update to 21.0.6 - still exist
Perform update to 21.0.7 - even more frequently scan performed.

Expected behaviour

Scan should not use a lot of bandwidth to get files information only. Even for preview generation unchanged files should not be downloaded again and again.

Actual behaviour

Seems a lot of data downloaded (around 5 MB/s for 10 Min = around 3 GB downloaded). With tcptrack find out that all traffic goes to only 1 external share - Magenta Cloud. Even for preview generation, files were not changed since 2017 (as above), but still downloaded to be analyzed.

Server configuration

Operating system: Ubuntu 20.04

Web server: Apache/2.4.51

Database: 10.3.31-MariaDB

PHP version: 7.4.25

Nextcloud version: 21.0.7

Updated from an older Nextcloud/ownCloud or fresh install: Updated

Where did you install Nextcloud from: Official

Signing status:

Signing status

``` No errors have been found. ```

List of activated apps:

App list

``` Enabled: - accessibility: 1.7.0 - activity: 2.14.3 - admin_audit: 1.11.0 - audioplayer: 3.2.2 - bruteforcesettings: 2.2.0 - calendar: 2.3.4 - camerarawpreviews: 0.7.13 - checksum: 1.1.2 - cloud_federation_api: 1.4.0 - comments: 1.11.0 - contacts: 4.0.6 - contactsinteraction: 1.2.0 - cospend: 1.3.7 - dashboard: 7.1.0 - data_request: 1.8.0 - dav: 1.17.1 - deck: 1.4.7 - drawio: 1.0.1 - external: 3.8.2 - extract: 1.3.2 - facerecognition: 0.8.3 - federatedfilesharing: 1.11.0 - files: 1.16.0 - files_automatedtagging: 1.11.0 - files_downloadactivity: 1.11.1 - files_external: 1.12.1 - files_mindmap: 0.0.25 - files_pdfviewer: 2.1.1 - files_retention: 1.10.2 - files_rightclick: 1.0.0 - files_sharing: 1.13.1 - files_trashbin: 1.11.0 - files_versions: 1.14.0 - files_videoplayer: 1.10.0 - firstrunwizard: 2.10.0 - forms: 2.3.0 - gpxedit: 0.0.14 - gpxmotion: 0.1.0 - gpxpod: 4.2.8 - integration_github: 1.0.0 - integration_gitlab: 1.0.0 - keeweb: 0.6.6 - logreader: 2.6.0 - lookup_server_connector: 1.9.0 - mail: 1.10.5 - maps: 0.1.9 - nextcloud_announcements: 1.10.0 - notes: 4.2.0 - notifications: 2.9.0 - oauth2: 1.9.0 - ocdownloader: 1.7.12 - password_policy: 1.11.0 - phonetrack: 0.6.9 - photos: 1.3.0 - polls: 3.3.0 - previewgenerator: 3.2.0 - privacy: 1.5.0 - provisioning_api: 1.11.0 - radio: 1.0.3 - recognize: 1.6.10 - recommendations: 1.0.0 - serverinfo: 1.11.0 - settings: 1.3.0 - sharebymail: 1.11.0 - side_menu: 2.0.1 - spreed: 11.3.3 - support: 1.4.0 - survey_client: 1.9.0 - systemtags: 1.11.0 - text: 3.2.0 - theming: 1.12.0 - twofactor_backupcodes: 1.10.0 - twofactor_totp: 6.1.0 - unsplash: 1.2.3 - updatenotification: 1.11.0 - user_status: 1.1.1 - video_converter: 1.0.3 - viewer: 1.5.0 - weather: 1.7.5 - workflowengine: 2.3.1 Disabled: - encryption - federation - files_accesscontrol - flowupload - impersonate - integration_jira - user_ldap - weather_status ```

Nextcloud configuration:

Config report

``` { "system": { "instanceid": "***REMOVED SENSITIVE VALUE***", "passwordsalt": "***REMOVED SENSITIVE VALUE***", "secret": "***REMOVED SENSITIVE VALUE***", "trusted_domains": [ "***REMOVED SENSITIVE VALUE***", "***REMOVED SENSITIVE VALUE***", "***REMOVED SENSITIVE VALUE***" ], "datadirectory": "***REMOVED SENSITIVE VALUE***", "overwrite.cli.url": "***REMOVED SENSITIVE VALUE***", "dbtype": "mysql", "version": "21.0.7.0", "dbname": "***REMOVED SENSITIVE VALUE***", "dbhost": "***REMOVED SENSITIVE VALUE***", "dbport": "", "dbtableprefix": "oc_", "mysql.utf8mb4": true, "dbuser": "***REMOVED SENSITIVE VALUE***", "dbpassword": "***REMOVED SENSITIVE VALUE***", "installed": true, "filesystem_check_changes": 0, "filelocking.enabled": true, "memcache.local": "\\OC\\Memcache\\APCu", "memcache.locking": "\\OC\\Memcache\\Redis", "memcache.distributed": "\\OC\\Memcache\\Redis", "redis": { "host": "***REMOVED SENSITIVE VALUE***", "port": 0, "timeout": 1.5 }, "default_phone_region": "DE", "mail_smtpmode": "smtp", "mail_smtpauthtype": "LOGIN", "mail_from_address": "***REMOVED SENSITIVE VALUE***", "mail_domain": "***REMOVED SENSITIVE VALUE***", "maintenance": false, "theme": "", "logfile": "\/var\/nextcloud\/data\/nextcloud.log", "loglevel": 0, "trashbin_retention_obligation": "14, auto", "versions_retention_obligation": "14, auto", "data-fingerprint": "7be51475e95ea13b254cd880319d98ec", "enable_previews": true, "enabledPreviewProviders": [ "OC\\Preview\\PNG", "OC\\Preview\\JPEG", "OC\\Preview\\GIF", "OC\\Preview\\BMP", "OC\\Preview\\XBitmap", "OC\\Preview\\Movie", "OC\\Preview\\PDF", "OC\\Preview\\MP3", "OC\\Preview\\TXT", "OC\\Preview\\MarkDown" ], "preview_max_x": 1920, "preview_max_y": 1080, "jpeg_quality": 90, "auth.bruteforce.protection.enabled": true, "simpleSignUpLink.shown": false, "mail_smtpsecure": "tls", "mail_smtpauth": 1, "mail_smtphost": "***REMOVED SENSITIVE VALUE***", "mail_smtpport": "587", "mail_smtpname": "***REMOVED SENSITIVE VALUE***", "mail_smtppassword": "***REMOVED SENSITIVE VALUE***", "twofactor_enforced": true, "twofactor_enforced_groups": [ "admin" ], "twofactor_enforced_excluded_groups": [], "has_rebuilt_cache": true, "updater.release.channel": "stable", "app_install_overwrite": [ "keeweb", "radio", "gpxedit" ], "mail_sendmailmode": "smtp", "updater.secret": "***REMOVED SENSITIVE VALUE***" } } ```

Are you using external storage, if yes which one: local, WebDav

Are you using encryption: no

Are you using an external user-backend, if yes which one: nope

Logs

Nextcloud log (data/nextcloud.log)

Nextcloud log

```json {"reqId":"CUkykZLjnpyev0OU97Gi","level":3,"time":"2021-11-17T08:14:26+00:00","remoteAddr":"80.146.191.222","user":"gas","app":"core","method":"GET","url":"/index.php/core/preview?fileId=10007334&c=32000000-56c97580b8a62&x=250&y=250&forceIcon=0&a=0","message":"Movie preview generation failed Output: [\"ffmpeg version 4.2.4-1ubuntu0.1 Copyright (c) 2000-2020 the FFmpeg developers\",\" built with gcc 9 (Ubuntu 9.3.0-10ubuntu2)\",\" configuration: --prefix=/usr --extra-version=1ubuntu0.1 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --arch=amd64 --enable-gpl --disable-stripping --enable-avresample --disable-filter=resample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librsvg --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-nvenc --enable-chromaprint --enable-frei0r --enable-libx264 --enable-shared\",\" libavutil 56. 31.100 / 56. 31.100\",\" libavcodec 58. 54.100 / 58. 54.100\",\" libavformat 58. 29.100 / 58. 29.100\",\" libavdevice 58. 8.100 / 58. 8.100\",\" libavfilter 7. 57.100 / 7. 57.100\",\" libavresample 4. 0. 0 / 4. 0. 0\",\" libswscale 5. 5.100 / 5. 5.100\",\" libswresample 3. 5.100 / 3. 5.100\",\" libpostproc 55. 5.100 / 55. 5.100\",\"/tmp/oc_tmp_xm56I1: Invalid data found when processing input\"]","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:93.0) Gecko/20100101 Firefox/93.0","version":"21.0.7.0","id":"6194c54e69998"} {"reqId":"CUkykZLjnpyev0OU97Gi","level":1,"time":"2021-11-17T08:14:23+00:00","remoteAddr":"80.146.191.222","user":"gas","app":"no app in context","method":"GET","url":"/index.php/core/preview?fileId=10007334&c=32000000-56c97580b8a62&x=250&y=250&forceIcon=0&a=0","message":"Deprecated event type for OCP\\IPreview:PreviewRequested: Symfony\\Component\\EventDispatcher\\GenericEvent is used","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:93.0) Gecko/20100101 Firefox/93.0","version":"21.0.7.0","id":"6194c54e69a89"} {"reqId":"o3cMfvFqbiUS3hagv1If","level":3,"time":"2021-11-17T08:14:19+00:00","remoteAddr":"80.146.191.222","user":"gas","app":"core","method":"GET","url":"/index.php/core/preview?fileId=10007334&c=32000000-56c97580b8a62&x=250&y=250&forceIcon=0&a=0","message":"Movie preview generation failed Output: [\"ffmpeg version 4.2.4-1ubuntu0.1 Copyright (c) 2000-2020 the FFmpeg developers\",\" built with gcc 9 (Ubuntu 9.3.0-10ubuntu2)\",\" configuration: --prefix=/usr --extra-version=1ubuntu0.1 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --arch=amd64 --enable-gpl --disable-stripping --enable-avresample --disable-filter=resample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librsvg --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-nvenc --enable-chromaprint --enable-frei0r --enable-libx264 --enable-shared\",\" libavutil 56. 31.100 / 56. 31.100\",\" libavcodec 58. 54.100 / 58. 54.100\",\" libavformat 58. 29.100 / 58. 29.100\",\" libavdevice 58. 8.100 / 58. 8.100\",\" libavfilter 7. 57.100 / 7. 57.100\",\" libavresample 4. 0. 0 / 4. 0. 0\",\" libswscale 5. 5.100 / 5. 5.100\",\" libswresample 3. 5.100 / 3. 5.100\",\" libpostproc 55. 5.100 / 55. 5.100\",\"/tmp/oc_tmp_7KkV5J: Invalid data found when processing input\"]","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:93.0) Gecko/20100101 Firefox/93.0","version":"21.0.7.0","id":"6194c54e6a0b7"} ```

szaimen commented 3 years ago

Hey, sounds like you are running a previewgenerator cronjob. Does the issue still happen if you disable this cronjob?

GAS85 commented 3 years ago

I wondering that execution duration of this is 0 in oc_jobs grafik

I will disable preview generator for a while to check if it stops.

It could be also Recognize app issue, it took all the cronjob time. grafik

szaimen commented 3 years ago

Yes, please report back if disabling those fixes it. If yes, it is most likely an issue with one of those apps.

GAS85 commented 3 years ago

Do not see spikes after disable of recognize app. I open a linked here ticket for it. Thanks for support!

GAS85 commented 2 years ago

Basically Ticket was not solved and Server part build an issue.

Summery

Recognize app do periodical checks of files via PROPFIND, but it is not root cause for mass file downloads marcelklehr/recognize/issues/116.
Any File Options request e.g. curl -X PROPFIND ... will cause full file download as per https://github.com/nextcloud/server/issues/30262#issuecomment-1010781345
This will cause an issue with Client external Storage (connected via WebDav) Sync, Simple Folder list causing all files download in /tmp folder #30282 and related tickets.

At the end this was correct ticket but I was lost in a different issues by debugging.

ghost commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity and seems to be missing some essential information. It will be closed if no further activity occurs. Thank you for your contributions.

GAS85 commented 2 years ago

Not stale at all...

ghost commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity and seems to be missing some essential information. It will be closed if no further activity occurs. Thank you for your contributions.

GAS85 commented 2 years ago

Please come back.

funktionierbar commented 2 years ago

I have the same problem. I had to unhook all external storage, else the login process for one particular user that had a lot of external files in a syndicated nextcloud was taking up to 15minutes, sometimes failing

nextcloud-command commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity and seems to be missing some essential information. It will be closed if no further activity occurs. Thank you for your contributions.

GAS85 commented 2 years ago

Bad bot 😃

nextcloud-command commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity and seems to be missing some essential information. It will be closed if no further activity occurs. Thank you for your contributions.

GAS85 commented 2 years ago

Still valid

nextcloud-command commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity and seems to be missing some essential information. It will be closed if no further activity occurs. Thank you for your contributions.

GAS85 commented 2 years ago

Still valid >D

nextcloud-command commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity and seems to be missing some essential information. It will be closed if no further activity occurs. Thank you for your contributions.

GAS85 commented 2 years ago

Still here..

szaimen commented 1 year ago

Hi, please update to 24.0.9 or better 25.0.3 and report back if it fixes the issue. Thank you!

My goal is to add a label like e.g. 25-feedback to this ticket of an up-to-date major Nextcloud version where the bug could be reproduced. However this is not going to work without your help. So thanks for all your effort!

If you don't manage to reproduce the issue in time and the issue gets closed but you can reproduce the issue afterwards, feel free to create a new bug report with up-to-date information by following this link: https://github.com/nextcloud/server/issues/new?assignees=&labels=bug%2C0.+Needs+triage&template=BUG_REPORT.yml&title=%5BBug%5D%3A+

GAS85 commented 1 year ago

Still present in 24.0.9

szaimen commented 1 year ago

Hi, please update to 25.0.7 or better 26.0.2 and report back if it fixes the issue. Thank you!

My goal is to add a label like e.g. 26-feedback to this ticket of an up-to-date major Nextcloud version where the bug could be reproduced. However this is not going to work without your help. So thanks for all your effort!

If you don't manage to reproduce the issue in time and the issue gets closed but you can reproduce the issue afterwards, feel free to create a new bug report with up-to-date information by following this link: https://github.com/nextcloud/server/issues/new?assignees=&labels=bug%2C0.+Needs+triage&template=BUG_REPORT.yml&title=%5BBug%5D%3A+

nextcloud / server