search

nextcloud / server

☁️ Nextcloud server, a safe home for all your data
https://nextcloud.com
GNU Affero General Public License v3.0
27.5k stars 4.08k forks source link

[Bug]: NC 30 Web UI does not show all copied files / folders after copying large amount of files and folders but appear to be there via WebDAV #48664

Open anolting opened 1 month ago

anolting commented 1 month ago

⚠️ This issue respects the following points: ⚠️

Bug description

Copying a large amount of files and folders using the web interface leads to a web view presentation where it seems as not all of the files and folders where copied. Only a fraction of the ae presented.

Steps to reproduce

Steps I used to reproduce this issue (at least in my installation). All steps are done via the web interface.

  1. Mount external NC storage using External Storage feature in AIO NC 30
  2. Navigate to folder containing large amounts of files and folders.
  3. Select all and use Copy/Move to copy all to a folder residing on the local NC. The amount is about 35000 files in approximitely 30 folders and 20 GB in size.
  4. Waiting about 24 hours to let the copying process finishing the job.
  5. After the copying process only a fraction of files and folders are shown.
  6. A WebDAV connection to the local AIO NC 30 shows all files and folder and a random check proves them all as accessible. See screen shot below.

The bug exists for a longer time period and I noticed it the first time after I started again using NC in version 24.

Expected behavior

All files and folder are indexed and presented in the web interface.

Nextcloud Server version

30

Operating system

Debian/Ubuntu

PHP engine version

PHP 8.2

Web server

Apache (supported)

Database engine version

PostgreSQL

Is this bug present after an update or on a fresh install?

None

Are you using the Nextcloud Server Encryption module?

None

What user-backends are you using?

Configuration report

{
    "system": {
        "one-click-instance": true,
        "one-click-instance.user-limit": 100,
        "memcache.local": "\\OC\\Memcache\\APCu",
        "apps_paths": [
            {
                "path": "\/var\/www\/html\/apps",
                "url": "\/apps",
                "writable": false
            },
            {
                "path": "\/var\/www\/html\/custom_apps",
                "url": "\/custom_apps",
                "writable": true
            }
        ],
        "appsallowlist": false,
        "check_data_directory_permissions": false,
        "memcache.distributed": "\\OC\\Memcache\\Redis",
        "memcache.locking": "\\OC\\Memcache\\Redis",
        "redis": {
            "host": "***REMOVED SENSITIVE VALUE***",
            "password": "***REMOVED SENSITIVE VALUE***",
            "port": 6379
        },
        "overwritehost": "nextcloud.your.domain.tld",
        "overwriteprotocol": "https",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "localhost",
            "nextcloud.your.domain.tld"
        ],
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "dbtype": "pgsql",
        "version": "29.0.7.1",
        "overwrite.cli.url": "https:\/\/nextcloud.your.domain.tld\/",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbport": "",
        "dbtableprefix": "oc_",
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "maintenance": false,
        "updatedirectory": "\/nc-updater",
        "loglevel": "2",
        "log_type": "file",
        "logfile": "\/var\/www\/html\/data\/nextcloud.log",
        "log_rotate_size": "10485760",
        "log.condition": {
            "apps": [
                "admin_audit"
            ]
        },
        "preview_max_x": "2048",
        "preview_max_y": "2048",
        "jpeg_quality": "60",
        "enabledPreviewProviders": {
            "1": "OC\\Preview\\Image",
            "2": "OC\\Preview\\MarkDown",
            "3": "OC\\Preview\\MP3",
            "4": "OC\\Preview\\TXT",
            "5": "OC\\Preview\\OpenDocument",
            "6": "OC\\Preview\\Movie",
            "7": "OC\\Preview\\Krita",
            "0": "OC\\Preview\\Imaginary"
        },
        "enable_previews": true,
        "upgrade.disable-web": true,
        "mail_smtpmode": "smtp",
        "trashbin_retention_obligation": "auto, 30",
        "versions_retention_obligation": "auto, 30",
        "activity_expire_days": "30",
        "simpleSignUpLink.shown": false,
        "share_folder": "\/Shared",
        "one-click-instance.link": "https:\/\/nextcloud.com\/all-in-one\/",
        "upgrade.cli-upgrade-link": "https:\/\/github.com\/nextcloud\/all-in-one\/discussions\/2726",
        "allow_local_remote_servers": true,
        "davstorage.request_timeout": 3600,
        "htaccess.RewriteBase": "\/",
        "dbpersistent": false,
        "files_external_allow_create_new_local": false,
        "trusted_proxies": "***REMOVED SENSITIVE VALUE***",
        "preview_imaginary_url": "***REMOVED SENSITIVE VALUE***",
        "default_phone_region": "DE",
        "mail_smtpsecure": "ssl",
        "mail_sendmailmode": "smtp",
        "mail_from_address": "***REMOVED SENSITIVE VALUE***",
        "mail_domain": "***REMOVED SENSITIVE VALUE***",
        "mail_smtphost": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpport": "465",
        "mail_smtpauth": 1,
        "mail_smtpname": "***REMOVED SENSITIVE VALUE***",
        "mail_smtppassword": "***REMOVED SENSITIVE VALUE***",
        "maintenance_window_start": 100,
        "preview_imaginary_key": "***REMOVED SENSITIVE VALUE***",
        "auth.bruteforce.protection.enabled": true,
        "ratelimit.protection.enabled": true
    }
}

List of activated Apps

Enabled:
  - activity: 2.21.1
  - admin_audit: 1.19.0
  - analytics: 5.0.1
  - analytics_sourcepack: 0.0.3
  - app_api: 3.2.1
  - appointments: 2.1.12
  - assistant: 1.1.0
  - calendar: 4.7.16
  - circles: 29.0.0-dev
  - cloud_federation_api: 1.12.0
  - collectives: 2.14.4
  - comments: 1.19.0
  - contacts: 6.0.0
  - contactsinteraction: 1.10.0
  - context_chat: 2.2.1
  - dashboard: 7.9.0
  - dav: 1.30.1
  - deck: 1.13.2
  - drawio: 3.0.3
  - federatedfilesharing: 1.19.0
  - federation: 1.19.0
  - files: 2.1.1
  - files_antivirus: 5.5.7
  - files_automatedtagging: 1.19.0
  - files_bpm: 1.5.0
  - files_downloadlimit: 2.0.0
  - files_external: 1.21.0
  - files_fulltextsearch: 29.0.1
  - files_pdfviewer: 2.10.0
  - files_reminders: 1.2.0
  - files_sharing: 1.21.0
  - files_trashbin: 1.19.0
  - files_versions: 1.22.0
  - firstrunwizard: 2.18.0
  - forms: 4.3.1
  - fulltextsearch: 29.0.1
  - fulltextsearch_elasticsearch: 29.0.1
  - integration_collaboard: 1.0.10
  - integration_openstreetmap: 1.0.12
  - logreader: 2.14.0
  - lookup_server_connector: 1.17.0
  - mail: 3.7.8
  - maps: 1.4.0
  - multiboards: 1.0.4
  - nextcloud-aio: 0.6.0
  - nextcloud_announcements: 1.18.0
  - notes: 4.11.0
  - notifications: 2.17.0
  - notify_push: 0.7.0
  - oauth2: 1.17.1
  - password_policy: 1.19.0
  - photos: 2.5.0
  - privacy: 1.13.0
  - provisioning_api: 1.19.0
  - recommendations: 2.1.0
  - related_resources: 1.4.0
  - richdocuments: 8.4.7
  - serverinfo: 1.19.0
  - settings: 1.12.0
  - sharebymail: 1.19.0
  - spreed: 19.0.9
  - support: 1.12.0
  - survey_client: 1.17.0
  - systemtags: 1.19.0
  - tables: 0.8.0
  - tasks: 0.16.1
  - text: 3.10.1
  - theming: 2.4.0
  - twofactor_backupcodes: 1.18.0
  - twofactor_totp: 11.0.0-dev
  - user_status: 1.9.0
  - viewer: 2.3.0
  - weather_status: 1.9.0
  - whiteboard: 1.0.3
  - workflowengine: 2.11.0
  - xwiki: 0.1.2
Disabled:
  - bruteforcesettings: 2.9.0
  - encryption: 2.17.0
  - suspicious_login: 7.0.0
  - user_ldap: 1.20.0

Nextcloud Signing status

No errors have been found.

Nextcloud Logs

This log is flooded with this two different messages:

{"reqId":"L5bKTvW1Q8toW4YjVu0Q","level":2,"time":"2024-10-11T12:08:29+00:00","remoteAddr":"172.31.0.1","user":"admin","app":"no app in context","method":"POST","url":"/","message":"AppManger::checkAppForUser - can't decode group IDs: Failed to construct console command 'OCA\\LdapWriteSupport\\Command\\GroupAdminsToLdap': Could not resolve OCA\\User_LDAP\\Helper! Class \"OCA\\User_LDAP\\Helper\" does not exist\nyes - json error code: 4","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:133.0) Gecko/20100101 Firefox/133.0","version":"29.0.7.1","data":[]}
{"reqId":"L5bKTvW1Q8toW4YjVu0Q","level":2,"time":"2024-10-11T12:08:29+00:00","remoteAddr":"172.31.0.1","user":"admin","app":"no app in context","method":"POST","url":"/","message":"AppManger::checkAppForUser - can't decode group IDs: Failed to construct console command 'OCA\\LdapWriteSupport\\Command\\GroupAdminsToLdap': Could not resolve OCA\\User_LDAP\\Helper! Class \"OCA\\User_LDAP\\Helper\" does not exist\nyes - json error code: 4","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:133.0) Gecko/20100101 Firefox/133.0","version":"29.0.7.1","data":[]}
{"reqId":"L5bKTvW1Q8toW4YjVu0Q","level":2,"time":"2024-10-11T12:08:29+00:00","remoteAddr":"172.31.0.1","user":"admin","app":"no app in context","method":"POST","url":"/","message":"AppManger::checkAppForUser - can't decode group IDs: Failed to construct console command 'OCA\\LdapWriteSupport\\Command\\GroupAdminsToLdap': Could not resolve OCA\\User_LDAP\\Helper! Class \"OCA\\User_LDAP\\Helper\" does not exist\nyes - json error code: 4","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:133.0) Gecko/20100101 Firefox/133.0","version":"29.0.7.1","data":[]}
{"reqId":"L5bKTvW1Q8toW4YjVu0Q","level":2,"time":"2024-10-11T12:08:29+00:00","remoteAddr":"172.31.0.1","user":"admin","app":"no app in context","method":"POST","url":"/","message":"AppManger::checkAppForUser - can't decode group IDs: Failed to construct console command 'OCA\\LdapWriteSupport\\Command\\GroupAdminsToLdap': Could not resolve OCA\\User_LDAP\\Helper! Class \"OCA\\User_LDAP\\Helper\" does not exist\nyes - json error code: 4","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:133.0) Gecko/20100101 Firefox/133.0","version":"29.0.7.1","data":[]}
{"reqId":"zclf2gNgxs2WIdHwDHnk","level":2,"time":"2024-10-11T12:08:29+00:00","remoteAddr":"172.31.0.1","user":"admin","app":"no app in context","method":"GET","url":"/apps/dashboard/","message":"AppManger::checkAppForUser - can't decode group IDs: Failed to construct console command 'OCA\\LdapWriteSupport\\Command\\GroupAdminsToLdap': Could not resolve OCA\\User_LDAP\\Helper! Class \"OCA\\User_LDAP\\Helper\" does not exist\nyes - json error code: 4","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:133.0) Gecko/20100101 Firefox/133.0","version":"29.0.7.1","data":[]}
{"reqId":"zclf2gNgxs2WIdHwDHnk","level":2,"time":"2024-10-11T12:08:29+00:00","remoteAddr":"172.31.0.1","user":"admin","app":"no app in context","method":"GET","url":"/apps/dashboard/","message":"AppManger::checkAppForUser - can't decode group IDs: Failed to construct console command 'OCA\\LdapWriteSupport\\Command\\GroupAdminsToLdap': Could not resolve OCA\\User_LDAP\\Helper! Class \"OCA\\User_LDAP\\Helper\" does not exist\nyes - json error code: 4","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:133.0) Gecko/20100101 Firefox/133.0","version":"29.0.7.1","data":[]}
{"reqId":"zclf2gNgxs2WIdHwDHnk","level":2,"time":"2024-10-11T12:08:29+00:00","remoteAddr":"172.31.0.1","user":"admin","app":"no app in context","method":"GET","url":"/apps/dashboard/","message":"AppManger::checkAppForUser - can't decode group IDs: Failed to construct console command 'OCA\\LdapWriteSupport\\Command\\GroupAdminsToLdap': Could not resolve OCA\\User_LDAP\\Helper! Class \"OCA\\User_LDAP\\Helper\" does not exist\nyes - json error code: 4","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:133.0) Gecko/20100101 Firefox/133.0","version":"29.0.7.1","data":[]}

Additional info

Image

joshtrichards commented 1 month ago

The logs remind of this: nextcloud/all-in-one#4806

joshtrichards commented 1 month ago

Are there any errors in your browser inspector Console tab or Network tab?

anolting commented 1 month ago

The logs remind of this: nextcloud/all-in-one#4806

As of my investigation this is another issue and I will place another issue as soon the reason for this one has been identified.

anolting commented 1 month ago

Are there any errors in your browser inspector Console tab or Network tab?

I hadn't open the browser inspector while copying the files. As I understand the architecture this happen on server and not by the browser.

joshtrichards commented 1 month ago

I hadn't open the browser inspector while copying the files. As I understand the architecture this happen on server and not by the browser.

The copying, yes. If I understood your report correctly though, the issue isn't the copying. It's that the files aren't displayed in the Web client, even though they're actually there based on your WebDAV checks (suggesting the copy was successful).

So I was referring to checking the browser inspect when you open that folder in the Web UI.

anolting commented 1 month ago

successful).

So I was referring to checking the browser inspect when you open that folder in the Web UI

A single digit number of error like this:

A setting with the same name is already registered [Settings.js:41:11](webpack:///nextcloud/apps/files/src/services/Settings.js)
    register Settings.js:41
    render main.js:58
    mounted FilesListHeader.vue:41
    VueJS 21
    updateNodes files.ts:77
    E pinia.mjs:1380
    fetchContent FilesList.vue:321
    dir FilesList.vue:273
    VueJS 11
    init vue-router.esm.js:3005
    init vue-router.esm.js:3004
    updateRoute vue-router.esm.js:2414
    transitionTo vue-router.esm.js:2263
    confirmTransition vue-router.esm.js:2402
    a vue-router.esm.js:2084
    Ee vue-router.esm.js:2095
    confirmTransition vue-router.esm.js:2396
    a vue-router.esm.js:2084
    a vue-router.esm.js:2088
    _ vue-router.esm.js:2384
    g vue-router.esm.js:2162
    _ vue-router.esm.js:2362
    a vue-router.esm.js:2087
    a vue-router.esm.js:2091
    Ee vue-router.esm.js:2095
    confirmTransition vue-router.esm.js:2392
    transitionTo vue-router.esm.js:2260
    push vue-router.esm.js:2606
    push vue-router.esm.js:3036
    push vue-router.esm.js:3035
    push router.ts:11
    goToRoute RouterService.ts:38

and some like this:


Uncaught SyntaxError: JSON.parse: unexpected character at line 1 column 1 of the JSON data
    onmessage index.js:94
    o index.js:86
    onclose index.js:108
    setTimeout handler*2534/o/window._notify_push_ws.onclose index.js:106
    o index.js:103
    onclose index.js:108
    setTimeout handler*2534/o/window._notify_push_ws.onclose index.js:106
    o index.js:103
    onclose index.js:108
    setTimeout handler*2534/o/window._notify_push_ws.onclose index.js:106
    o index.js:103
    onclose index.js:108
    setTimeout handler*2534/o/window._notify_push_ws.onclose index.js:106
    o index.js:103
    onclose index.js:108
    setTimeout handler*2534/o/window._notify_push_ws.onclose index.js:106
    o index.js:103
    onclose index.js:108
    setTimeout handler*2534/o/window._notify_push_ws.onclose index.js:106
    o index.js:103
    onclose index.js:108
    setTimeout handler*2534/o/window._notify_push_ws.onclose index.js:106
    o index.js:103
    onclose index.js:108
[index.js:94:28](webpack:///notifications/node_modules/@nextcloud/notify_push/dist/index.js)
    onmessage index.js:94
    (Async: EventHandlerNonNull)
    o index.js:86
    AsyncFunctionNext self-hosted:800
    (Async: async)
    onclose index.js:108
    (Async: setTimeout handler)
    onclose index.js:106
    (Async: EventHandlerNonNull)
    o index.js:103
    AsyncFunctionNext self-hosted:800
    (Async: async)
    onclose index.js:108
    (Async: setTimeout handler)
    onclose index.js:106
    (Async: EventHandlerNonNull)
    o index.js:103
    AsyncFunctionNext self-hosted:800
    (Async: async)
    onclose index.js:108
    (Async: setTimeout handler)
    onclose index.js:106
    (Async: EventHandlerNonNull)
    o index.js:103
    InterpretGeneratorResume self-hosted:1413
    AsyncFunctionNext self-hosted:800
    (Async: async)
    onclose index.js:108
    (Async: setTimeout handler)
    onclose index.js:106
    (Async: EventHandlerNonNull)
    o index.js:103
    InterpretGeneratorResume self-hosted:1413
    AsyncFunctionNext self-hosted:800
    (Async: async)
    onclose index.js:108
    (Async: setTimeout handler)
    onclose index.js:106
    (Async: EventHandlerNonNull)
    o index.js:103
    InterpretGeneratorResume self-hosted:1413
    AsyncFunctionNext self-hosted:800
    (Async: async)
    onclose index.js:108
    (Async: setTimeout handler)
    onclose index.js:106
    (Async: EventHandlerNonNull)
    o index.js:103
    InterpretGeneratorResume self-hosted:1413
    AsyncFunctionNext self-hosted:800
    (Async: async)
    onclose index.js:108
    (Async: setTimeout handler)
    onclose index.js:106
    (Async: EventHandlerNonNull)
    o index.js:103
    InterpretGeneratorResume self-hosted:1413
    AsyncFunctionNext self-hosted:800
    (Async: async)
    onclose index.js:108
joshtrichards commented 1 month ago

Uncaught SyntaxError: JSON.parse: unexpected character at line 1 column 1 of the JSON data

So it may have something to do with the notify_push app (either that or you have second issue there).

anolting commented 1 month ago

Uncaught SyntaxError: JSON.parse: unexpected character at line 1 column 1 of the JSON data

So it may have something to do with the notify_push app (either that or you have second issue there).

As far I understand the usage of vueJS the errors have no connection to the bug this issue is based on.

IMHO, the vueJS messages, for example, so far only show a presumably missing error handling or indicate incorrect assumptions regarding the presence of certain information.

anolting commented 1 month ago

To make troubleshooting a little more targeted and not to randomly search for problems, I have carried out further tests.

To do this, I copied a section of one of the missing folders (missing in the web interface) again using NC's file management functions and made sure that there were only a few hundred files. These folders and the file can now be seen in the web interface.

So you could now simply copy the files in small steps. This doesn't seem to make much sense to me, it's not a workaround, and the problem seems to be possibly related to the lack of indexing of the copied data.

After reapeating this I noticed (which I have forgotten) the unfortunate chosen behavior to not overwrite an existing but instead add the string (maybe incorrect translated into english) "*_copy" to the end of the file name.