nextcloud / server

☁️ Nextcloud server, a safe home for all your data
https://nextcloud.com
GNU Affero General Public License v3.0
26.28k stars 3.96k forks source link

Filenames with special characters (German "Umlaute") not found by "files:scan" #30585

Open Ra72xx opened 2 years ago

Ra72xx commented 2 years ago

Steps to reproduce

  1. Create a file containing special characters (e.g. test-aöü-test)
  2. Sync file to Nextcloud instance (by web ui, Linux client, Gnome file manager and integrated Nextcloud client)
  3. occ files:scan will say "stat: der Aufruf von statx für '/xxxx/test--test' ist fehlgeschlagen: Datei oder Verzeichnis nicht gefunden". (call of statx has failed, file not found).

Expected behaviour

File should be found whatever characters the filename contains.

Actual behaviour

The file is not found by the file scan. Note that the filename as shown by the command output is missing all non-ASCII(?) characters and therefore obviously wrong. Also: The file seems to be synced correctly (at least I hope so)! All clients and the web ui handle the file completely normally. So the error is not really obvious as long as you are not running the scan manually. occ files:cleanup does not remove or notice this "missing" file.

Server configuration

Operating system: Raspian OS, current version (32bit with 64bit kernel). Locale set to UTF8.de-DE

Web server: Apache 2.4 with PHP-FPM

Database: MariaDB 10.3 Locale of the tables set to utf8mb4_bin

PHP version: 8.0 Locale seems IMHO to be set correctly both for cli and fpm.

Nextcloud version: (see Nextcloud admin page) 22.2.3

Updated from an older Nextcloud/ownCloud or fresh install: Long-running instance, migrated from ownCloud years ago.

Where did you install Nextcloud from: tar package

Signing status:

Signing status ``` No errors have been found. ```

List of activated apps:

App list ``` Enabled: - accessibility: 1.8.0 - activity: 2.15.0 - bookmarks: 10.0.3 - bruteforcesettings: 2.3.0 - calendar: 3.0.4 - circles: 22.1.1 - cloud_federation_api: 1.5.0 - comments: 1.12.0 - contacts: 4.0.7 - dav: 1.19.0 - federatedfilesharing: 1.12.0 - files: 1.17.0 - files_external: 1.13.0 - files_fulltextsearch: 22.0.1 - files_pdfviewer: 2.3.1 - files_retention: 1.11.1 - files_rightclick: 1.1.0 - files_sharing: 1.14.0 - files_trashbin: 1.12.0 - files_versions: 1.15.0 - firstrunwizard: 2.11.0 - fulltextsearch: 22.0.1 - fulltextsearch_elasticsearch: 22.0.1 - gpxpod: 4.3.0 - impersonate: 1.9.0 - logreader: 2.7.0 - lookup_server_connector: 1.10.0 - maps: 0.1.10 - metadata: 0.15.0 - nextcloud_announcements: 1.11.0 - notes: 4.3.0 - notifications: 2.10.1 - oauth2: 1.10.0 - password_policy: 1.12.0 - passwords: 2021.12.20 - privacy: 1.6.0 - provisioning_api: 1.12.0 - qownnotesapi: 21.12.0 - richdocuments: 4.2.3 - serverinfo: 1.12.0 - settings: 1.4.0 - sharebymail: 1.12.0 - systemtags: 1.12.0 - tasks: 0.14.2 - text: 3.3.0 - theming: 1.13.0 - twofactor_backupcodes: 1.11.0 - twofactor_nextcloud_notification: 3.3.1 - twofactor_totp: 6.2.0 - unroundedcorners: 1.0.3 - updatenotification: 1.12.0 - user_ldap: 1.12.1 - viewer: 1.6.0 - workflowengine: 2.4.0 Disabled: - admin_audit - collectives - contactsinteraction - dashboard - encryption - federation - files_markdown - files_texteditor - files_videoplayer - photos - recommendations - support - survey_client - user_status - weather_status ```

Nextcloud configuration:

Config report ``` { "system": { "passwordsalt": "***REMOVED SENSITIVE VALUE***", "secret": "***REMOVED SENSITIVE VALUE***", "trusted_domains": [ "localhost", "mydomain", "mydomain2" ], "datadirectory": "***REMOVED SENSITIVE VALUE***", "dbtype": "mysql", "version": "22.2.3.0", "overwrite.cli.url": "https:\/\/mydomain\/nextcloud", "overwritehost": "mydomain:8443", "overwritewebroot": "\/nextcloud", "htaccess.RewriteBase": "\/nextcloud", "dbname": "***REMOVED SENSITIVE VALUE***", "dbhost": "***REMOVED SENSITIVE VALUE***", "dbport": "", "dbtableprefix": "oc_", "dbuser": "***REMOVED SENSITIVE VALUE***", "dbpassword": "***REMOVED SENSITIVE VALUE***", "installed": true, "instanceid": "***REMOVED SENSITIVE VALUE***", "memcache.local": "\\OC\\Memcache\\APCu", "ldapIgnoreNamingRules": false, "ldapProviderFactory": "OCA\\User_LDAP\\LDAPProviderFactory", "default_language": "de", "default_locale": "de_DE", "default_phone_region": "DE", "defaultapp": "files", "mysql.utf8mb4": true, "maintenance": false, "theme": "", "loglevel": 3, "mail_smtpmode": "sendmail", "mail_smtpsecure": "tls", "mail_sendmailmode": "smtp", "mail_smtpauth": 1, "mail_smtpauthtype": "LOGIN", "mail_smtphost": "***REMOVED SENSITIVE VALUE***", "mail_smtpname": "***REMOVED SENSITIVE VALUE***", "mail_smtppassword": "***REMOVED SENSITIVE VALUE***", "updater.release.channel": "stable", "has_rebuilt_cache": true, "trashbin_retention_obligation": "auto, 30", "versions_retention_obligation": "auto, 60", "session_keepalive": false, "auto_logout": false, "session_lifetime": 604800, "remember_login_cookie_lifetime": 63072000, "app_install_overwrite": [ "ocdownloader", "twofactor_admin", "bruteforcesettings", "mindmap_app" ], "data-fingerprint": "830f24f7d337cd03c421a52196fcd5e8" } } ```

Are you using external storage, if yes which one: local/smb/sftp/... local, webdav Are you using encryption: yes/no no Are you using an external user-backend, if yes which one: LDAP/ActiveDirectory/Webdav/... ldap

LDAP configuration (delete this part if not used)

LDAP config ``` +-------------------------------+---------------------------------------------+ | Configuration | s01 | +-------------------------------+---------------------------------------------+ | hasMemberOfFilterSupport | | | homeFolderNamingRule | | | lastJpegPhotoLookup | 0 | | ldapAgentName | | | ldapAgentPassword | *** | | ldapAttributesForGroupSearch | | | ldapAttributesForUserSearch | | | ldapBackupHost | | | ldapBackupPort | | | ldapBase | dc=domain,dc=de | | ldapBaseGroups | ou=groups,dc=domain,dc=de | | ldapBaseUsers | ou=people,dc=domain,dc=de | | ldapCacheTTL | 7200 | | ldapConfigurationActive | 1 | | ldapDefaultPPolicyDN | | | ldapDynamicGroupMemberURL | | | ldapEmailAttribute | mail | | ldapExperiencedAdmin | 0 | | ldapExpertUUIDGroupAttr | | | ldapExpertUUIDUserAttr | | | ldapExpertUsernameAttr | | | ldapExtStorageHomeAttribute | | | ldapGidNumber | gidNumber | | ldapGroupDisplayName | cn | | ldapGroupFilter | (&(|(objectclass=groupOfNames))) | | ldapGroupFilterGroups | | | ldapGroupFilterMode | 0 | | ldapGroupFilterObjectclass | groupOfNames | | ldapGroupMemberAssocAttr | member | | ldapHost | localhost | | ldapIgnoreNamingRules | | | ldapLoginFilter | (&(|(objectclass=inetOrgPerson))(uid=%uid)) | | ldapLoginFilterAttributes | | | ldapLoginFilterEmail | 0 | | ldapLoginFilterMode | 0 | | ldapLoginFilterUsername | 1 | | ldapMatchingRuleInChainState | unknown | | ldapNestedGroups | 0 | | ldapOverrideMainServer | | | ldapPagingSize | 500 | | ldapPort | 389 | | ldapQuotaAttribute | | | ldapQuotaDefault | | | ldapTLS | 0 | | ldapUserAvatarRule | default | | ldapUserDisplayName | displayname | | ldapUserDisplayName2 | | | ldapUserFilter | (|(objectclass=inetOrgPerson)) | | ldapUserFilterGroups | | | ldapUserFilterMode | 0 | | ldapUserFilterObjectclass | inetOrgPerson | | ldapUuidGroupAttribute | auto | | ldapUuidUserAttribute | auto | | turnOffCertCheck | 0 | | turnOnPasswordChange | 0 | | useMemberOfToDetectMembership | 1 | +-------------------------------+---------------------------------------------+ ```

Client configuration

Browser: Brave, Firefox in current versions Nextcloud client 3.4.1 and 3.3.6 on Linux Current client on Android ...

Operating system: Linux, Android

Logs

Web server error log

Web server error log ``` No errors ```

Nextcloud log (data/nextcloud.log)

Nextcloud log ``` No errors in the log during occ files:scan ```

Browser log

Browser log ``` IMHO not relevant ```
Ra72xx commented 2 years ago

Here a SQL dump of oc_filecache for some missing entry:

`-- phpMyAdmin SQL Dump -- version 5.0.4deb2 -- https://www.phpmyadmin.net/

-- Host: localhost:3306 -- Erstellungszeit: 11. Jan 2022 um 05:54 -- Server-Version: 10.3.31-MariaDB-0+deb10u1 -- PHP-Version: 8.0.14

SET SQL_MODE = "NO_AUTO_VALUE_ON_ZERO"; START TRANSACTION; SET time_zone = "+00:00";

/!40101 SET @OLD_CHARACTER_SET_CLIENT=@@CHARACTER_SET_CLIENT /; /!40101 SET @OLD_CHARACTER_SET_RESULTS=@@CHARACTER_SET_RESULTS /; /!40101 SET @OLD_COLLATION_CONNECTION=@@COLLATION_CONNECTION /; /!40101 SET NAMES utf8mb4 /;

-- -- Datenbank: nextcloud


-- -- Tabellenstruktur für Tabelle oc_filecache

CREATE TABLE oc_filecache ( fileid bigint(20) NOT NULL, storage bigint(20) NOT NULL DEFAULT 0, path varchar(4000) COLLATE utf8mb4_bin DEFAULT NULL, path_hash varchar(32) COLLATE utf8mb4_bin NOT NULL DEFAULT '', parent bigint(20) NOT NULL DEFAULT 0, name varchar(250) COLLATE utf8mb4_bin DEFAULT NULL, mimetype bigint(20) NOT NULL DEFAULT 0, mimepart bigint(20) NOT NULL DEFAULT 0, size bigint(20) NOT NULL DEFAULT 0, mtime bigint(20) NOT NULL DEFAULT 0, storage_mtime bigint(20) NOT NULL DEFAULT 0, encrypted int(11) NOT NULL DEFAULT 0, unencrypted_size bigint(20) NOT NULL DEFAULT 0, etag varchar(40) COLLATE utf8mb4_bin DEFAULT NULL, permissions int(11) DEFAULT 0, checksum varchar(255) COLLATE utf8mb4_bin DEFAULT NULL ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin ROW_FORMAT=COMPRESSED;

-- -- Daten für Tabelle oc_filecache

INSERT INTO oc_filecache (fileid, storage, path, path_hash, parent, name, mimetype, mimepart, size, mtime, storage_mtime, encrypted, unencrypted_size, etag, permissions, checksum) VALUES (10206292, 3, 'files/Dropbox/öödödö.xlsx', '5e0bcdcefc394d5795cde22225178885', 10200978, 'öödödö.xlsx', 97, 6, 5439, 1620699860, 1620699860, 0, 0, '9f92dd020e09025036873de12a916b61', 27, 'SHA1:dc76494488f2593e1d99bb52f0b7f573f665d715');

-- -- Indizes der exportierten Tabellen

-- -- Indizes für die Tabelle oc_filecache

ALTER TABLE oc_filecache ADD PRIMARY KEY (fileid), ADD UNIQUE KEY fs_storage_path_hash (storage,path_hash), ADD KEY fs_parent_name_hash (parent,name), ADD KEY fs_storage_mimetype (storage,mimetype), ADD KEY fs_storage_mimepart (storage,mimepart), ADD KEY fs_storage_size (storage,size,fileid), ADD KEY fs_mtime (mtime), ADD KEY fs_size (size);

-- -- AUTO_INCREMENT für exportierte Tabellen

-- -- AUTO_INCREMENT für Tabelle oc_filecache

ALTER TABLE oc_filecache MODIFY fileid bigint(20) NOT NULL AUTO_INCREMENT, AUTO_INCREMENT=10221848; COMMIT;

/!40101 SET CHARACTER_SET_CLIENT=@OLD_CHARACTER_SET_CLIENT /; /!40101 SET CHARACTER_SET_RESULTS=@OLD_CHARACTER_SET_RESULTS /; /!40101 SET COLLATION_CONNECTION=@OLD_COLLATION_CONNECTION /;`

solracsf commented 2 years ago

Similar to https://github.com/nextcloud/server/issues/7762

szaimen commented 1 year ago

Hi, please update to 24.0.9 or better 25.0.3 and report back if it fixes the issue. Thank you!

My goal is to add a label like e.g. 25-feedback to this ticket of an up-to-date major Nextcloud version where the bug could be reproduced. However this is not going to work without your help. So thanks for all your effort!

If you don't manage to reproduce the issue in time and the issue gets closed but you can reproduce the issue afterwards, feel free to create a new bug report with up-to-date information by following this link: https://github.com/nextcloud/server/issues/new?assignees=&labels=bug%2C0.+Needs+triage&template=BUG_REPORT.yml&title=%5BBug%5D%3A+

kimjohans commented 1 year ago

Same problem with german Umlaute here in NC 25.0.4 (up-to-date stable):

Using nextcloudcmd to sync folders locally by script. One folder is named "Büro" (german for "Office"), but nextcloudcmd is not able to find that folder. The Windows & Linux UI version can select that folder and is syncing without problems; only nextcloudcmd is struggeling:

call: nextcloudcmd -u buero -p *** --non-interactive --path "\Büro" c:\destpath https://x.y.de

No change, if the path is given with out without quotation marks

error:

02-25 11:23:38:883 [ info nextcloud.sync.networkjob ]: OCC::LsColJob created for "https://x.y.de" + "/B├╝ro/" "OCC::DiscoverySingleDirectoryJob" 02-25 11:23:38:883 [ info nextcloud.sync.engine ]: shouldDiscoverLocaly "" true 02-25 11:23:38:914 [ warning nextcloud.sync.networkjob ]: QNetworkReply::ContentNotFoundError "Server replied \"404 Not Found\" to \"PROPFIND https://x.y.de/remote.php/dav/files/buero/B├╝ro/\"" QVariant(int, 404) 02-25 11:23:38:914 [ info nextcloud.sync.networkjob.lscol ]: LSCOL of QUrl("https://x.y.de/remote.php/dav/files/buero/B├╝ro/") FINISHED WITH STATUS "ContentNotFoundError Server replied \"404 Not Found\" to \"PROPFIND https://x.y.de/remote.php/dav/files/buero/B├╝ro/\"" 02-25 11:23:38:914 [ warning nextcloud.sync.discovery ]: LSCOL job error "Error transferring https://x.y.de/remote.php/dav/files/buero/B├╝ro/ - server replied: Not Found" 404 QNetworkReply::ContentNotFoundError 02-25 11:23:38:914 [ warning nextcloud.sync.discovery ]: Server error in directory "" 404 02-25 11:23:38:914 [ warning default ]: Sync error: "Server replied with an error while reading directory \"\" : Error transferring https://x.y.de/remote.php/dav/files/buero/B├╝ro/ - server replied: Not Found" 02-25 11:23:38:914 [ info nextcloud.sync.engine ]: Sync run took 32 ms

FYI: User named "buero" - folder within that user "buero" is named "Büro"

vegaz337 commented 1 year ago

any update on this ? Upload through browser does work and scanning some folders work too but files won't be scanned

Ra72xx commented 1 year ago

I haven't had any issue like this any more for months. However, much has changed in the meantime (NC updates, switch from MariaDB to PostgreSQL, updates of the underlying OS), so I cannot really say what has helped in my case.

jandusek4 commented 11 months ago

I have a similar issue with Czech characters. Neither folders nor files containing Czech characters are visible/syncs from Windows nextcloud client. Such characters include (ě,č,ž,...). Folder names and file names with these characters can be created in web UI without a problem. And again are not synced with windows client. I was trying to move from owncloud to next cloud but if this issue is not resolved I would have to consider going back :( Using containerized version nextcloud:27-apache, postgres:15-alpine, redis:6-alpine

Update (upgraded from 25 over 26 to 27 and problem persists)