[Bug]: Files displays broken encoding when ⚰ is added to Markdown file

[code-surfer]

⚠️ This issue respects the following points: ⚠️

• [X] This is a bug, not a question or a configuration/webserver/proxy issue.
• [X] This issue is not already reported on Github OR Nextcloud Community Forum (I've searched it).
• [X] Nextcloud Server is up to date. See Maintenance and Release Schedule for supported versions.
• [X] I agree to follow Nextcloud's Code of Conduct.

Bug description

After upgrading nextcloud to 26.0.5, I noticed that some Markdown files are shown with broken encoding in the editor. They were fine in many nextcloud versions up to 23.0.12. I updated in steps to 26.0.5. I also updated to 27.0.2, but the bug persists.

All files are encoded in UTF-8. I've tracked it down to: when I add a character ⚰ (U+26B0) it breaks. With äöü in the file, it displays correctly. With äöü⚰ in the file, it displays as äöü⚰ The files on the filesystem are fine. It's a problem with the nextcloud files editor.

I've used docker images nextcloud:23-fpm .. nextcloud:27-fpm

Steps to reproduce

1. In Files, create a test.md file
2. Click it to open it in the editor
3. Add test äöü
4. Close the file
5. Open it again: it displays correctly
6. Add ⚰. Still looking fine
7. Close the file
8. Open it again: it displays test äöü ⚰

Expected behavior

The file show display correctly as it did in older nextcloud versions.

Installation method

Community Docker image

Nextcloud Server version

27

Operating system

Debian/Ubuntu

PHP engine version

Other

Web server

Nginx

Database engine version

MariaDB

Is this bug present after an update or on a fresh install?

Upgraded to a MAJOR version (ex. 22 to 23)

Are you using the Nextcloud Server Encryption module?

None

What user-backends are you using?

• [X] Default user-backend (database)
• [ ] LDAP/ Active Directory
• [ ] SSO - SAML
• [ ] Other

Configuration report

{                 
    "system": {                                     
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [     
            "nextcloud.timesink.de",
            "nc.timesink.de"                                 
        ],                                             
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "overwrite.cli.url": "https:\/\/nextcloud.timesink.de",                                                             
        "dbtype": "mysql",                               
        "version": "27.0.2.1", 
        "dbname": "***REMOVED SENSITIVE VALUE***",       
        "dbhost": "***REMOVED SENSITIVE VALUE***",           
        "dbport": "",                       
        "dbtableprefix": "oc_",     
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "logtimezone": "UTC", 
        "instanceid": "***REMOVED SENSITIVE VALUE***",                                                                                                                                                                                            [0/28]
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",   
        "trusted_domains": [                                                                                                
            "nextcloud.timesink.de",
            "nc.timesink.de"           
        ],                             
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "overwrite.cli.url": "https:\/\/nextcloud.timesink.de",                                                             
        "dbtype": "mysql",     
        "version": "27.0.2.1", 
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbport": "",                       
        "dbtableprefix": "oc_",             
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "logtimezone": "UTC",               
        "installed": true,                  
        "maintenance": false,               
        "theme": "",                        
        "loglevel": 2,            
        "mysql.utf8mb4": true,                         
        "memcache.local": "\\OC\\Memcache\\Redis",                                                                                                                                                                                                      
        "memcache.distributed": "\\OC\\Memcache\\Redis",      
        "memcache.locking": "\\OC\\Memcache\\Redis",    
        "redis": {
            "host": "***REMOVED SENSITIVE VALUE***",
            "port": 6379                              
        },                                              
        "mail_smtpmode": "smtp",                  
        "mail_smtpsecure": "ssl",
        "mail_sendmailmode": "smtp",
        "mail_from_address": "***REMOVED SENSITIVE VALUE***",
        "mail_domain": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpauthtype": "PLAIN",                    
        "mail_smtpauth": 1,                                                                                                 
        "mail_smtphost": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpport": "465",
        "mail_smtpname": "***REMOVED SENSITIVE VALUE***",
        "mail_smtppassword": "***REMOVED SENSITIVE VALUE***",
        "updater.release.channel": "stable",
        "default_phone_region": "DE"
    }                                             
}

List of activated Apps

Enabled:         
  - activity: 2.19.0
  - bookmarks: 13.0.1            
  - bruteforcesettings: 2.7.0 
  - calendar: 4.4.4   
  - circles: 27.0.1
  - cloud_federation_api: 1.10.0
  - comments: 1.17.0     
  - contacts: 5.3.2
  - contactsinteraction: 1.8.0
  - cookbook: 0.10.2  
  - dashboard: 7.7.0      
  - dav: 1.27.0            
  - federatedfilesharing: 1.17.0 
  - federation: 1.17.0   
  - files: 1.22.0    
  - files_accesscontrol: 1.17.0
  - files: 1.22.0                                                                                                                                                                                                                                 [0/89]
  - files_accesscontrol: 1.17.0
  - files_markdown: 2.4.1    
  - files_pdfviewer: 2.8.0
  - files_rightclick: 1.6.0
  - files_sharing: 1.19.0     
  - files_texteditor: 2.15.1                      
  - files_trashbin: 1.17.0                              
  - files_versions: 1.20.0                          
  - firstrunwizard: 2.16.0
  - gpoddersync: 3.8.1                              
  - logreader: 2.12.0   
  - lookup_server_connector: 1.15.0
  - nextcloud_announcements: 1.16.0
  - notes: 4.8.1                 
  - notifications: 2.15.0           
  - oauth2: 1.15.1                                           
  - password_policy: 1.17.0                            
  - photos: 2.3.0                    
  - privacy: 1.11.0        
  - provisioning_api: 1.17.0                             
  - recommendations: 1.6.0     
  - related_resources: 1.2.0                             
  - serverinfo: 1.17.0                                       
  - settings: 1.9.0                         
  - sharebymail: 1.17.0             
  - support: 1.10.0
  - survey_client: 1.15.0
  - systemtags: 1.17.0                         
  - tasks: 0.15.0
  - theming: 2.2.0  
  - twofactor_backupcodes: 1.16.0
  - updatenotification: 1.17.0
  - user_status: 1.7.0
  - viewer: 2.1.0  
  - weather_status: 1.7.0       
  - workflowengine: 2.9.0
Disabled:          
  - admin_audit: 1.17.0       
  - encryption: 2.15.0
  - files_external: 1.19.0
  - suspicious_login: 5.0.0
  - text: 3.8.0 (installed 2.0.0)
  - twofactor_totp: 9.0.0
  - user_ldap: 1.17.0

Nextcloud Signing status

No errors have been found.

Nextcloud Logs

No response

Additional info

No response

[juliushaertl]

I cannot reproduce that, but haven't tried the community docker image yet. Any chance you can share a copy of such a file here as an attachment for checking the file encoding directly?

[code-surfer]

The file is fine, I thoroughly checked the encoding. It is also displaying fine in the Nextcloud Android app. The problem is in the dual pane Markdown editor that opens when I click the file in .../index.php/apps/files?dir=/Notizen/Medien&openfile=122800 which gets me to .../apps/files/?dir=/Notizen/Medien&fileid=88758#filestexteditor.

# cat vol/nextcloud/data/rj/files/Notizen/Medien/test.md
test äöü ⚰

# recode utf8..dump < vol/nextcloud/data/rj/files/Notizen/Medien/test.md
UCS2   Mne   Description

0074   t     latin small letter t
0065   e     latin small letter e
0073   s     latin small letter s
0074   t     latin small letter t
0020   SP    space
00E4   a:    latin small letter a with diaeresis
00F6   o:    latin small letter o with diaeresis
00FC   u:    latin small letter u with diaeresis
0020   SP    space
26B0      

Only when I edit the file and the editor saves it, then the encoding gets messed up. I removed the symbol that triggers the bug, but still:

# cat vol/nextcloud/data/rj/files/Notizen/Medien/test.md
test äöü

# recode utf8..dump < vol/nextcloud/data/rj/files/Notizen/Medien/test.md
UCS2   Mne   Description

0074   t     latin small letter t
0065   e     latin small letter e
0073   s     latin small letter s
0074   t     latin small letter t
0020   SP    space
00C3   A?    latin capital letter a with tilde
00A4   Cu    currency sign
00C3   A?    latin capital letter a with tilde
00B6   PI    pilcrow sign
00C3   A?    latin capital letter a with tilde
00BC   14    vulgar fraction one quarter

I restored the file. Here is the base64-encoded content for your diagnostics:

# base64 vol/nextcloud/data/rj/files/Notizen/Medien/test.md
dGVzdCDDpMO2w7wg4pqwCg==

By the way, the bug is not triggered by every character beyond U+00FF. With the € U+20AC it works fine. Same with ✄ U+2704. But with ⚰ U+26B0 it breaks.

Code point	Result
20AC	fine
26A5	broken
26AF	broken
26B0	broken
26B1	broken
26B2	broken
26FF	fine
2704	fine

[juliushaertl]

Ah, you are using a different app then, let me move the issue