search

nextcloud / bookmarks

🔖 Bookmark app for Nextcloud
https://apps.nextcloud.com/apps/bookmarks
GNU Affero General Public License v3.0
999 stars 173 forks source link

floccus unable to synchronize bookmarks: "Malformed UTF-8 characters, possibly incorrectly encoded" #2203

Closed 2GetApp closed 5 days ago

2GetApp commented 1 month ago

Describe the bug floccus was not able to synchronize bookmarks (got http 500 error). The server log shows: "Malformed UTF-8 characters, possibly incorrectly encoded". This happened after I upgraded bookmarks app to 14.2.3. Folder names are redacted from the provided log bellow. I have folder names with special characters (space and ampersand &), as well as Japanese and Chinese characters, but everything worked fine before upgrading.

To Reproduce Steps to reproduce the behavior:

  1. Upgrade bookmarks app to the recent release (14.2.3)
  2. Open browser extension floccus and click "Sync now"
  3. See error

Expected behavior Bookmarks are supposed to be synchronized successfully.

Screenshots If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

Server (please complete the following information):

Additional context Add any other context about the problem here.

Web server error log

[05/Aug/2024:05:26:59 +0000] **REDACTED** **REDACTED** **REDACTED** "GET /index.php/apps/bookmarks/public/rest/v2/folder/-1/children?layers=-1 HTTP/2.0" 500 2454 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36"

Nextcloud log (nextcloud/data/nextcloud.log)

{
    "reqId": "QNFyCr0d1CPodMaqDIQQ",
    "level": 3,
    "time": "2024-08-05T05:26:55+00:00",
    "remoteAddr": "**REDACTED**",
    "user": "**REDACTED**",
    "app": "index",
    "method": "GET",
    "url": "/index.php/apps/bookmarks/public/rest/v2/folder/-1/children?layers=-1",
    "message": "Malformed UTF-8 characters, possibly incorrectly encoded",
    "userAgent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36",
    "version": "29.0.4.1",
    "exception": {
        "Exception": "JsonException",
        "Message": "Malformed UTF-8 characters, possibly incorrectly encoded",
        "Code": 5,
        "Trace": [
            {
                "file": "/var/www/html/lib/public/AppFramework/Http/JSONResponse.php",
                "line": 71,
                "function": "json_encode",
                "args": [
                    [
                        "success",
                        [
                           "**REDACTED**"
                        ]
                    ],
                    4194305
                ]
            },
            {
                "file": "/var/www/html/lib/private/AppFramework/Http/Dispatcher.php",
                "line": 183,
                "function": "render",
                "class": "OCP\\AppFramework\\Http\\JSONResponse",
                "type": "->",
                "args": []
            },
            {
                "file": "/var/www/html/lib/private/AppFramework/App.php",
                "line": 184,
                "function": "dispatch",
                "class": "OC\\AppFramework\\Http\\Dispatcher",
                "type": "->",
                "args": [
                    [
                        "OCA\\Bookmarks\\Controller\\FoldersController"
                    ],
                    "getFolderChildren"
                ]
            },
            {
                "file": "/var/www/html/lib/private/Route/Router.php",
                "line": 338,
                "function": "main",
                "class": "OC\\AppFramework\\App",
                "type": "::",
                "args": [
                    "OCA\\Bookmarks\\Controller\\FoldersController",
                    "getFolderChildren",
                    [
                        "OC\\AppFramework\\DependencyInjection\\DIContainer"
                    ],
                    [
                        "-1",
                        "bookmarks.folders.get_folder_children"
                    ]
                ]
            },
            {
                "file": "/var/www/html/lib/base.php",
                "line": 1050,
                "function": "match",
                "class": "OC\\Route\\Router",
                "type": "->",
                "args": [
                    "/apps/bookmarks/public/rest/v2/folder/-1/children"
                ]
            },
            {
                "file": "/var/www/html/index.php",
                "line": 49,
                "function": "handleRequest",
                "class": "OC",
                "type": "::",
                "args": []
            }
        ],
        "File": "/var/www/html/lib/public/AppFramework/Http/JSONResponse.php",
        "Line": 71,
        "message": "Malformed UTF-8 characters, possibly incorrectly encoded",
        "exception": {},
        "CustomMessage": "Malformed UTF-8 characters, possibly incorrectly encoded"
    }
}

Browser log

2024-08-05T05:26:37.735Z Starting sync process for account redacted@redacted.domain
2024-08-05T05:26:37.739Z QUEUING POST https://redacted.domain/index.php/apps/bookmarks/public/rest/v2/lock
2024-08-05T05:26:37.739Z FETCHING POST https://redacted.domain/index.php/apps/bookmarks/public/rest/v2/lock
2024-08-05T05:26:43.958Z Receiving response for POST https://redacted.domain/index.php/apps/bookmarks/public/rest/v2/lock
2024-08-05T05:26:43.958Z Using "merge default" strategy (no cache available)
2024-08-05T05:26:43.959Z Retrieving local tree
2024-08-05T05:26:43.959Z Retrieving server tree
2024-08-05T05:26:43.959Z QUEUING GET https://redacted.domain/index.php/apps/bookmarks/public/rest/v2/bookmark?page=1&limit=1
2024-08-05T05:26:43.959Z FETCHING GET https://redacted.domain/index.php/apps/bookmarks/public/rest/v2/bookmark?page=1&limit=1
2024-08-05T05:26:50.053Z Receiving response for GET https://redacted.domain/index.php/apps/bookmarks/public/rest/v2/bookmark?page=1&limit=1
2024-08-05T05:26:50.053Z QUEUING GET https://redacted.domain/index.php/apps/bookmarks/public/rest/v2/folder/-1/hash
2024-08-05T05:26:50.053Z FETCHING GET https://redacted.domain/index.php/apps/bookmarks/public/rest/v2/folder/-1/hash
2024-08-05T05:26:51.081Z Receiving response for GET https://redacted.domain/index.php/apps/bookmarks/public/rest/v2/folder/-1/hash
2024-08-05T05:26:51.081Z Filtering out unaccepted local bookmarks
2024-08-05T05:26:51.081Z Filtering out invalid server bookmarks
2024-08-05T05:26:51.081Z Filtering out duplicate bookmarks
2024-08-05T05:26:51.081Z Loading sparse tree as necessary
2024-08-05T05:26:51.081Z Merge strategy: Load complete tree from server
2024-08-05T05:26:51.081Z QUEUING GET https://redacted.domain/index.php/apps/bookmarks/public/rest/v2/folder/-1/children?layers=-1
2024-08-05T05:26:51.081Z FETCHING GET https://redacted.domain/index.php/apps/bookmarks/public/rest/v2/folder/-1/children?layers=-1
2024-08-05T05:26:59.457Z Receiving response for GET https://redacted.domain/index.php/apps/bookmarks/public/rest/v2/folder/-1/children?layers=-1
2024-08-05T05:26:59.458Z Syncing failed with E019: HTTP status 500. Failed GET request. Check your server configuration and log.
2024-08-05T05:26:59.459Z QUEUING DELETE https://redacted.domain/index.php/apps/bookmarks/public/rest/v2/lock
2024-08-05T05:26:59.459Z FETCHING DELETE https://redacted.domain/index.php/apps/bookmarks/public/rest/v2/lock
2024-08-05T05:27:03.726Z Receiving response for DELETE https://redacted.domain/index.php/apps/bookmarks/public/rest/v2/lock
github-actions[bot] commented 1 month ago

Hello :wave:

Thank you for taking the time to open this issue with the bookmarks app. I know it's frustrating when software causes problems. You have made the right choice to come here and open an issue to make sure your problem gets looked at and if possible solved. I'm Marcel and have been maintaining this software the last few years. I currently work for Nextcloud but maintain this app in my free time, because it is not an official Nextcloud product. My day job at Nextcloud is pretty awesome but sadly leaves me with less time for side projects like this one than I used to have. I still try to answer all issues and if possible fix all bugs here, but it sometimes takes a while until I get to it. Until then, please be patient. Note also that GitHub is a place where people meet to make software better together. Nobody here is under any obligation to help you, solve your problems or deliver on any expectations or demands you may have, but if enough people come together we can collaborate to make this software better. For everyone. Thus, if you can, you could also look at other issues to see whether you can help other people with your knowledge and experience. If you have coding experience it would also be awesome if you could step up to dive into the code and try to fix the odd bug yourself. Everyone will be thankful for extra helping hands! One last word: If you feel, at any point, like you need to vent, this is not the place for it; you can go to the forum, to twitter or somewhere else. But this is a technical issue tracker, so please make sure to focus on the tech and keep your opinions to yourself. (Also see our Code of Conduct. Really.)

I look forward to working with you on this issue Cheers :blue_heart:

marcelklehr commented 1 month ago

mmmh, that's strange. Could you perhaps export your bookmarks in the web UI and send the export to me for reproduction?

2GetApp commented 1 month ago

mmmh, that's strange. Could you perhaps export your bookmarks in the web UI and send the export to me for reproduction?

Sorry to leave out the most important information. My bookmarks contain just too much sensitive information to be shared. I am still trying to find out the relevant bookmark entries that are related to this bug.

After exporting all bookmarks and importing them to a fresh account, synchronization proceeds normally for the new account. So my current guess is that the generated fields (notes, generated previews and fetched html, etc) from some web pages with faulty encoding caused the problem.

marcelklehr commented 1 month ago

After exporting all bookmarks and importing them to a fresh account, synchronization proceeds normally for the new account. So my current guess is that the generated fields (notes, generated previews and fetched html, etc) from some web pages with faulty encoding caused the problem.

That's a good lead!

marcelklehr commented 1 month ago

@2GetApp Could you try making these changes locally in your environment to see if they fix your issue? https://github.com/nextcloud/bookmarks/pull/2204/files

2GetApp commented 1 month ago

@2GetApp Could you try making these changes locally in your environment to see if they fix your issue? https://github.com/nextcloud/bookmarks/pull/2204/files

After some experimentation (edit note with messy contents like 製ä½, move to trash bin and restore) on suspicious folders and bookmarks, my account is able to synchronize now. But I somehow lost the content that caused the problem because I had to do the experimentation on the original account (new account synchronizes normally). Now I am not able to reproduce the bug myself :(

I have to wait for the content to be regenerated in the new account (I don't know how to generate them immediately). I will try the fix if I reproduce the bug.

2GetApp commented 1 month ago

Still not able to reproduce it. I will just give up and upgrade the bookmarks app.🥲

reary1993 commented 1 week ago

I ran into a very similar issue: Desktop:

OS: Debian 12
Browser: Firefox
Version: 115.14.0esr

Server:

OS: Ubuntu 22.04 arm64
HTTP server: apache2
Database: 10.11.8-MariaDB-0ubuntu0.24.04.1
PHP version: 8.3
Nextcloud version: 29.0.5
Bookmarks app version: 14.2.4

The error message I get when I try to sync my bookmarks from my browser to Nextcloud is: DbalException An exception occurred while executing a query: SQLSTATE[22007]: Invalid datetime format: 1366 Incorrect string value: '\xE8\xE2\x80\xA6' for columnnextcloud.oc_bookmarks.titleat row 1 I found out that the the following link is causing the issue: https://github.com/cirosantilli/china-dictatorship#cute-cat-theory-of-digital-activism

I also have another setup of Nextcloud where the only difference is the database being mysql not mariadb. There, the syncing goes without an error.

marcelklehr commented 1 week ago

@reary1993 Could you try making these changes locally in your environment to see if they fix your issue? #2204 (files)

reary1993 commented 6 days ago

@marcelklehr I can confirm that #2204 fixed my issue. Thanks for your work.

marcelklehr commented 5 days ago

Yay!