xbel file corruption preventing sync

[Simplenuity]

Which version of floccus are you using?

5.2.4

How many bookmarks do you have, roughly?

about 1.5k

Are you using other means to sync bookmarks in parallel to floccus?

No

Sync method

WebDAV

Which browser are you using? In case you are using the phone App, specify the Android or iOS version and device please.

Firefox 113.0.2 (64-bit), Google Chrome 126.0.6478.182 (Official Build) (64-bit), Opera (outdated version) 92.0.4561.43

Which version of Nextcloud Bookmarks are you using? (if relevant)

n/a

Which version of Nextcloud? (if relevant)

28.0.5.1

What kind of WebDAV server are you using? (if relevant)

Nextcloud instance offered by Webo.hosting

Describe the Bug

After some undefined time (can be some days or weeks) configured auto-sync stops working with error:

"Cannot read properties of undefined (reading 'tagName')"

It turns out that this is caused by file corruption of the synced xbel file. The content of the file gets randomly cut off at some stage, i.e. part of it is missing at the end of the file.

Only one browser is configured to always upload local changes, all other browsers are configured to only download while discarding local changes.

The installed Nextcloud client is working fine. All files are kept in sync, including the corrupt xbel file. No other file has been corrupted therefore I assume the corruption might have to do with the floccus upload process? Is there a way to influence the way floccus uploads the xbel file? For example is it possible to only replace an older file online after the local xbel file has completely and successfully been uploaded? Or is the upload incremental anyway?

Expected Behavior

The integrity of the xbel file is secured no matter whether it's completely replaced by a new file version or the file online gets incrementally updated.

To Reproduce

Unfortunately no steps to reproduce available. File corruption seems to occur randomly.

Debug log provided

• [ ] I have provided a debug log file

[github-actions[bot]]

Hello :wave:

Thank you for taking the time to open this issue with floccus. I know it's frustrating when software causes problems. You have made the right choice to come here and open an issue to make sure your problem gets looked at and if possible solved. I'm Marcel and I created floccus and have been maintaining it ever since. I currently work for Nextcloud which leaves me with less time for side projects like this one than I used to have. I still try to answer all issues and if possible fix all bugs here, but it sometimes takes a while until I get to it. Until then, please be patient. Note also that GitHub is a place where people meet to make software better together. Nobody here is under any obligation to help you, solve your problems or deliver on any expectations or demands you may have, but if enough people come together we can collaborate to make this software better. For everyone. Thus, if you can, you could also have a look at other issues to see whether you can help other people with your knowledge and experience. If you have coding experience it would also be awesome if you could step up to dive into the code and try to fix the odd bug yourself. Everyone will be thankful for extra helping hands! One last word: If you feel, at any point, like you need to vent, this is not the place for it; you can go to the forum, to twitter or somewhere else. But this is a technical issue tracker, so please make sure to focus on the tech and keep your opinions to yourself.

I look forward to working with you on this issue Cheers :blue_heart:

[marcelklehr]

Hey @Simplenuity Thank you for the feedback!

The installed Nextcloud client is working fine. All files are kept in sync, including the corrupt xbel file. No other file has been corrupted therefore I assume the corruption might have to do with the floccus upload process?

That seems likely, yes. What is the rough size of the full, non-corrupted file? It may be too big for the server. The nextcloud client uses some nextcloud specific hacks to circumvent such limitations.

Is there a way to influence the way floccus uploads the xbel file? For example is it possible to only replace an older file online after the local xbel file has been completely and successfully been uploaded? Or is the upload incremental anyway?

I'm afraid we'd be running into other race conditions then: What if the client machine goes offline before the file was moved in place?

I think the best course of action is to talk to your hoster, if that is possible.

[Simplenuity]

Hi @marcelklehr, Thanks for getting back to me.

The file size is currently about 270 kB.

Of course I can open a ticket with them. For doing so and my own understanding, could you please let me know how the update process works? Does the sync process trigger off the replacement of the online xbel file with any local change? Or is the online file incrementally updated only with the changed parts? Any further information on Floccus that might be relevant for them for troubleshooting would of course be appreciated.

[marcelklehr]

The file size is currently about 270 kB.

Mh, that size doesn't seem very large, but that is after the corruption, right?

could you please let me know how the update process works?

So, floccus just does a PUT request on the file and uploads the new contents in the HTTP body. That's standard webdav. No incremental uploads like the Nextcloud Desktop client does.

[Simplenuity]

Mh, that size doesn't seem very large, but that is after the corruption, right?

No, that's the intact file. Depending on how much gets cut of the corrupted file is proportionally smaller.

Thanks for the additional info. Let's see what I'll get from the hoster.

[marcelklehr]

Cool, keep me posted :)

[Simplenuity]

Hey @marcelklehr

here the compiled summary of the communication with the hoster:

In logs we can only see many PUT actions on file and it seems that file is updated and opened many times. Perhaps the file got corrupted during many updates. How and why this happened can maybe be found in Nextcloud client logs if they are enabled.

We have no information on which application does this, but you should know how you configured Floccus and Nextcloud client applications.

I presume that Floccus update xbel file when there are any new changes, like adding or changing the bookmark. Nextcloud client should also update file on server when the file is changed on your disk, but it seems that the issue happened at this point and Nextcloud client didn't recognize the change.

To my understanding the last statement is not correct as Floccus is only accessing the xbel file residing online on the server - please correct me if I'm wrong.

Here some observations/hypotheses/ideas:

• It seems like the file corruption is occurring when their is a network degradation. No proof/reproducable steps available though.

• A complete network interruption doesn't seem to be the problem as in this (manually initiated) case the Floocus error message indicates a network error. This error never showed up when the file got corrupted.

• From my memory I have not experienced the file corruption whithout the Nextcloud client running, i.e. syncing also the xbel file from the server to a local folder. However I couldn't identify any clear error message in the log files in connection with file corruption, but will check more timely in the future.

• I'd like to make the following feature request: add an additional sync option: a local file. This would have at least the following advantages:

  • Floccus could be also used without any online cloud service available, for example to locally sync the bookmarks (in form of the xbel or html file) across different browsers and browser profiles.
  • The user could use any kind of network sync option they have available to sync between devices.
  • IF at all the occurring file corruption is caused by simultaneously accessing the Nextcloud server via Floccus and the Nextcloud client (maybe via a degraded network connection), then the corruption could hopefully be avoided by using the Nextcloud client for syncing while using Floccus only to update the local file(?).

Thanks for taking the time to look into this issue and for your consideration.

[Simplenuity]

Hi @marcelklehr Please allow me to follow up on my last post. If you prefer a separate ticket for the feature suggestion, please let me know.

[marcelklehr]

Ah, thanks for the follow-up @Simplenuity! Your message got lost in my inbox. So, the hoster wasn't too helpful it seems :/

From my memory I have not experienced the file corruption whithout the Nextcloud client running, i.e. syncing also the xbel file from the server to a local folder. However I couldn't identify any clear error message in the log files in connection with file corruption, but will check more timely in the future.

Perhaps you could confine floccus sync to a folder that is not synced via the nextcloud client?

add an additional sync option: a local file

Sigh. Yes, that would be a superb feature. It's sadly not possible to access the file system directly from a web extension. You can try using LoFloccus instead which is basically a local WebDAV server, allowing you to sync floccus to a local file. But a direct floccus to filesystem connection is not possible sadly :(

[Simplenuity]

Hey @marcelklehr Thanks for getting back to me.

The sync to the desktop was used for offline backup reasons. But at least for troubleshooting I temporarily can find another solution and exclude the folder.

I certainly will check out LoFloccus. Thanks for the suggestion.

Please feel free to close the issue if you want it out of the way. In case I'll gain any new insights I'll report back.

Thanks again for all the effort and this very valuable extension!