floccusaddon / floccus

:cloud: Sync your bookmarks privately across browsers and devices
https://floccus.org
Mozilla Public License 2.0
5.69k stars 240 forks source link

Problems with never ending sync process #482

Closed Klemens2 closed 3 years ago

Klemens2 commented 4 years ago

Describe the bug Sync-Process isn't stopping. Der Sync-Prozess für einen Nextcloud-Lesezeichenordner rennt tage- bis wochenlang - mit unklarem Ergebnis ob fertig.

To Reproduce Every time I start one of my three Nextcloud-Bookmark-Folders he is running for days/weeks and I'm not sure if it ends sucessfully.

Expected behavior Just successfully and ending synchronisation and not needing days or weeks.

Desktop:

Server:

Descreption of problem (in german) including debug logs Debug log Ich habe mich an folgender Anleitung orientiert: [(https://decatec.de/home-server/nextcloud-lesezeichen-synchronisieren-mit-chrome-firefox/comment-page-1/#comment-27427)]

Ich habe leider auch ein gröberes Problem mit dem Syncen von tausenden Lesezeichen: Ich habe drei Synchronisationen eingerichtet ("Lesezeichen-Menü", "Lesezeichen-Symbolleiste" und "Weitere Lesezeichen" [war auf Firefox so eingerichtet]. Die Synchronisation von Lesezeichen-Menü dauert wortwörtlich tage- bis wochenlang (ist erst einmal durchgelaufen und dann mit einer Fehlermeldung).

Bei der ersten Anlage hat es mir auch Ordner dupliziert. Ich habe dann einfach die doppelten Ordner gelöscht (eine funktionierende Lösung war mir in dem Moment wichtiger als ein etwaiger Datenverlust).

Die Ordner am Server habe ich aber abweichend nicht bspw. "FavBar" genannt sondern so wie auch in Firefox (müsste doch keinen Unterschied ausmachen, oder?)

Wenn ich die Protokoll-Datei richtig verstehe, ist darin stets folgendes zu entnehmen: Der Log fängt wie folgt an:

2020-03-19T07:55:24.645Z (nextcloud-folders)UPDATE - #425;56Lesezeichen-Bezeichnung parentId: 56 2020-03-19T07:55:24.645Z Fetching single bookmark 2020-03-19T07:55:24.730Z (nextcloud-folders)UPDATE - #435;56[Lesezeichen-Bezeichnung parentId: 56 2020-03-19T07:55:24.730Z Fetching single bookmark 2020-03-19T07:55:24.800Z (nextcloud-folders)UPDATE - #431;56Lesezeichen-Bezeichnung parentId: 56 2020-03-19T07:55:24.800Z Fetching single bookmark 2020-03-19T07:55:24.948Z (nextcloud-folders)UPDATE - #429;56Lesezeichen-Bezeichnung parentId: 56 2020-03-19T07:55:24.949Z Fetching single bookmark 2020-03-19T07:55:25.306Z COMPARE { localItem: undefined, cacheItem: - #GRNAQnWmXCLqLesezeichen-Bezeichnung parentId: cJzsVkMfTsUq, serverItem: - #1683;191Lesezeichen-Bezeichnung parentId: 191 } 2020-03-19T07:55:25.306Z (nextcloud-folders)REMOVE { id: '1683;191' } 2020-03-19T07:55:25.306Z New locally: { localChild: - #KaYFDR7XGlK0[Lesezeichen-Bezeichnung](Lesezeichen-URL-Verknüpfung parentId: 5bzxNpfYCjzy } 2020-03-19T07:55:25.541Z COMPARE { localItem: undefined, cacheItem: - #dZk_r8tRvCn4[Lesezeichen-Bezeichnung](Lesezeichen-URL-Verknüpfung) parentId: cJzsVkMfTsUq, serverItem: - #1685;191Lesezeichen-Bezeichnung parentId: 191 } 2020-03-19T07:55:25.542Z (nextcloud-folders)REMOVE { id: '1685;191' } 2020-03-19T07:55:25.542Z New locally: { localChild: - #mc2GsLR8UWRqLesezeichen-Bezeichnung parentId: 5bzxNpfYCjzy } ... ...

und so geht's weiter, bis sich anscheinend gewisse Prozesse zu wiederholen beginnen:

2020-03-19T07:56:36.518Z create branch: This bookmark was moved here from somewhere else in from tree 2020-03-19T07:56:36.986Z Received bookmarks from server [ - #383Lesezeichen-Bezeichnung1 parentId: 51,

...und so wieter, bis es schließlich wie folgt endet:

Bei diesem "Compare"-Vorgang, bevor es wieder neu einliest, steht jedes Mal ein anderes Lesezeichen. Abers stets so eine komische „local items“ bzw. „parent-ID“-Nr. Danach folgt immer so eine Log-Zeile:

Datumsstempel create branch: This bookmark was moved here from somewhere else in from tree.

Später war immer wieder folgende Vorgangsabfolge dokumentiert:

2020-03-29T23:42:38.725Z Skipping subtree 2020-03-29T23:42:38.967Z Overriding title of built-in node unfiled_____ Weitere Lesezeichen => Other Bookmarks

Den Sta“rt eines Sync-Prozesses finde ich erst am Tag nachdem der Log beginnt (20.3.2020):

2020-03-20T07:53:26.819Z Starting sync process for account Benutzer@nextcloud-Domain 2020-03-20T07:53:26.848Z Using normal strategy 2020-03-20T07:53:27.590Z Overriding title of built-in node toolbar_____ Lesezeichen-Symbolleiste => Bookmarks Bar 2020-03-20T07:53:46.594Z Received initial folders from server [] 2020-03-20T07:53:46.619Z Loading sparse tree as necessary 2020-03-20T07:53:46.626Z LOADCHILDREN + #3[undefined] parentId: undefined, hash: 0

  • 2[Weitere Lesezeichen] parentId: 3, hash: undefined

  • 3[Lesezeichen-Symbolleiste] parentId: 3, hash: undefined

  • 1[Lesezeichen-Menü] parentId: 3, hash: undefined

Dann geht’s mit dem Sub-Ordnern etc. weiter bis es schließlich wie folgt endet:***

2020-03-20T07:53:47.202Z COMPARE { localItem:

Dann beginnt noch einmal der Sync-Prozess und arbeitet bis zu den Sync-Ordnern ab, bis dann wieder der Compare-Vorgang beginnt:

2020-03-29T23:42:38.721Z COMPARE { localItem:

…bis es schließlich wie folgt endet (Bin hier um 9 Tage verrutscht, aber bis dahin macht er halt ständig das gleiche) :

   - #2274;2[***Lesezeichen-Bezeichnung***](Lesezeichen-URL-Verknüpfung***) parentId: 2 }

2020-03-29T23:42:38.725Z Skipping subtree 2020-03-29T23:42:38.967Z Overriding title of built-in node unfiled_____ Weitere Lesezeichen => Other Bookmarks 2020-03-29T23:42:38.982Z Successfully ended sync process for account Nextcloud-Benutzer@Nextcloud-Domain

Der Sync-Prozess beginnt dann jeweils am nächsten Tag erneut. Bei floccus wird der eine Sync-Prozess für das Lesezeichen-Menü jedoch als durchgängig bearbeitend angezeigt (während die anderen beiden schnell durchlaufen oder derzeit eben nicht mehr da nicht mehr parallel abarbeitend eingestellt).

Kann mir jemand weiterhelfen?

Danke vielmals im Voraus!

Klemens2 commented 4 years ago

Anbei noch ein Screenshot (Fortschrittsanzeige war aber die meiste Zeit nicht so am Ende wie derzeit sondern ist auch lang eher am Beginn und eher in der Mitte festgehangen). grafik

Klemens2 commented 4 years ago

Nachtrag: Jetzt habe ich bei dem obersten, probleme-verursachenden Konto kurz die Fehlermeldung E027 (wenn ich mich nicht verschaut habe) sehen können. In der Log-Datei finde ich davon nichts. Die anderen Konten zeigen nach wie vor E017 an.

marcelklehr commented 4 years ago

Hi, Depending on how many bookmarks you have it's possible that it simply takes longer to sync them than the sync interval, which means it'll always be syncing. The relatively poor performance with larger collections of bookmarks is a known issue that I'm actively working on. The next version of the bookmarks app will hopefully bring another performance improvement.

Until things settle down, you have basically two options:

  1. Decrease the sync frequency in the account options. Default is 15 minutes.
  2. Use the webdav sync method which is known to be faster. It will only sync to a file in your nextcloud, though, not with the bookmarks app.
Klemens2 commented 4 years ago

Ich bin so frei und antworte deutsch (ich habe erst im Schreiben des urspünglichen gesehen, dass dies möglich ist bzw. beantwortet werden kann).

Als Sync-Frequenz habe ich 1440 Min (= 1x täglich) eingestellt. Für mich ist die Frage, warum dennoch andauernd diese häufigen Wiederholungen innerhalb kürztester Zeit starten - ich vermute, die fressen (unnötig?) viel Ressourcen und deren Sinn hat sich mir noch nicht erschlossen. Ich hätte vermutet, dass da ein Fehler die Ursache ist, aber ich kenne mich zuwenig aus, um das zu beurteilen.

Jedenfalls vielen Dank für die Rückmeldung sowie für die Erstellung und Weiterentwicklung von Floccus!

marcelklehr commented 4 years ago

Mmmmh. The sync interval is not authoritative. That means, if you make changes to your bookmarks it will sync on demand. The purpose of the sync interval is to avoid missing any upstream changes.

wohali commented 4 years ago

I'm having the same issue. I've set my sync interval to 1800. The laptop CPU spikes once the sync starts, and never ends, spinning the fans up. I've had to restart the browser (Firefox, 68.7.0ESR, Linux) to get it to stop.

Forgot to mention: if I cancel the sync, the cancel never cancels. I have to quit FF and restart it. Then, on return, I get the error overlay on the icon.

I can then set the "Trigger sync from scratch" option, and force a sync, and that sync succeeds.

No syncs after that succeed, they all get stuck. The logfile never shows any content other than the one successful sync I forced.

wohali commented 4 years ago

Update: This only happens on my Linux 68.7.0 FF install. It's not happening on my FF 75.0 Windows install. Perhaps the issue is with the 68.7 ESR release?

CAPKLU commented 4 years ago

Hello guys, I meet the same E027 issue and don't find any log too. It seem like that when I delete the .xbel.lock file ( I backup all that file of course :)), and it work again. Hope to help you. Moreover, it is interesting that this happens when the size of .xbel file is exactly 64kb.

marcelklehr commented 4 years ago

Can you confirm or verify if this issue is caused by the circumstances desribed in #505 ?

wohali commented 4 years ago

I have a very large set of bookmarks - a few thousand - so I don't know how to confirm or deny this is the root cause easily. But that looks like a good reproducible case. If you fix it, I can try your fix and tell you if it solves my problem.

CAPKLU commented 4 years ago

I have only one bookmarks and use the Webdav service from service provider (provide only https://, not http://). So I guess that is not similar to #505 .

marcelklehr commented 4 years ago

@wohali The logfile never shows any content other than the one successful sync I forced.

So, with v4.2.x it should at least log something now on these never-ending sync runs.

Klemens2 commented 4 years ago

For me, it's running now after last update. Very happy, so I decided to make a little sponsorship to Marcel :-)

wohali commented 4 years ago

@marcelklehr [EDIT] OK, I see it hanging halfway through now.

The "Cancel Sync" button does indeed disable the sync but it doesn't actually stop the sync in progress.

The logfile is uploaded to your private server as floccus-4.2.1-2020-06-08-4.log. Sorry the name's not more descriptive.

marcelklehr commented 4 years ago

@wohali I think I've tracked down the problem in your case to two bookmarks in the same folder that differ only in the protocol (http/https). I haven't been able to repdroduce this, yet. But given the right circumstances it could happen that floccus thinks a bookmark was moved, even though it wasn't, and then waits indefinitely for that other occurrence to be found. I'll email you the bookmark in question. EDIT: Removing that bookmark may solve the problem.

wohali commented 4 years ago

@marcelklehr OK, so I first deleted BOTH of the bookmarks you mentioned while still on 4.2.1, and tried a sync, as well as "trigger sync from scratch." Both still hung.

I then upgraded to 4.2.2, and tried a sync. Both bookmarks somehow came back (http: and https:) and the sync hung again. I deleted both bookmarks, then triggered a sync from scratch. Now, everything is fine.

So I think you have a new bug to file, and I'm out of the woods. THANK YOU for the help!

nunnsby commented 4 years ago

I am still having this problem with 4.2.2, specifically with Chrome on Win10 currently.

nunnsby commented 4 years ago

So, upon further testing, the never-ending sync seems limited to the one Chrome instance that the changes were made on. In fact, it appears that it has never even attempted to update the webdav stack as the webdav file hasn't changed, and I don't see any lock files appearing on the webdav server. Seems something is causing it to NOT get to actual sync stage.

nunnsby commented 4 years ago

Deleted the Account. Recreated a new account. Apart from experiencing #561 and #565, I get the following error in the logs:

2020-06-11T14:13:06.235Z { [Error: 1 errors]
list: [ [Error: E008: Folder to move out of doesn't exist] ] }
2020-06-11T14:13:06.236Z [Error: E008: Folder to move out of doesn't exist]
2020-06-11T14:13:06.236Z Syncing failed with E008: Folder to move out of doesn't exist
2020-06-11T14:13:06.248Z onSyncFail

Appears this is one of the causes. Other errors are cache item undefined. One such example:

2020-06-11T14:13:05.795Z COMPARE { localItem:   - #1335[Archer C7 v4 support - For Developers - OpenWrt Forum](https://forum.openwrt.org/t/archer-c7-v4-support/3924/36) parentId: 1328,
  cacheItem: undefined,
  serverItem:   - #2733[Archer C7 v4 support - For Developers - OpenWrt Forum](https://forum.openwrt.org/t/archer-c7-v4-support/3924/36) parentId: 2182 }
2020-06-11T14:13:05.796Z Bookmark unchanged
nunnsby commented 4 years ago

So, the endless sync problem gone.

But, it has been replaced by a non-working sync. The sync says completed, but it doesn't sync. It doesn't amend the changes made on browser 1 (Chrome), to browser 2 (Chrome), even though browser 1 says sync completed and browser 2 says sync completed. Looking in the logs, I do not see the bookmark I added to 1 appearing in 2.

marcelklehr commented 4 years ago

@nunnsby I'm sorry for your troubles. As you seem to have a new problem, can you create a new bug report for this and fill out the issue template with the details of your setup and steps to reproduce? That would be really helpful in tackling this :relaxed: :blue_heart:

marcelklehr commented 4 years ago

I've since rewritten the sync algorithm to avoid this kind of problems. Head over to https://github.com/marcelklehr/floccus/issues/666 if you'd like to help test the new version.

marcelklehr commented 4 years ago

I'd be very grateful if you could try out the release candidate: #666 -- I promise, you'll get extra cookies :) heart

marcelklehr commented 3 years ago

v4.4.8 is out now, which fixes problems like these. I'm closing this for now. Feel free to open a new issue if you continue to experience problems.

github-actions[bot] commented 1 year ago

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.