Has anyone experienced bookmark database corruption? (I have not)

[Gitoffthelawn]

First, I want to make it clear that I have not experienced any problems with this extension, so, readers, please don't be alarmed in any way. :)

Not long ago, I installed a different well-regarded bookmarks-related extension, and it caused corruption in the Firefox bookmarks database. It is still unclear if the bug lies in Firefox or that extension. Long story short, I had to spend many hours recovering bookmarks from backups from before that extension was installed, and then adding/editing many items.

I want to avoid having to do that again! :+1:

Before installing this extension in Firefox, are there any known issues or concerns regarding bookmark database corruption related to its use?

[aaFn]

Hi @Gitoffthelawn, so far that is more like the reverse :-) Some users here are using BSP2 to get around their huge bookmarks DB (60000 to 80000 items) which was in the past in some way corrupted, and BSP2 allows them to find again their lost bookmarks / subtrees :-D

See for example https://github.com/aaFn/Bookmark-search-plus-2/issues/5#issuecomment-368577828 https://github.com/aaFn/Bookmark-search-plus-2/issues/60#issuecomment-431202150

If you are unsure, best is to go to the library, then export your bookmarks before installing. Then you can try the add-on, and if there is a problem:

• tell me here, so that I can look after it
• restore your bookmarks from the export, and you will be safe.

Now, all that BSP2 does is through the bookmarks API https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/API/bookmarks so it is hard to corrupt anything from the add-on (only cause would be an FF problem).

And BSP2 does nothing to the bookmark DB until you do an action like create, insert (copy or drag & drop), modify and delete. Unless you do one of these actions, all is read only, so even if there is a bug in BSP2 (that happens, I usually try to corect them quickly), this is only a display thing, nothing bad happens to the DB itself.

I'll stop here and will let other users express themselves. Thanks for any feedback, aaFn.

[LooDom]

Hi aaFn ! Same issue since last upgrade (march 31): Bookmarks: 16197 Favicons to fetch: 0 (0.0%) Folders: 2400 Separators: 29 Oddities: 0

My Bookmark are sync with FF so I doubt that the DB should be corrupted. As said by Gitoffthelawn the standard FF search work fine. I deactivated BSP2 then saved Bookmark and shutdown FF then reopened it and reactivated BSD2 : no result ! Notice from files "masto" work and "mastodon" d'nt. FF 66.0.2 was released 2019-03-27 too (win 7 Pro 64bits). Thanks for that useful tool anyway.

[aaFn]

Hello @LooDom, so I guess you mean that you wait for search results, but none is coming, correct ? Any error message logged in the browser console ?

Do:

• Ctrl+Shit+J to see the browser console
• Empty it using the trash bin at top left
• Then execute your search
• Tell me if you see any message appear there, and if yes, can you copy it here ?

Thank you, aaFn.

[LooDom]

Hi @aaFn It work now. Sorry for the inconvenience :( Indeed I found some detail in browser console, but nothing seems related to BSP2 Thank again for this useful tool ! @LooDom

[aaFn]

Hello @LooDom , ok, glad this is ok, and thank you for coming back here to say so. Closing the issue then.

[Gitoffthelawn]

@aaFn Would you mind keeping this issue open for about another week, to see if anyone is having any issues? Thanks for your patience and help! :+1:

[aaFn]

ok, done

[mabloom]

I have problems with database corruption.

A large sequence of consecutive bookmarks appears twice. Attempting to make additions or deletions to something within one group actually changes it's counterpart in the other group.

Adding a bookmark the normal way makes the addition to the second group. Modifications and deletions made to the first group occur in the second group

Example:Using CTRL-D I create a bookmark of this page. It goes into a folder named bsplus in the second group. To provide a distinction, I use "Properties" in the sidebar to rename the FIRST bsplus to bsplus-first. Except after I've clicked "save" it has not been renamed. Instead, the SECOND bsplus was renamed to bsplus-first

Worse, I tried to delete an apparently empty "first folder ", but instead the second folder of the same name (which contained many bookmarks) was deleted.

The duplication is only seen in Bookmark Plus. The corruption has no effect on Mozilla's own presentation of bookmarks (other than the consequence of deleting the "wrong" folder, that is).

This happened after a disk crash. I replaced the disk, reinstalled the OS, and restored my home directory from backup. The OS verion of firefox was fairly old, so I downloaded the current firefox.

Thus the cause could have a lot of culprits.

In the hope that uninstalling and reinstalling bookmark search plus, would fix the problem, I tried doing so, but it persisted following reinstallation. Presumably, the extension's own database was cached, even through the extension's temporary absence.

Assuming that removing the bookmark search plus database would set things straight, how would I go about doing that?

Is the database a in a file that I can just delete?
If not, is there some way to tell bookmark search plus to clear the bookmark database, so it can be rebuilt from scratch? Is there perhaps a special cookie setting that bookmark search plus recognizes as a request to clear the database and then reset the cookie's value?

[aaFn]

Hello @mabloom , the problem is not with BSP2 but with the Firefox bookmark DB. It has apparently a folder object somewhere (the common top of both common trees) which I qualify as a "view".

Therefore, the corresponding subtree is only once in the FF DB, but it is shown twice on two different paths = anything you do on one path also happens on the other path .. normal, these are the same and unique objects, seen twice.

Origin of that type of corruption is not BSP2. It is not able to create such objects. BSP2 shows things reliably, so it allows to see that kind of corruption in the FF DB.

You can continue to live like that, from experience it doesn't cause any problem, except that additional "view" (or link if you prefer) to the same subtree.

I am not exactly sure how to correct that kind of FF originated corruption, I think some other people here found a way, but didn't make note of exact conditions or procedure they applied.

[mabloom]

Hello, @aaFn,

It would not surprise me that Firefox code could be the root source of unexpected behavior in BSR+ , because ... Begin_Tangent Flame_ON Firefox code is horrible.

• when I did an strace of Firefox's execution earlier to see if I could determine why was is so slow, I saw that the vast majority of it's execution time is spent effectively busylooping using needlessly repeated syscalls and combinations of nonblocking reads on descriptors that are almost never ready due to the way they use zero timeout poll() calls (and hence nearly always return -1)!
  :strace is the linux system call tracer - SystraceNT is supossed to provide equivalent functionality for Windows (but if it only has a GUI, it's output might not be as easy to filter as that of strace)

Flame OFF End_Tangent

I must have described the problem I'm experiencing poorly.
You said "anything you do on one path also happens on the other path .. normal, these are the same and unique object that is, seen twice." But that is very much not the behavior I was reporting.

If all that was going on was a mere mirroring of the presentation, I would not be so concerned. If it was filesystem code, and we were talking about an opportunity for unintended file deletion, and I was responsible for it, I'd mark it "showstopper" and refuse to do a handoff.

To try to describe it more precisely, I'll use the following notation: ` **F**_n_: An instance of a folder under discussion **C**_n_: An instance of a folder containing an **F**_n_ ` B: bookmarks If what was happening was that doing "anything" to F1 (an F contained within C1 a containing folder) caused the very same operation to appear to have been performed on F2 (an F that was present in C2 (a different C)), then I would agree that it was not of serious concern. I'd just see it as a curious, but mild annoyance.

As a specific example, container C1 contains a folder F1, which is empty. Elsewhere, a container C2 is seen to contain a folder F2 which is not empty, but contains multiple bookmarks {B1,B2,B3,B4,...B**n}.

When I perform a delete on F1, there (at first impression) appears to have no effect. C1 continues to contain the obviously empty folder F1. What is not obvious at first is that there actually was an effect. A very serious one. A folder within a different container has changed state in a very serious way: When you examine C2 you discover that folder F2 (together with all its bookmarks) has ceased to exist, both within the extension and on Firefox's Library window.

A similar although far less serious anomaly appears to occur when attempting a rename. Let's say that F1's name is Lucy, and you want to change it Patty. You select the folder, and using the right click menu, issue a rename. You may feel frustrated upon discovering that nothing happened, F1's name is still Lucy. It isn't until you examine C2, that you discover that F2's name (which had been Lucy) is now Patty. And F1's name is still Lucy...

Having to bounce around all over the tree to perform further operations on the folders and containers involved, and/or figure out where your bookmarks really are (and what names exist where) is a major annoyance.

But that is nothing compared to actually losing your bookmarks because you tried to use one of the extension's major "selling points": The ability to trim excess folders and duplicate bookmarks as part of re-organizing a bookmark tree that has grown to be unwieldy.

But getting to the non-technical content of your reply.

You're not seriously saying "Because their software is not up to snuff, I don't need for they and I to even try to play nice with each other", are you?

If the corruption is Mozilla's doing, and if it's also the case that they their code doesn't see the kind of consequences that BSR+ users see, then the chances are zero to none that they are going to fix it, which will remain a problem for BSR+ users.

That's why they teach defensive programming: "If the data is imperfect, don't let it's state screw your own users". Make sure your code can withstand bad data. At least that's what I was taught in the early eighties. And that's the way I always coded in my pre-retirement days.

Maybe the proliferation of Microsoft software, of people taking the existence of "blue screens" for granted, of people doing things like relying on the catching of exceptions instead of pre-emptively testing for edge conditions, and of people thinking "disk, ram, and cpu are nearly free. Why do I need to take subjects like "combinatorial algorithms"? O(infinity^infinity) is fine by me" have taken their toll on today's teaching syllabuses, to the detriment of the public.

(that's one of my biggest pet peeves, one which was refreshed earlier today when I examined the output from running strace on both firefox and chrome -- both of which are equally horrible as the interactions within a single product between code from "too many cooks" with different ideas of "the right way to code" have slowed the browsers to a tiny fraction of their potential speed)

[mabloom]

Regardless of the cause of the problem, one thing I can state for certain is that it did not present itself at 11 AM, just before I shut down the system. When I logged in after the process of installing a new disk, reloading Linux, and restoring my home directory from backup, and then upgrading Firefox from the version that came with Linux to the current version, the problem was there the first time I started Firefox (before trying to create a new bookmark).

I presume you have your own mechanism to rebuild the BSP+ database from scratch. How could I do that myself? Is it kept in a file that I could just delete? Or is there a cookie I could set that tells BSP+ to zero out it's database? Or is there a cookie I could set that tells BSP+ to rebuild it's database from scratch?

I'm not asking you to "fix" anything. Just to supply some information. It would certainly be appreciated. I want to be able to use BSP+ again.

[mabloom]

I think found a solution. I uninstalled BSP+, exited Firefox, restarted Firefox, and then reinstalled BSP+. BSP+ now appears to be working as it should.

Previously disabling and re-enabling it had had no effect. But reinstalling BSP+ from scratch seems to have repaired the errant behavior.

[aaFn]

Hello @mabloom , no magic, I am just using that call to the FF API to get the full bookmark tree the first time BSP2 is run https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/API/bookmarks/getTree

And then I cache it in local store, because there is a bug in FF which makes that call quite slow, cf. https://bugzilla.mozilla.org/show_bug.cgi?id=1396364 and all modifications to bookmarks are sent to BSP2 by FF which faithfully applies them on the cache to keep it in sync. Local store is a thing managed by FF, and since FF 66 this is an sqlite DB as I get it. Add-ons cannot create files by themselves.

You can force a re-read from the API in the BSP2 options page, by clicking on the Reload from FF API now button, if for some reason a de-sync occurs .. and so no need to uninstall / reinstall.

Hope that helps :-)

[mabloom]

@aaFn Thanks for the response. Yes, it does help.

I had been looking for something like that before I first wrote, but completely missed it.

I think my difficulty recognizing it may have been due to my thinking about it from the point of view of a user, not that of an implementer, and Reload from FF API now is how an implementer thinks of it, whereas a user is looking for something like "Reload the BSP2+ database".

The average user isn't thinking about the API. Some (many?) may not even know what the acronym API means. (In my day, I had implemented my own API's (as well as kernel interfaces), yet I completely missed this).

[aaFn]

You are correct, I should rename the button to comething more explicit for any user.

I propose Reload all bookmarks from FF .. how does that sound ?

[Muddy7]

I've had a recurring DB corruption problem not yet mentioned in this thread. I have so far observed this problem only on my Windows 7 device.

After a period of time, I find that folders and bookmarks that I am creating in BSP2 are failing to appear in the native FF bookmarks (and therefore, of course, will not transfer to my other devices with FF Sync). When this begins to happen, it also becomes impossible to delete or rename any folders or bookmarks in BSP2 (or is this just the folders and bookmarks I created since this problem began? I can't remember).

Solution: I have to remove and then reinstall BSP2. However, before doing this, I have to recreate in native FF bookmarks any bookmarks and folders I have created after this problem began to manifest itself otherwise they will be lost forever.

This is a regular and recurring problem.

Another thing that is maybe related: I find that on this same Windows 7 device, the BSP2 search window can often become VERY slow, sometimes accepting only 2 or 3 characters before freezing/thinking-about-it for up to even a minute (!!) before continuing to accept more characters. I should add that this device (which is the main device I use) is generally sluggish, sometimes particularly so in Firefox.

It would be nice if there is some way this could be resolved as, as you can imagine, it can be quite annoying. However, despite this considerable annoyance, I still definitely prefer BSP2 to FF native bookmarks because of essential features it has which are missing from the native version!

[Gitoffthelawn]

@Muddy7 Thank you for your detailed report. I have 2 questions for you:

1. Do you use any other bookmarks-related FF extensions?
2. When you have to recreate bookmarks and folders, how do you know which ones need to be recreated?

Thanks!

[Muddy7]

Thanks, @Gitoffthelawn.

In answer to your questions:

1. No
2. I don't. Rather, I have to remember as best as I can. As a result, I'm sure I lose some bookmarks each time :-(

[Gitoffthelawn]

@Muddy7 You're welcome, and thank you!

Regarding (2), that's what I figured. And, given that complication, it surprises me a little that you continue to use BSP2! It's undeniable that BSP2 adds critical and much-needed functionality to Firefox. In your case, you value that functionality so much that you're willing to go through occasional headaches to get the benefits of that wonderful functionality. That is strong testimony as to the importance of the functionality provided by BSP2!

I very much want to be able to use BSP2 as well, but -- for me -- time and energy spent trying to recover recent bookmarks would be too big of a headache.

I'm hoping that @aaFn will be able to determine where the issue(s) reside, and be able to either correct BSP2 or determine the problem is completely with @mozilla and Firefox, and file the necessary bugzilla reports.

BTW, in my original post I did not get into specifics regarding the other extension that resulted in bookmarks database corruption for me (still not sure if it is an issue with that extension or with Firefox itself). In the event that it may help either @aaFn or @cadeyrn (or both!), I'll provide the link here: https://github.com/cadeyrn/bookmarks-organizer/issues/105

[Muddy7]

@Gitoffthelawn

The key for me is to recognise the presence of the BSP2 bug as quickly as possible after the moment it starts appearing, and then immediately remove and reload the extension. I have also decided as a matter of policy, to remove and reload the extension every weekend, hoping that this will help to act as a preventive measure.

[Muddy7]

It's undeniable that BSP2 adds critical and much-needed functionality to Firefox. In your case, you value that functionality so much that you're willing to go through occasional headaches to get the benefits of that wonderful functionality. That is strong testimony as to the importance of the functionality provided by BSP2!

Yes, I'm a great fan of BSP2.

I should add that the computer I use mostly, my Windows 7 device, and on which I encounter this problem, can at times be temperamental, and I have wondered if my BSP2 problems may be at least partly connected to this. I'm just saying that so you don't let your decision whether or not to add BSP2 to your browser be too much based on my perhaps untypical experience.

[Gitoffthelawn]

@Muddy7 Thanks! :) If I didn't spend so much time and effort cleaning up from the bookmarks database corruption related to the Bookmarks Organizer extension, I would definitely be less hesitant. But, IIRC, that snafu took about 4-5 hours to diagnose and clean up.

@aaFn Determining if the issue is related to specific extension(s) or is a FF issue will be key. Do you know, for performing all bookmarks-related tasks, if FF itself uses the same Bookmarks API as extensions use?

[aaFn]

Hello gents, sorry, have been travelling and in meetings a lot ..

I had in the past a look at some of the FF native bookmark sidebar and libraries code, and the answer is clearly no, it doesn't use the WebExtension API. It has its own internal code with direct access to internal DB tables and queries of course, this is called "places".

Same by the way for display, I am quite convinced it uses OS level primitives, and not HTML objects like we are obliged to do in add-ons (much faster and less consuming than us ..).

The initial Bookmark Search Plus extension by Alice0775 which was a XUL extension was directly acting at that level by being integrated with the native FF bookmark code, and this is not anymore possible.

---

On bookmark corruptions, frankly that can only happen by FF itself. See the API https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/API/bookmarks, we can only either get information, or act (create, update, delete, move) through text fields like title, name, id of parent. The bookmark structure is fully managed by the API anf FF.

The most complex object on the modification API's is this one ->https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/API/bookmarks/CreateDetails Nothing which can break or corrupt anything ...

---

@Muddy7 , when bookmarks that you are creating in BSP2 are failing to appear in the native FF bookmarks, could you:

• open the browser console (Ctrl+Shift+J),
• then empty it (top left trash icon)
• then create a new bookmark / folder which is failing to appear in FF bookmarks
• and copy here the messages you see ? In case we can catch something ...

Also, when you see that things are becoming slow on the search box, could you open about:performance in a tab, and see what could be eating cpu time or big memory to report it/them here ? You will see BSP2 and other add-ons appearing as "module", and rest should be "tab".

By the way, how many boookmarks do you have ? (see the counts in the BSP2 options page)

And when things are becoming erratic, could you try the button "Reload from FF API now" in the BSP2 options page, and verify if that cures the problem or not, rather than uninstalling / reinstalling ? There might be a de-synchronization happening, in case some API events are not properly navigating between FF and BSP2 ...

Thanks, aaFn.

[Gitoffthelawn]

@aaFn Thanks for the details. :)

Based on what you wrote above as well as what @cadeyrn wrote in https://github.com/cadeyrn/bookmarks-organizer/issues/105, I have a hunch that the Firefox Bookmarks API may have a bug or two.

It's such a critical API that I would love to see it working properly. I'm not sure how I can help with that without investing 4+ hours performing extensive testing. Unfortunately, I'm too overbooked to invest that much time right now. If anyone has a 5-15 minute task they need me to perform, just ask, and I'll try to pencil it into my day.

[aaFn]

Let's wait for if @Muddy7 can give us any error messages / exception appearing on the browser console when he experiences the problem.

Hopefully there will be something, and if I am able to catch it, I can at worst take that as an indication of desync between BSP2 and the FF Bookmarks internal DB, and force a resync myself for robustness.

Of course, best case is I am able to interpret what is happening and have a workaround, but I have some doubt about it.

[Muddy7]

Thanks, @aaFn for your response and your very clear instructions re troubleshooting for a non-technical guy like myself!

Up until now, this problem was manifesting itself about once every 2 to 4 weeks (though it may be that it was more frequent but I was just not catching it quickly enough after it first appeared). But suddenly this week, it seems to have gone into overdrive, manifesting itself three times so far(!), the most recent being this morning (which is good because it means I have been able to respond fairly rapidly to your request).

So this is what happened:

I was using my Windows 7 device.

I was wanting to bookmark an old, fact-checking newspaper article about Boris Johnson and his hullabulla about the EU and bendy bananas (for those not closely following the story about Britain and Brexit with all its arcane ins and outs, this will be a meaningless story to you so don't worry! But the point about the article for those in the know is that it completely rubbishes Boris' silly EU banana claims!)

Anyway, I had decided to place it in two bookmark folder locations: Bookmarks Toolbar > World > Current Affairs > EU and Bookmarks Toolbar > World > Current Affairs > Boris Johnson, "Boris Johnson" being a new folder I created.

As far as I remember, I then checked FF native bookmarks and, well and good, these 2 new bookmarks and 1 new folder had been generated there as well. But FF Sync seemed to be eternally spinning and the two new entries were not syncing to my Windows 10 device. So I closed and reopened FF on Windows 7. Then, as far as I remember, I found that the 2 new entries had disappeared from the native bookmarks but were still present on BSP2. Curiously however, the new Boris Johnson folder that I had created was indeed present in the native bookmarks but empty. So I manually created the bookmark in the native bookmarks in the already existing and the newly created folder. I then went to BSP2 and, as far as I rememeber, I found two instances of the bookmark in the already existing folder and two instances of the newly created folder, each with one instance of the bookmark. I hope you are managing to follow me :-0. I am recounting as best as I can remember as I did not take detailed notes of all this :-(.

Next, as instructed, (from here on I took notes as I was working) I opened the FF browser console, and attempted to empty it using the Trash Bin icon. It didn’t seem to empty, though it did appear to partially empty. So I closed and reopened FF and this time I was successful.

Incidentally there were about 150 entries there, most of which said: Request to access cookie or storage on “https://cdn.krxd.net/partnerjs/xdi/proxy.3d2100fd7107262ecb55ce6847f01fa5.html#!kxcid=JVZiE3vn&kxt=https%3A%2F%2Fwww.theguardian.com&kxcl=cdn&kxp=” was blocked because it came from a tracker and content blocking is enabled. https://www.theguardian.com/politics/2016/may/11/boris-johnson-launches-the-vote-leave-battlebus-in-cornwall (Btw this link is the URL link I had just added to bookmarks) or something very similar and a few of which said XML Parsing Error: not well-formed Location: http://localhost:27019/ Line Number 1, Column 1: or something very similar.

Using BSP2, I then bookmarked this page to Bookmarks Toolbar > World > Current Affairs, then created a new folder “BSP2” hence Bookmarks Toolbar > World > Current Affairs > BSP2 into which I placed the new bookmark. I found this was correctly created in FF native bookmarks.

Also, the result of the above was that miraculously everything seemed to have returned to normal on both FF native bookmarks and BSP2 inasmuch as (and here I stopped taking notes but I think my memory is correct) I discovered in both native and BSP just one Boris Johnson folder with one instance of the new bookmark and ditto for the EU folder.

However, now I look again and duplicates have returned to BSP2. Aaargh!!

I check in FF Native Bookmarks and all seems to be OK (no duplicates). I then check for Sync on my Windows 10 device and find that Sync has happened OK except for the fact that the Boris Johnson folder is incorrectly placed at the bottom of the Current Affairs folder and is empty. I go back to my Windows 7 device and discover that my FF browser is slower and buggily hanging more than ever that I’ve seen before, and discover this is apparently due to the open Browser Console window. This Browser Console window has yet again been populated with the same error message as before. I close and reopen the Browser Console and FF. Try to clear the Browser Console entries using the Trashcan icon without success. Try to close and reopen FF and retry clearing Browser Console entries several times but without success.

I discover that I can delete the BSP2 Boris Johnson folder that is correctly duplicated in native bookmarks but I can’t delete the other "rogue" BSP2 Boris Johnson folder.

This is becoming a nightmare as you can well imagine. So I decide to remove and reinstall BSP2.

But before doing that, I check on your last question: checking performance in about:performance when BSP2 search is slow. However I encounter no slowness at this moment, and so have decided to report back on that at a later time.

Phewww!!!!!!!!!!

EDIT: Just seen your other two instructions at the end (before uninstalling and reinstalling BSP2) Number of bookmarks: 6618 - Number of folders: 1829 Tried your suggestion to click the button "Reload from FF API now". It seems to have worked, in clearing out the "rogue" BSP2 2 bookmarks and 1 folder so I'll try this suggestion next time I have "rogue" BSP2 bookmarks" not duplicating in FF native (though obviously I'll have to make sure I've created them in native bookmarks before doing this). Thanks!

[aaFn]

Ok, thanks @Muddy7 for this very detailed report.

So, for some strange reason, here is what I interpret is happening:

1. you create your folder and 2 bookmarks
   • everything is going fine in the native FF bookmark DB so far, since you see they are there
   • everything is going fine in BSP2 also, meaning it received the acknowledge from the native FF bookmark code (DB and API) that they were created, and so it stores them in its own cache
2. something was going wrong in FF, and you decided to close it
3. as it turns out, the native FF DB didn't really store its new version ... but BSP2 did
4. so when you restarted FF, it had "forgotten" about the 2 bookmarks you had created, but not the folder, meaning that FF had written at least on disk the folder modification in its DB
5. from then we have a de-sync between FF and BSP2:
   • in FF, the 2 bookmarks were not there
   • in BSP2, they are showing, because it read things from its own cache to be fast (cf. slowness bug I mentioned previously)
   • when you re-created the 2 bookmarks in native FF, they were added both in FF DB and in BSP2 cache as 2 new objects (their title / url doesn't make them identical objects, bookmarks have a unique Id of them, whatever their contents, and so they are seen different by BSP2 - and the FF DB of course)
   • .. etc ...
6. when you re-sync-ed BSP2 and the FF DB by pressing the button, BSP2 discarded its cache and rebuilt it from the FF internal DB => all is good

Obviously, to see if there is a way to catch the problem, we would need the Browser console to be open at 1), and to catch any error there.

Or better, since you observe the problem after the fact, when your sync is spinning at 2), and before closing FF, suspecting something is going wrong there, do the procedure of opening the browser console, cleanup to get rid of old messages, and create one more bookmark in BSP2 .. and dump the contents of the Browser console here.

---

Now I understand this is painful, and given the above, there is a way to avoid the de-sync, at the cost of bigger load times for BSP2:

• Go into BSP2 options page
• Check the Revert to old behavior of reloading full bookmark tree from FF API at each FF restart.. option
• Then BSP2 will re-sync at each load, and you won't have the side effects of steps 5) and further ..

At this stage, all seems to be a problem in FF. Let's see if you have further information when it occurs again ...

[Muddy7]

Thanks!

Sounds about right as a resumé, bearing in mind that my report was only to the best of my recall abilities (will try to double check next time).

For browser console dump to this page, should this be a dump of data BEFORE cleanup or AFTER creating one more BSP2 bookmark?

And will the Revert to old behavior of reloading full bookmark tree from FF API at each FF restart.. option make it less likely that I will be able to witness this bug and so do the above for your help? Also, will it mean that any bookmarks created in BSP2 during any bug problem will be automatically cleared out at next FF restart thus losing those bookmarks?

[aaFn]

The browser console contents copy to here should be JUST AFTER doing the additional test operation of creating one bookmark, which itself is AFTER cleaning up the console contents.

Ticking the option I mentioned will not change anything to the frequency of the FF delay (or bug ?) in saving its internal bookmark DB, but it will re-sync BSP2 at each restart, then avoiding the effects of having a different content in FF native and in BSP2.

From what you say, the problem of "forgetting" created bookmarks could occur anyway at any time in FF native ... unless they forget to save only when we are creating a new bookmark through the API .. in which case it would affect anything like an add-on creating bookmarks, but not operations from other sources.

Hard to tell. If through your traces I find a way that BSP2 can become aware of the problem in FF, at least I can code something to warn you ...

[Muddy7]

Tested your option of Revert to old behavior of reloading full bookmark tree from FF API at each FF restart.. and it doesn't look as painful as I feared. Yes, a bit slower than usual loading but not so much so. And it's not like loading at BSP2 Install when there is a delay between loading bookmarks and loading favicons.

I've left this option enabled, and will monitor.

Will continue reporting any bug appearance and behaviour.

Thanks!

[aaFn]

Yes, the duration of of the load from FF API depends on many factors:

• Number of bookmark items of course
• Whether you use tags or not (tags are stored as objects in the same DB table as bookmarks in FF !!! crazy ..)
• Speed of your system (cpu, SSD or not, RAM .. etc ..)

So it is hard to tell in advance, but there are cases where it can be very long. Average cases I am gaining a factor of 2, but some users can see a factor of 10 or even more, I believe this is exponential ..

[Muddy7]

If you're talking of the tags as shown in the Properties window of FF native bookmarks, as far as I know I've never used them (never seen the use of them—that's not saying they are not useful, just that if they are, I've never understood in what way)

Unfortunately :-( , I don't use SSD on my Windows 7 device. Ram=4GB. Cpu=Intel Core i3 M370 @ 2.40GHz.

It's a slow computer.

[Muddy7]

You asked for a screenshot of about:performance next time I had a BSP2 search slowdown.

Just had a search slowdown, so here was the screenshot:

[aaFn]

Strange, there is no module at all listed. Only "Tab". You should see BSP2 at least .. and I see you probably have other add-ons from the icons in the toolbar, they should show also

Something is going wrong, but not sure what.

Which FF version do you have ? You have a little orange triangle with an exclamation mark on the Menu button,, it might want to say something to you ..

[Muddy7]

Yeah, the orange triangle is about one of my add-ons, that I have had disabled for some time (but not removed as may use in future), demanding extra permissions.

FF version 66.0.3 (64-bit)

FYI I have 8 extensions, 3 of which are currently disabled, and 3 Plugins.

For the rest, I don't know.

[aaFn]

So 5 extensions ("module" lines) should show in the report .. and at least BSP2 for sure. I checked in a Win7 VM I have, with the same version of FF. Something is not good, but can't tell what.

[Muddy7]

Being completely untechnically minded, I've no idea what. Any ideas??

UPDATE: Interesting! Now the 5 extensions appear in about:performance. I don't know if this is connected but, for the first time I can remember in some time, the orange triangle about giving more permissions to an extension I have had disabled for some time, has disappeared

[aaFn]

ah .. let's see how that goes now then ... and if slowness occurs again, I'll be interested in the about:performance report

[aaFn]

@mabloom , 2.0.60 is out with a new name for the reload button, as discussed.

[aaFn]

@Muddy7, please see https://github.com/aaFn/Bookmark-search-plus-2/issues/103#issuecomment-507068395, I suspect I found the root cause for the problem of FF saying ok on modifications, but not really saving behind the scene.

That would be due to the new Indexed DB format used for local storage by FF since version 66. It is having bad performances, with very high cpu usage for several seconds, for users with large numbers of bookmarks, and my theory is that the CPU high consumption is resulting in FF forgetting or timing out on its internal DB saves, without saying. It has improved as we observed with FF 67.0.4, but it's not yet at the same level as before FF 66, by far.

These CPU spikes are rendering the BSP2 UI quite unresponsive at times, so I made a workaround to delay saves to local storage, and my intuition is that this should also address the problems that you were experiencing of modifications made through BSP2 not saved by FF to its internal DB, and so not retrieved on next restart.

Let me know .. thank you, aaFn.

[aaFn]

Supposing I found a cause of the problem, due to long saving times of FF 66+, happening at a moment that FF didn't like, and that the workaround / enhancement published in 2.0.63 is avoiding. So closing this issue.

Let me know if that comes back. Thank you, aaFn.

[Gitoffthelawn]

@Muddy7 Are you still around and using this extension? Any updates on how it's going for you? Are you still needing to uninstall/reinstall this extension frequently or perform other workarounds?

[Muddy7]

Are you still around...

Yes, I am!

...and using this extension?

Yes!

I'm sorry I've been incommunicado for some time. This is because I've been snowed down (still am somewhat) :-(

@aaFn, I still love Bookmark search plus to bits! Also, have experienced no problems since you reported end June (above) your appearing to have resolved the problem. In the last few days I have unchecked the workaround that you had recommended in the Options ('Revert to old behavior of loading full bookmark tree from Firefox API at each restart'). Will report back if I encounter any issues.

'Fraid not being very computer-literate I don't quite follow the technical lingo in para. 4 of https://github.com/aaFn/Bookmark-search-plus-2/issues/103#issuecomment-507068395 where you seem to talk about a rare exception in which your fix may not work. However, maybe it is not important for me to understand? Maybe this is a situation that I am very unlikely to encounter??

Again, thanks @aaFn for a great extension that frankly I find difficult to imagine living without. For me, it has all but replaced the FF native bookmarks. And thanks, @GitofftheLawn, for prompting me to reply after a long silence.

When I find more time, I'll try to become more active again as there are one or two tiny tweaks I'd like to suggest to improve even further this excellent extension.

[aaFn]

Glad to see this is working fine :-)

I do not believe you are falling in the rare case I was citing:

• this is for people with very high number of bookmarks -> you had only a little less than 7000 at the time, which is "small" compared to the huge numbers I am seeing,
• and plagued with the "slow save" symptom -> which seems to have improved as FF is going up in versions.

Again, thank you both for your feedback, and will be happy to hear about the improvement tweaks you are seeing when you have time.

[Gitoffthelawn]

@Muddy7 Welcome back! :) Thanks so much for your report. Definitely let us know how it goes with 'Revert to old behavior...' unchecked.

Are you still needing to uninstall/reinstall this extension frequently or perform other workarounds?

---

@aaFn You're welcome. The big thanks is to you for developing this tool!

What do you consider a "very high" number of bookmarks that may qualify a system for the rare case?

[aaFn]

60000 bookmarks starts to be a frontier where I had lots of troubles and significantly had to put efforts to play around FF limitations or constraints. As can be seen in the different issues, some users have 80000 or more ..

[Muddy7]

Wow! 60,000 bookmarks+!!! And I thought I had a lot LOL. And btw what a good memory you have @aaFn for how many bookmarks I have!

No, @Gitoffthelawn, no reinstalls for some time now. Which is great! I still had the reload bookmarks at each restart from FF API in Options menu as I mentioned (only workaround I ever used), but as I say have since a few days ago got rid of that and will see if anything changes.

"Slow save". Hmmm. Could that mean that if I ever add a bookmark and immediately close FF that I could lose that bookmark?? (If so, I can try and avoid that scenario) (Or is that only for the people with 60,000+ bookmarks??)

[Muddy7]

I suspect what I am about to say is no longer relevant as @aaFn says he believes he has got to the root of the problem, but it should be noted that my recent reports no longer relate to my Windows 7 laptop (as did all my previous reports) since I now rarely use this device. My main working machine is now my Windows 10 device, so my latest reports relating to BSP2 behaviour come from this device. The reason I mention this is because my Windows 7 device could (and can) be temperamental:

I should add that the computer I use mostly, my Windows 7 device, and on which I encounter this problem, can at times be temperamental, and I have wondered if my BSP2 problems may be at least partly connected to this.

https://github.com/aaFn/Bookmark-search-plus-2/issues/93#issuecomment-484379430