Critical Performance Issues (Freezes Firefox for 60 Seconds when Switch Sidebars) + Frequent Corruption + "Add Bookmark Here" + Quick Rename/Move Usability Enhancements

[PowerWeb5]

I have encountered several critical performance and reliability issues, which I've detailed below, together with suggested ways to fix those. I really appreciate your having developed this incredibly useful extension (with features that should be built into Firefox but it still sadly lacks) but also have some suggestions for a few quick enhancements that I believe will greatly improve its usefulness, such as with "Add Bookmark Here" and Quick Move/Rename/Bookmark Editing, which I've also detailed below together with details and suggested fixes for critical performance and frequent bookmark corruption/syncing issues (which occur despite having tried different options for workarounds, and like seen in screenshot and related issues referenced further below).

Critical Issues, Suggested Fixes & Enhancements

1. BSP locks up Firefox for 60 seconds (making it unresponsive) when have 40K bookmarks (30 seconds to display + 30 seconds after without any noticeable visual changes) every time I switch to/from the Sidebar regardless of settings (w/ or w/o Pause favicon loading and/or Old immediate loading) even with only 15 bookmarks/folders shown and had all favicons and bookmarks loaded and cached during same session.

   • Instead, I suggest keeping a cached version of BSP2 panel in memory if possible when switching Sidebar tabs during the same browser session and load favicons JIT for expanded folders.
   • It actually freezes/locks up Firefox for 20+ seconds (so clicking tabs don't even respond) just when switching to/from different Sidebar and even when had already loaded fully before that and/or had Paused Favicon loading, and have only a few top level bookmarks/folders expanded and shown.
   • Even when I enable "Revert to old behavior of immediate favicon display" it still takes 20 seconds before even the first bookmark is shown (even when only have 15 or so top-level bookmarks shown/expanded to, like seen in provided screenshot)
     • Specifically what is this setting supposed to do and how does loading work without it?
     • It doesn't seem to improve even initial performance on opening Sidebar and it takes even longer before the first bookmark is shown when this is enabled for some reason, though it doesn't lock up for as long after all bookmarks are shown and loads favicons even faster for top-level ones).
   • Maybe a hybrid of "Revert to old behavior of immediate favicon display" can be enabled by default?
   • Opening BSP2 sidebar takes 30 seconds, and freezes/locks up Firefox for much of that time, every time I open it! This happens even after I just switched to another Sidebar (eg. Tree Style Tab) to use it for 10 seconds and switch back to BSP2, and then it takes 30 seconds of locking up Firefox all over again!
   • It takes almost just as long (30+ sec) even after I Pause favicon loading enabled.
   • It takes almost just as long even when it shows "Loading saved state" vs. when it shows "Load bkng page" vs. first opening after Firefox startup
   • Maybe can refactor to more async / parallelized loading can help speed this up, along with more JIT loading.
   • Maybe can save more (eg. serialize your entire virtual DOM or JS objects to background tab after sidebar is first closed) so that even Load saved state doesn't take so long too like it does now, or even serialize HTML elements)
   • This occur for high-end quad-core gaming laptop, with specs as detailed at end of this post.
   • Often right click > New Bookmark or Properties or New Folder all often end up very slow. This may be when performed while still loading in background.
     • It causes UI to lockup, and often it takes up to 30 second before the UI appears, so that the bookmark or new folder shows up with name like New Folder or New Bookmark in BSP2 but no dialog shows to edit the bookmark, so I have to manually right click > Properties and then end up seeing 2 dialogs appear eventually.
     • And when BSP2 properties dialog shows, there is a long delay after I click in the Name or Address textbox before the cursor shows there. And sometimes Firefox ends up slowing down a lot or locking up too.

2. Need to load favicons for visible bookmarks (for expanded top-level folders in BSP2) first, and load others JIT, by default

   • JIT load others bookmarks if/when their folder is expanded.
   • This is slow even when have "0 Favicons to fetch" under stats (had fetched all previously, even in same session or previous session) and even whether or not had Pause favicons loading
   • Delay loading others in background
   • Ideally load in a more thread-friendly manner to avoid locking up Firefox so that can't even switch tabs for 60 seconds after open BSP2 (20-30 seconds before and 20-30 seconds after first bookmarks appear or favicons all appear)
     • For example, maybe sleep in between favicon loads and/or only load favicons for 1-level under the expanded just 1-level under the expanded folders (and folders which will appear expanded when expand those top-level ones)
   • Currently, Bookmarks toward the bottom take up to 30 second to load, it seems with having to wait for favicons to be loaded for all folders, recursively or even top-level collapsed folders, before loading them
     • This is for bookmarks which appear at top-level but show in order after folders that have thousands of bookmarks under them (even if only once get many levels deeper than what's expanded)
     • Instead, please perform JIT favicon loading. And maybe even caching in your database if needed?
     • This slows down Firefox and locks it up for 30 seconds even when have Pause Favicon loading setting, and/or even when had previously loaded all bookmarks in the same session, 10 seconds ago.

3. Bookmarks keep ending up in "Other Bookmarks" folder, and never able to be deleted, very frequently, and even deleting from BSP2 itself fails to auto-resync or auto-remove orphaned bookmarks from your own cached database (which should be occurring instead, as detailed below):

   • This has happened maybe 10 times over the past year, and always the same case: it ends up under Other Bookmarks folder.

   • This is not just an issue of them being "Out-of-Sync" (as the developer had referred to in reply to related issue #97) because of the following reasons, and I would suggest implementing auto-fixing/detection in the ways described below.

   • The bookmark always appears in the same "Other Bookmarks" folder (even though it was never in that folder previously).

     • Does this mean you might be able to detect when the issue occurs, for example you fail to retrieve the bookmark folder, and therefore just default to showing under "Other Bookmarks" when that happens?
     • If so, you could and IMO should auto handle this common issue by auto triggering a Refresh/Resync Bookmarks.
     • I do not now and have never in past had bookmarks in that folder. Even new bookmarks don't end up there by default (I use "Bookmark Tab Here" and "Default Bookmark Folder" extensions, and previously, "Add Bookmark Here 2" to avoid them ever being added into Other Bookmarks, even by default, so that I always add directly into a specific folder as they are created).

   • When viewing/editing an orphaned/left over bookmark, you should be able to check and detect that at least when trying to edit it via BSP2's Properties dialog.

     • When that occurs, can't you just remove it from BSP2 cache, for example, if, when trying to re-sync or edit/update it in Firefox's DB, you find that no bookmark exists with that bookmark ID or for that URL?
     • Also, can't you resync/check syncing whenever a bookmark is clicked in BSP2 (if won't slow down) or at least whenever Properties dialog is shown or when trying to move/rename in BSP2?

   • If you are monitoring changes to all bookmarks, then shouldn't you at least be able to detect and apply changes made to a bookmark even if you had "missed 1 change notification"?

     • When I move/delete/rename/change URL for a bookmark in Firefox Bookmark Manager, etc. still even at least those edits should be applied (even if you missed the original edit events) to the bookmark in BSP2, however that is not occurring.
     • When this issue happens, do you end up having a bookmark in your cache with a place ID which doesn't exist in Firefox places DB or is it simply that you happened to miss one or more edit events (but still have the same place ID)? Either way, some kind of correction (even if to auto remove the orphaned bookmark from your cached DB) or handling of future edits should occur, and yet that isn't happening.

   • All of the following methods for trying to move/update/delete this bookmark once I see it incorrectly shown under Other Bookmarks (when I would suggest that you handle all of the following cases, able to detect, update/resync that bookmark or at least remove the left-over bookmark from your cache when detecting no corresponding bookmark exists in Firefox Places DB):

     1. Deleting the bookmark via right click in BSP2 sidebar > Delete Bookmark fails to even remove it
     2. Drag-dropping bookmark in BSP2 sidebar fails to move or delete that left-over/out-of-sync bookmark
     3. Right-click in BSP2 > Properties… fails then changing the title fails to edit it.
     4. Using Ctrl-D and then A) moving to another folder, B) renaming, C) changing URL, or D) deleting the bookmark in BSP2, even after closing BSP2, switching to another panel, restarting Firefox and reopening it and expanding and collapsing Other Bookmarks in BSP2.
     5. Searching by name and URL in Bookmarks Manager and ensuring its deleted there or deleting it there, from both Bookmarks and History.

   • There should be no reason that even attempting to Delete Bookmark for a bookmark in the BSP2 sidebar ends up failing to delete it from the BSP2 sidebar.

     • You should at least be able to remove such left-over or out-of-sync bookmarks from your own cached database that way.
     • This is critical to support to at easy, quick, low-side effect and intuitive to find workaround for such common issues.

4. Add right click menu that should be shown when right clicking background of BSP2 sidebar (not a specific bookmark, eg. like available in Tree Style Tab).

   • Refresh Bookmarks – to perform "Reload all bookmarks from FF now"
     • Simply providing "Reload all bookmarks from FF now" button hidden under 1 advanced setting towards middle of the Options page isn't sufficient, since not only is it hard to access, but, many, like myself are likely to never find or know of it, and have to put up with the issues or reinstall the extension (as I'd done several times before)
     • Or so users don't just stop using it entirely (as I'd also done considering how annoying and common these issues are with bookmarks that I know don't exist but that show up, sometimes 10 at a time, in very visible place all the time)
   • Options – to take you to the options page
   • Pause/Resume Favicon Fetching – for this or any other options that are the primary workaround for slowing down Firefox (as BSP2 does slow down Firefox very often)

5. Right click on Folder > Add Bookmark Here is critical to add

   • New Bookmark which just starts off showing empty default title and URL fields is almost useless (never found a reason to use).
   • You could just populate that by default with the current tab, since user can always easily edit to replace with custom values if/when ever needed to manually type or paste in a bookmark.
   • This is possibly the most critical feature to making BSP2 useful, since, without that, I rarely find BSP2 useful, since main intended use case for it would be to add bookmarks into folders.
   • Otherwise, its far more convenient to just add to Bookmarks Toolbar (via "Bookmark Tab Here" and "Default Bookmark Folder" extensions, and previously, "Add Bookmark Here 2") (even though with that way, I have to keep manually digging down into folders each time to find the folder to add into)
   • That is a major pain this extension could otherwise be incredibly useful in solving, especially with how many people are interested in finding an alternative to "Add Bookmark Here 2" now that that extension no longer works with Firefox.
   • Currently, even the standard Bookmarks sidebar is far more useful for most of my daily bookmark use (bookmark creation) now that it supports context menu extension with "Bookmark Tab Here" and "Default Bookmark Folder" (which also provides an Add Bookmark Here option) extensions

6. Quick Rename via Right-click on bookmark > Rename should be supported

   • With in-line editing (quick, intuitive, for the label as shown in the tree, like can edit with Bookmark Manager, et.c)
   • without having to show or wait for the tedious and very slow Properties dialog, which sometimes takes 60 seconds to show and slows down Firefox when it appears), due to how common this is.

7. Edit boxes should be shown at bottom of BSP2 sidebar to allow for fast editing as true Bookmark Manager replacement) for the following fields:

   • Name
   • Location
   • Folder
   • Keyword
   • Description (hidden by default, low priority)
   • Keywords (hidden by default, low priority)
   • Note: Without this, I often have to find the bookmark in Bookmark Manager or manually from navigating in menus or toolbar, in order to view/edit these things.

8. Enhancements/fixes for the Properties dialog (and critical performance issues with it)

   • Make Folder editable (with drop-down of recent folders and/or folder selector)
   • Show a title for properties window (instead of just the moz-extension:// URL)
   • Make faster (so doesn't take 5-60 seconds to show like now)
   • Fix so doesn't slow down Firefox while waiting for it to open or even while its show, as often does now.
   • Make faster to show when doing New Bookmark or New Folder in BSP2, since often seems to fail to show or almost fail to show (not appearing until up to 60 seconds later or only showing a 2 duplicate windows after I right click > Properties for the new unnamed bookmark/folder in BSP2)

Screenshot of Issues

You can see in the below screenshot how few top-level bookmarks and folders I have. You can also see, as the only bookmark under "Other Bookmarks" in BSP sidebar, yet another frequently encountered bookmark which remains out-of-sync, shown under "Other Bookmarks" (as always occurs for those, despite it never having been in that folder at any point) even after trying to right click > Delete Bookmark it in BSP Sidebar and even after the bookmark was deleted via Ctrl-D (as you can see with Star icon not light up, for the active tab opened for that URL):

Related Issues

• 97 BSP2 can not delete a bookmark

My Configuration

• Win10 x64 with 16GB RAM
• Firefox Nightly 68.0a1 (2019-05-18) (64-bit)
• Optimized/defragged places DB
• Bookmark Stats (from BSP Options page)
  • Bookmarks: 40018
  • Favicons to fetch: 0 (0.0%)
  • Folders: 2600
  • Separators: 0
  • Oddities: 0
• BSP Options (seen in below screenshot):

[aaFn]

Hello @PowerAccess , wow long post .. I believe you have the record :-)

Not sure I will be able to address all in one response, let me try a few essential things first:

Performance topic:

• Old behavior of immediate favicon display = means that favicons are put in the sidebar HTML from start of writing the DOM tree contents.
  • This is slowing down the display operation because FF has more things to do at once
  • When this option is not set (default), contents is written without favicons first (= blank), and then in background they are set. This speeds up the initial display a little because FF has less things to do.
• Switching sidebars = this clears off the DOM tree (HTML) page of the sidebar. Natural FF behavior, nothing can be changed on that. When showing the sidebar again, have to write again the DOM tree to FF.
• Caching the DOM tree itself = bad idea, things can change on the bookmarks while the front UI BSP2 sidebar code is not displaying and not running. And it won't know it has changed, so it has to be rebuilt each time BSP2 sidebar is activated. Plus there are as many trees as there are FF windows instances running. And this cannot be stored in background page for windows in private mode, they do not have access to the background page.
• Anyway, the DOM tree is very fast to build from my measurements, so that's normally not it.
• The problem is more the display operation by FF itself, so whatever the method to give the DOM tree to FF, I do not expect much difference.
• One way to know more precisely is that you should activate the "Traces"option. Then on next BSP2 sidebar display, get the contents of the traces box and put it here
  • You will find a ""Display duration:" line in there, giving the duration of phase of giving the DOM tree + displaying it (did not find a good way to dissociate both so far)

Strange

• Switching to a different sidebar should not have the 20s delay / lock since FF stops the front code when you do that.
• Same for right click and making the menu appear for editing things. Again There might be something else going on .. how does your browser console contents looks like when you do that, are there tons of messages appearing in there, and can you share here ?

Display strategy

• Displaying only open folders and inserting lines in the displayed HTML table when opening folders is a thing I thought of as my last step of display optimization #2, but this is a lot of code and I do not have the time for that for now
• It will have to wait, sorry for that.

Bookmarks in Other bookmarks

• This is how FF does it. Remember that BSP2 is simply displaying real time what FF does.
• When FF creates a new bookmark using the Ctrl-D key, or star in the address field, it immediately creates the bookmark, and it places it in the Other bookmarks folder.
• BSP2 gets notified of that and displays as instructed to do
• When you change the fodler of the bookmark, FF instructs BSP2 to "Move" it to another folder

Desync

• This started with FF 66 as much as I can tell, but BSP2 code didn't change on that.
• So it is hard to get a good feeling at what is going on here.
• From what I see, this is because FF doesn't save to disk what it told BSP2 to do, for some reason.
• So on next restart, the thing is lost in FF, but not in BSP2
• Problem is that add-ons do not have access to the places DB .. so no easy way to detect a problem
• I didn't find a good strategy for that new FF problem.
• And I am not experiencing it myself, I cannot really try / investigate .. etc ..
• Maybe, ... if you could share with me your bookmark DB ? (do an export to HTML format) Are you ready to share with me ? That could help trying to get the problem myself and investigate / develop workarounds for this new FF problem ..

Miscellaneous

• Keyword, Description, and also Tags .. are not accessible by add-ons today, so BSP2 cannot do that
• BSP2 is mimicking the native FF bookmark sidebar behavior, or trying to stay close to it, not to improve it on UI or to make different choices. So all your ideas of inline editing and other are very good, but not matching the design point of BSP2.
• Remember that the goal of BSP2 is to disappear when FF provides a real search function showing their place in the tree. => BSP2 does not intend to be a bookmark manager, but to provide a missing (and esssential to me) function of what should be the bookmark sidebar :-)

I am sure I missed many of what you say. Can we do this first round though ? Thanks, aaFn

[RiverRhine]

Wow, thanks for that detailed description PowerAccess and your reply aaFn!

I recently ran into similar performance issues and wanted to document it as well. My JSON FF bookmark export is ~11MB. I can hardly use the extension anymore because of ~20sec freeze of UI as described by PowerAccess. Extension used to work great and much faster before some (automatic) updates of FF and plugins. So don't know when it started: maybe ~4 weeks ago?

Even adding a new folder, or draging a link into the BSP2 UI-tree takes 10-30 seconds.

But anyway: THANKS! Since "Show parentfolder" died, this extension must have saved countless peoples nerves ;-).

---

System:

• Win7 x64 U
• i7-3740QM @ 2,7GHz 32GB RAM
• FF 67.0.2 (64-Bit)
• 1TB SSD

Bookmarks: 28710 Favicons to fetch: 0 (0.0%) Folders: 5199 Separators: 8 Oddities: 0

[aaFn]

Hello @RiverRhine , thank you. As a first step, could I ask you to:

• Activate the "Traces"option in the BSP2 add-on options page.
• Then click on the yellow star+ icon twice in the toolbar to hide the BSP2 sidebar and then to re-display it
• Let me know your perception of the elapsed time of appearance of the sidebar after the second click on the yellow star+
• Get the contents of the traces box and put it here (we will compare it with your perception)

If you could repeat it 3 times to see if there are variations.

[RiverRhine]

Hi aaFn, ok, here are the results:

1st hide+show Bookmarktree:

Load local store duration: 59 ms Background load local store duration: 5160 ms Background bypassed FF API for tree load Background tree load duration: 64 ms Background tree build duration: 62 ms Background save duration: 14968 ms Get tree duration: 1 ms PlatformOs: win Setting Windows variations structureVersion: -bnlist-spfldr-img16 disableFavicons_option: false pauseFavicons_option: true Display duration: 4833 ms Total duration: 4893 ms fontFamily = '"Segoe UI"' fontSize = '12px' Stats: Bookmarks: 28782 Favicons to fetch: 39 Folders: 5191 Separators: 8 Oddities: 0

BSP2 version: 2.0.62

2nd run:

Load local store duration: 53 ms Background load local store duration: 5160 ms Background bypassed FF API for tree load Background tree load duration: 64 ms Background tree build duration: 62 ms Background save duration: 14968 ms Get tree duration: 1 ms PlatformOs: win Setting Windows variations structureVersion: -bnlist-spfldr-img16 disableFavicons_option: false pauseFavicons_option: true Display duration: 4335 ms Total duration: 4389 ms fontFamily = '"Segoe UI"' fontSize = '12px' Stats: Bookmarks: 28782 Favicons to fetch: 39 Folders: 5192 Separators: 8 Oddities: 0

BSP2 version: 2.0.62

(I tested some more, but I saw no relevant variations in timing)

… now create and name new folder (takes ~23 sec to finish):

Load local store duration: 40 ms Background load local store duration: 5160 ms Background bypassed FF API for tree load Background tree load duration: 64 ms Background tree build duration: 62 ms Background save duration: 14968 ms Get tree duration: 1 ms PlatformOs: win Setting Windows variations structureVersion: -bnlist-spfldr-img16 disableFavicons_option: false pauseFavicons_option: true Display duration: 4132 ms Total duration: 4173 ms fontFamily = '"Segoe UI"' fontSize = '12px' Stats: Bookmarks: 28782 Favicons to fetch: 39 Folders: 5194 Separators: 8 Oddities: 0

BSP2 version: 2.0.62

In general UI is very slow. Highligthing a row in the tree under the mouse is delayed. The whole process of creating and naming a sub-folder takes ~25 seconds (!)

Thank you

[aaFn]

Indeed, something is slowing you down, I see:

• Background load local store duration: 5160 ms
• Background save duration: 14968 ms
• Display duration: around 4 s

The first two are really not normal, especially considering that you have an SSD. The third one also looks long given you have an i7.

Could you open a new tab, and type in the address bar: about:performance ? Then, if you have a way to copy the screen here, that would be great.

Also, by chance, did you look at your task manager to see if there is not a CPU hog somewhere ?

[RiverRhine]

performance data: BR

[aaFn]

Ok, so something is definitely slowing BSP2 in your config, and it could be coming from inside it, although it's not supposed to keep cpu as all actions are short lived (I suppose Hoch means High ..).

Let's continue digging if you agree .. :-)

1. Could you look at the Windows Task manager (click on show details then on the processes, and then expand Firefox to see its threads), and place here the captured image of the firefox + its threads line ?
2. Could you open the browser console (Ctrl+Shit+J), click on the bin icon on top left to empty it, then do an operation on BSP2 which is slow like create a sub-folder, come back to the browser console and copy its contents here (text copy and paste) ?

Hopefully, that will allow me to catch something.

Thank you in advance.

[RiverRhine]

Love to help. Welcome!

Feedback:

although it's not supposed to keep cpu as all actions are short lived

Info: Yes, but I made the snapshot while/after opening BSP2. During the ~20sec UI locks and CPU high.

Red arrow = Close/open BSP2 tree:

Could you look at the Windows Task manager (click on show details then on the processes, and then expand Firefox to see its threads), and place here the captured image of the firefox + its threads line?

Seems to be different in W7Ultimate64. I used ProcExp:

and

FF was in idle above.

Could you open the browser console (Ctrl+Shit+J), click on the bin icon on top left to empty it, then do an operation on BSP2 which is slow like create a sub-folder, come back to the browser console and copy its contents here (text copy and paste) ?

Create new folder -> Browser console output:

as text:

TypeError: this.appMenuStatus is undefined browser-sync.js:364:24 updatePanelPopup chrome://browser/content/browser-sync.js:364 updateAllUI chrome://browser/content/browser-sync.js:197 maybeUpdateUIState chrome://browser/content/browser-sync.js:110 showFxaToolbarMenu chrome://browser/content/browser.js:486 onDOMContentLoaded chrome://browser/content/browser.js:1444 onDOMContentLoaded self-hosted:985 this._box is null browser-sidebar.js:328 get isOpen chrome://browser/content/browser-sidebar.js:328 get currentID chrome://browser/content/browser-sidebar.js:335 onWindowOpen chrome://browser/content/parent/ext-menus.js:994 handleEvent chrome://extensions/content/parent/ext-tabs-base.js:1592 editor is null UrlbarValueFormatter.jsm:303 _formatSearchAlias resource:///modules/UrlbarValueFormatter.jsm:303 update resource:///modules/UrlbarValueFormatter.jsm:59 formatValue chrome://browser/content/urlbarBindings.xml:584 _setValueInternal chrome://global/content/bindings/autocomplete.xml:598 set_value chrome://global/content/bindings/autocomplete.xml:282 URLBarSetURI chrome://browser/content/browser.js:2794 onLocationChange chrome://browser/content/browser.js:5021 callListeners chrome://browser/content/tabbrowser.js:746 _callProgressListeners chrome://browser/content/tabbrowser.js:759 _callProgressListeners chrome://browser/content/tabbrowser.js:4908 onLocationChange chrome://browser/content/tabbrowser.js:5244 _callProgressListeners resource://gre/modules/RemoteWebProgress.jsm:156 receiveMessage resource://gre/modules/RemoteWebProgress.jsm:295 startObserving moz-extension://dde5f1c4-8d49-4884-b363-30b001058176/privileged/FirefoxMonitor.jsm:247 _onOpenWindow moz-extension://dde5f1c4-8d49-4884-b363-30b001058176/privileged/subscripts/EveryWindow.jsm:48 _onOpenWindow self-hosted:989 _delayedStartup chrome://browser/content/browser.js:1721 _delayedStartup self-hosted:985 editor is null UrlbarValueFormatter.jsm:303 _formatSearchAlias resource:///modules/UrlbarValueFormatter.jsm:303 update resource:///modules/UrlbarValueFormatter.jsm:59 formatValue chrome://browser/content/urlbarBindings.xml:584 onSecurityChange chrome://browser/content/browser.js:5175 callListeners chrome://browser/content/tabbrowser.js:746 _callProgressListeners chrome://browser/content/tabbrowser.js:759 _callProgressListeners chrome://browser/content/tabbrowser.js:4908 onSecurityChange chrome://browser/content/tabbrowser.js:5271 _callProgressListeners resource://gre/modules/RemoteWebProgress.jsm:156 receiveMessage resource://gre/modules/RemoteWebProgress.jsm:312 startObserving moz-extension://dde5f1c4-8d49-4884-b363-30b001058176/privileged/FirefoxMonitor.jsm:247 _onOpenWindow moz-extension://dde5f1c4-8d49-4884-b363-30b001058176/privileged/subscripts/EveryWindow.jsm:48 _onOpenWindow self-hosted:989 _delayedStartup chrome://browser/content/browser.js:1721 _delayedStartup self-hosted:985 Unchecked lastError value: Error: ID already exists: open-link-in-new-temporary-container-tab ExtensionCommon.jsm:679 withLastError resource://gre/modules/ExtensionCommon.jsm:679 create chrome://browser/content/child/ext-menus.js:138 Tastenereignis ist in manchen Tastaturlayouts nicht verfügbar: Taste="i" Modifikatoren="accel,alt,shift" ID=“key_browserToolbox” browser.xul Cannot send function call result: other side closed connection (call data: ({path:"bookmarks.update", args:["kZiqxitKj5Ai", {title:"dbg", url:null}]}))

Info 1: No enties in browser console while show/hide/show BSP2 Info 2: Black magic, very strange!: First time after opening the Browser Console, the tree was fast and responsive for ONE create of a subfolder. Slowed down again afterwards (don't laugh pls ;-))

Tested : Disabled this plugins temporarily: and

... but doesn't make a difference. (have to go offline now till tonight ... I continue then.)

[aaFn]

Info 2: Black magic, very strange!: First time after opening the Browser Console, the tree was fast and responsive for ONE create of a subfolder. Slowed down again afterwards (don't laugh pls ;-))

Ouch .. that makes me suspect that the save to disk is the problem as it is lasting 14s ... and there's one each time you make a modification to the tree (normal, we do not want to lose its state). This is usually very fast and goes unnoticed, but when being that long that must impact normal operations.

As a matter of fact, ever since FF 66 where Firefox migrated the add-on local storage to IndexedDB, I have seen random problems appear for users with big numbers of bookmarks. Unfortunately, there is no easy way to come back to the old mode once it has been migrated by FF automatically. See: https://www.mozilla.org/en-US/firefox/66.0/releasenotes/ https://blog.mozilla.org/addons/2019/02/15/extensions-in-firefox-66/ https://blog.mozilla.org/addons/2018/08/03/new-backend-for-storage-local-api/ https://www.reddit.com/r/firefox/comments/ar78wr/firefox_extensions_to_use_different_storage_type/egmdkul/ https://bugzilla.mozilla.org/show_bug.cgi?id=1406181

And there's nothing else in the trace that reveals anything meaningful :-(

One way to verify if my suspicion is right would be to:

• After restart and waiting all things calmed down (like wait a few minutes doing nothing with FF, verify on Windows Task Mgr that cpu is idle .. etc ..)
• Do one of the things which can be slow
• Verify it is going fast this time
• Wait again all things calm down (and at least 14s or more ..)
• Do another one

If the last one is fast again, we're in the problem that stores are long (and also read).

14 seconds anyway is a lot !!

I am not sure how that can be optimized, I do not know IndexedDB's in Firefox, and I read that the sqlite form used for add-on local storage is not easily readable / controllable.

Maybe a simple naive question:

• Where is your Firefox profile stored ?
• And is there free space room on that drive ?
• Maybe do a cleanup (with system files) and a defrag on that drive ... (optimize disk)

To find where your profile is, do the following:

• Type %appdata% in the address bar of a new File explorer window
• Go to Mozilla\Firefox directory
• Open the profiles.ini file
• You should see inside the path to your Firefox profile data
• If IsRelative=1, the path starts from that directory, if IsRelative=0, the path is absolute and shows it in clear

I do not know where the add-on data is stored anymore, it was in the browser-extension-data directory in the past, but now it's somewhere else I do not know, however I presume it is still in the profile directory

[RiverRhine]

Back again. Pulling things together to provide the details ...

Yesterday FF updtaed to latest (67.0.4 (64-Bit)). It seems to bee a little faster now ?!

Ouch .. ... I do not know where the add-on data is stored anymore, it was in the browser-extension-data directory in the past, but now it's somewhere else I do not know, however I presume it is still in the profile directory ... since FF 66 where Firefox migrated the add-on local storage to IndexedD ...

Ouch=ACK. A devs nightmare losing control about storage ;-). At least Moz had good intentions (perf+sec)) Anyway, going with Moz and FF through it's lows and highs for all the years ;-).

Digging into it:

After restart and waiting all things calmed down (like wait a few minutes doing nothing with FF, verify on Windows Task Mgr that cpu is idle .. etc ..) Do one of the things which can be slow Verify it is going fast this time Wait again all things calm down (and at least 14s or more ..) Do another one

Done. Result: 1st, second and so on are about the same. (can't reproduce the "once fast" ?!)

Starting here: FF idle, github page and BSP2 options page open:

My BSP2 options:

Operation profiled: "create subfolder (~5sec), type new folder name (~2 sec)+enter, folder created (~2sec)"

during operation:

... after operation:

Traces: (After folder operation)

Load local store duration: 396 ms Background load local store duration: 1834 ms Background bypassed FF API for tree load Background tree load duration: 82 ms Background tree build duration: 12 ms Background save duration: undefined ms Get tree duration: 2098 ms PlatformOs: win Setting Windows variations structureVersion: -bnlist-spfldr-img16 disableFavicons_option: false pauseFavicons_option: true Display duration: 1604 ms Total duration: 4098 ms fontFamily = '"Segoe UI"' fontSize = '12px' Stats:

Bookmarks: 28805 Favicons to fetch: 105 Folders: 5210 Separators: 8 Oddities: 0

BSP2 version: 2.0.62 Favicon display duration: 8058 ms

FF console (started fresh ):

Conclusion: "Somehow" faster after last update (strange coincidence?) but still slow.

To your suggestions/questions:

Where is your Firefox profile stored ? And is there free space room on that drive ? Maybe do a cleanup (with system files) and a defrag on that drive ... (optimize disk)

Prog+profiles all live on C=SSD. SSD-performance:

I do not know where the add-on data is stored anymore, it was in the browser-extension-data directory in the past, but now it's somewhere else I do not know, however I presume it is still in the profile directory

... yepp, anything else would be insane ;-).

BTW, I made an other test on my Macpro (SSD as well): Found an older FF version (ups, don't rember version) there!: BSP2 lighning fast like ever. After update to FF current: "Folder create" takes ~2 sec.

hmmm. Where to go next?! BR and thx for your work!

[aaFn]

Thanks for the very detailed investigation.

Yes the Samsung 850 EVO is a good beast, I have two of them, speed like hell :-) And it looks like you have plenty of space, so that's not the problem indeed (was asking just in case ..).

Recurring schema, I see like last time two errors on moz-extension://dde5f1c4-8d49-4884-b363-30b001058176 inside the browser console trace.

To know which extension this is:

• Click on Tools -> Add-ons, go to the little "parameter wheel" button, click on it to get a menu and then click on "Debug add-ons"
  • This will bring you to a detailed list of modules with their internal UUID's
  • You should find there the one with dde5f1c4-8d49-4884-b363-30b001058176 by searching on dde5f1c4
• Another way: open a new Tab, type "about:memory" in its address field
  • Then click on the "Verbose" checkbox for "Show memory reports, and then on "Measure"
  • After a few seconds, you'll get a very detailed report about the heap
  • Search for the string dde5f1c4, and you should find a few lines saying to you which add-on this is
• Then, just in case, see if deactivating it would solve the problem ?

---

However, your last experiment does point at something related to FF version since this is the only thing varing between he two attempts you made, before and after the FF upgrade :-(

• And the fact that it indeed appears to have divided by 2 or more from around 4s to 1.8 s with 67.0.4 upgrade goes in the same direction
  • Notice also how the display duration improved from a steady 4s in your previous traces to 1.6s this time
• On my side, I compared save and load performances between now and the past, and they are roughly the same,
  • But I only have around 2100 bookmarks, so not at the same scale than you
• I retrieved a Win10 VM I had with around 28000 bookmarks I had artificially created by script (basically all the same) for scalability tests purposes
  • It was at FF 65 and BSP2 2.0.52 .. all fine and fast when creating a folder, small spurt in cpu but nothing bad
  • Moved to BSP2 2.0.62 .. all the same
  • Upgraded FF to 67.0.4 .. BAM .. got an increase in save from 2s to 16s !!
  • And each time I create a folder, same symptom than you, some time before I get the creation window, and a long burst of CPU on FF between 90 and 100% !!

So that's defnitely it, somewhere, most probably with FF66 and the way they changed how add-ons data is stored internally, they degraded the function :-(

I am not sure what to do from there.

• I can of course try to find save strategies which are better with the new FF versions, but that will create a disruption in my save format, with problems for migration to handle, and the fact that it could be better with newer FF versions, but not with older ones pre-66, and so need to manage two formats at the same time :-(
• Or I could wait a little to see if FF improves .. the recent improvement in FF 67.0.4 is encouraging

For now, I'd prefer to stay in an observation period if you agree. Being over-reactive while FF is stabilizing after their 66 version big change is probably not wise, and a loss of time if I find some compromise whiich only lives a few days until their next update .. I'll think more meanwhile if something low cost can be made as workaround.

[RiverRhine]

Hi there! Thanks for the feedback and hints.

Click on Tools -> Add-ons, go to the little "parameter wheel" button, click on it to get a menu and then click on "Debug add-ons" This will bring you to a detailed list of modules with their internal UUID's You should find there the one with dde5f1c4-8d49-4884-b363-30b001058176 by searching on dde5f1c4

... didn't work. No such UUID (and extension) But

Another way: open a new Tab, type "about:memory" in its address field

Then click on the "Verbose" checkbox for "Show memory reports, and then on "Measure" After a few seconds, you'll get a very detailed report about the heap Search for the string dde5f1c4, and you should find a few lines saying to you which add-on this is

worked. What I found is not a visible extension, but this:

17 (100.0%) -- extensions
├───1 (05.88%) ── Extension(id={6d96bb5e-1175-4ebf-8ab5-5f56f1c79f65}, name="Deaktivierungs-Add-on von Google Analytics", baseURL=moz-extension://071d0795-02b6-4892-a956-89b4a0b7997c/)
├───1 (05.88%) ── Extension(id={c607c8df-14a7-4f28-894f-29e8722976af}, name="Temporary Containers", baseURL=moz-extension://c9a3f5ff-bfd9-40ac-ac5e-7f792388c9c5/)
├───1 (05.88%) ── Extension(id={e824f0e4-72f7-4295-8879-d8ff4bf348d2}, name="Invidition", baseURL=moz-extension://35ed773c-4b12-4c1a-86e0-10fc3c2a094e/)
├───1 (05.88%) ── Extension(id=@testpilot-containers, name="Firefox Multi-Account Containers", baseURL=moz-extension://baa727c9-9294-41d0-87a7-735e6823769a/)
├───1 (05.88%) ── Extension(id=bookmarksearchplus2@aafn.org, name="Bookmark search plus 2", baseURL=moz-extension://c276880d-bc74-4ff2-9e27-6f1bba801d99/)
├───1 (05.88%) ── Extension(id=formautofill@mozilla.org, name="Form Autofill", baseURL=moz-extension://df6ea9b2-0815-4e76-9cb4-3e18ff94e1a9/)
**├───1 (05.88%) ── Extension(id=fxmonitor@mozilla.org, name="Firefox Monitor", baseURL=moz-extension://dde5f1c4-8d49-4884-b363-30b001058176/)**
├───1 (05.88%) ── Extension(id=hotfix-update-xpi-intermediate@mozilla.com, name="hotfix-update-xpi-intermediate", baseURL=moz-extension://13771935-216b-4440-a0c0-7d7dffe69f1e/)
├───1 (05.88%) ── Extension(id=https-everywhere@eff.org, name="HTTPS Everywhere", baseURL=moz-extension://dd58bf0c-209d-4678-bf36-4ffd90e62dfa/)
├───1 (05.88%) ── Extension(id=jid1-BoFifL9Vbdl2zQ@jetpack, name="Decentraleyes", baseURL=moz-extension://dfc646a2-51d1-4a19-9969-6502fa20242b/)
├───1 (05.88%) ── Extension(id=jid1-MnnxcxisBPnSXQ@jetpack, name="Privacy Badger", baseURL=moz-extension://cc108c4c-52ec-474b-a2df-61600874e87c/)
├───1 (05.88%) ── Extension(id=jid1-TMndP6cdKgxLcQ@jetpack, name="Mate Translate – Übersetzer, Wörterbuch", baseURL=moz-extension://d4afde02-d117-41b1-85df-934dd163ec8f/)
├───1 (05.88%) ── Extension(id=jsonview@brh.numbera.com, name="JSONView", baseURL=moz-extension://a5aac941-9cd3-4f66-bf17-1201f5bd6229/)
├───1 (05.88%) ── Extension(id=screenshots@mozilla.org, name="Firefox Screenshots", baseURL=moz-extension://6d0bd23f-c79b-4e25-abc3-3d61576bd57f/)
├───1 (05.88%) ── Extension(id=uBlock0@raymondhill.net, name="uBlock Origin", baseURL=moz-extension://b85953fb-6321-485b-9907-ced13d92c69b/)
├───1 (05.88%) ── Extension(id=wayback_machine@mozilla.org, name="Wayback Machine", baseURL=moz-extension://2fc2a9c5-5a4d-484e-bcf8-811988ba1456/)
└───1 (05.88%) ── Extension(id=webcompat@mozilla.org, name="Web Compat", baseURL=moz-extension://e71a1d4e-0911-48a1-9347-d999a015ef8c/)

with ├───1 (05.88%) ── Extension(id=fxmonitor@mozilla.org, name="Firefox Monitor", baseURL=moz-extension://dde5f1c4-8d49-4884-b363-30b001058176/)

... no idea what that is.

All occurences of that for reference. Just in case that makes sense to you:

So that's defnitely it, somewhere, most probably with FF66 and the way they changed how add-ons data is stored internally, they degraded the function :-(

ACK.

...find save strategies which are better with the new FF versions ,,, ... manage two formats at the same time :-(

Don't do that ;-). What a mess adding a second storage abstraction and migration layer!

For now, I'd prefer to stay in an observation period if you agree. Being over-reactive while FF is stabilizing after their 66 version big change is probably not wise, and a loss of time if I find some compromise whiich only lives a few days until their next update .

100 ACK, but on thing: Couldn't you maybe point the MozDevs to "that storage performance situation" so that they can test and improve the storage backend with that information? Then it's a little biased and not a random walk for their improvements ;-).

Anyway: You created one of the "couldn't live without" - extensions here! Must have been "some hours ..."

As you said: Let's wait some "versions of FF" and come back to this, if things don't improve or get worse. If you update the FF devs, put me in that loop pls.

BR Andreas

BTW: In which timezone you are in?

[aaFn]

Hello @RiverRhine,

On Firefox monitor, I found this: https://blog.mozilla.org/futurereleases/2018/06/25/testing-firefox-monitor-a-new-security-tool/ https://winaero.com/blog/mozilla-enables-firefox-monitor-extension-firefox-67/ https://monitor.firefox.com/

I am in the same timezone as you I believe :-) CET / CEST https://www.timeanddate.com/time/zones/cet

I thought of a quick workaround, not curing the problem, but moving it so that it is less painful hopefully:

• If I observe that the save duration is significant (let's say more than 2s)
• Then introduce a delay on a save request, so that it isn't executed imlmedaitely and it doesn't interfere with actions being run at that moment
• Like wait with a timer of 10s before really executing the demanded save
• Also, maybe restart that timer to 10 s again if another demand for save happens during that delay period

Drawback would be a small chance of losing something in the BSP2 cache if the user exits from Firefox before a triggered 10s timer for save completely elapsed and the save could really occur. Hence the delay value should not be too big. And to limit such a drawback to a minority of users, only do that when I see the save time is long on their config.

Advantage should be that the save wouldn't perturb anymore immediate ongoing actions (only later ... with a chance the user doesn't see it because he stopped interacting).

How does that sound ?

[RiverRhine]

Hi @aaFn, thx for the info about "monitor". An OK approach. But I hope Moz doesn't go the way to far to overcrowd/overfeature everything ...

Yes, CET here as well ;-)

Your suggestion sounds great: It would decouple the UI Interactions from the book keepings/sync in the background. The risk of loosing some links is def. worth a more fluent and nonblocking UI. The result would be a "real time UI tree" again, right?

Would love it!

[aaFn]

The result would be a "real time UI tree" again, right?

That is the purpose yes .. let me try then.

[RiverRhine]

Great. Small suggestion (and just a guess): There should be an event/hook for FF exit/termination. To avoid any data loss this could probably also call the "store/sync to disk" function additional to the call from the timer? Might be just some more lines? But: not a dev anymore, so forget pls if nonsense ;-)! Thx for your efforts again.

[aaFn]

2.0.63 is out with the workaround to restore UI responsiveness. It appears to work on my tests, try on your side.

As said above, save to local store is delayed by 10s after last action on the UI, meaning that each action on the BSP2 UI restarts the 10s timer delaying the save.

I didn't find anything to hook for FF stop, the closest event would be browser.windows.onRemoved (https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/API/windows/onRemoved), however it is triggered when any window is closed, including the property one for editing bookmarks, so I decided not to use it as an indication of possible FF close to force a save.

Impact, if a "large number of bookmarks" user with "slow save" (> 2s) closes before the 10s elapse, is that BSP2 will be de-stnchronized from Firefox on next restart, loading a state corresopnding to the last save which occurred. Remedy for such rare cases will be to hit the Reload all bookmarks from FF now button in the BSP2 options page, so nothing will be lost. Looks like an acceptable drawback, which will hit anyway only a few user base, but for the best of coming back to UI responsiveness :-) (hopefully)

I also suspect that this long and strong cpu consumption by FF on saves with large numbers of bookmarks is at the origin of some of you guys seeing bookmark modifications occasionally not saved by FF in its internal DB, resulting in loss of work when you reload from the internal DB. At least, the phenomenon also started to appear fi FF66 and the new Indexed DB format on local storage, so that should have a high correlation factor. With delayed save by 10s, I am hoping this won't occur anymore.

@RiverRhine, @PowerAccess, let me know how that goes. Cheers, aaFn.

[RiverRhine]

Man, that's it. Works like a charm here! "New folder test + drop link to it": There is still a ~1-2sec delay until the folder name dialog opens. Which is absolutely ok for me and no issue. After a while (?10s) the "heavy cpu" kicks in: Best to be seen in core 6 and 7 above. First spike: Dialog+ typing. Second spike: You know the magic..

Didn't test if UI is blocked then, I guess not.

Anyway: green light from my side. Many thanks and keep BSP2 up ;-).

Btw: I didn't see loss of data so far, but will have an eye on it in future.

[aaFn]

Nearly 2 weeks, it seems we are good now with the workaround :-) Closing the issue, let me know if something comes back.

[RiverRhine]

Still all green lights frome here. Again: Thx!