On the doorstep sat his old woman, With the same broken wash-tab before her

[vitaaliy]

Hi aaFn, hi all, Just feeling like the Old Man from the famous Russian folk tale "Fisherman and little fish". Things went worse probably when I installed FF v. 60.0.1. 64 bit. FF loading became very tiresome... too long, chewing almost all 8 Gb of RAM in the saw way: increase - decrease (while I remain in the standard FF bookmarks mode). When I need the help of BSP-2 and switch it on - the thing goes into endless loop - Loading bookmarks. 5... 10 minutes pass and nothing happens. The only way out I found - delete BSP-2 from Extensions and then reload it anew. So it works only one time after installation. Any hints?

[aaFn]

Hello @vitaaliy , not sure .. While you are on the standard FF bookmark, BSP2 front end is not running, so this saw thing shouldn't happen.

Now from our last discussion on #5, I get it that you were somewhere still in the process of migrating all icons to 16x16 mode. I suppose you didn't have the occasion to let it run alone during a night, or this is crippling because things are too frequent and queueing .. so somehow something was still triggering it, or there is too much in queue and that takes a long time to dequeue ?? It shouldn't, but supposing something was still happening, the best plan B thing to do was indeed what you did = uninstall and reinstall. This has cleared all local storage and so cut any background migration task of favicons.

Now, what do you mean by

it works only one time after installation

?

Is it that the "saw phenomenon" is happening again ? FF 60 certainly did things in the bad direction on my system also, but that was solved with 60.0.1. So that's not it.

I can only think that now you are in the long loop of getting the favicons from fresh, which is long because of the sheer number of bookmarks you have. By the way, I was never sure of why you didn"t have the saw phenomenon in the first place when you initially loaded BSP2 ... that memory problem is inherent to the fetch() and storage things as I could see it = they are creating that memory problem when used intensively, and I didn't find a way to get rid of that so far, I could only decrease it by spacing the requests .. but I guess on your system they are still too close to each other

The best way to cut it out is to:

• go to the options page of BSP2
• tick the Disable favicon fetching option
• stop and restart BSP2 (like twice Ctrl-Q, or click twice on the button in the toolbar if you still have it)
• stop firefox to remove any queued storage and fetch requests
• restart firefox after it has been completely stopped (check on the Windows task list to make sure ..)

That should stop the crazy saw thing .. The only drawback is you won't have the fancy favicons, but you do not care as the main usage of BSP2 you have is to search your bookmarks and lost subtree parts .. :-)

Let me know how that goes, or more details if still problems. Maybe activate the traces and show here what you get in the trace textarea ... and also show the contents of the Browser console (Ctrl+Shift+J).

[vitaaliy]

Now, what do you mean by it works only one time after installation ?

Alas, it is the key predicament. The life cycle of my BSP2 installation is from the moment of its downloading and using to find the bookmark needed. If I return to the standard FF bookmarks the second try to switch BSP2 on plunges into the endless loop loading the bookmarks. It means that if I really try to use BSP2 once more the whole magic should be repeated: [deleting - downloading - using until the standard bookmarks call]. So, you see, I am far from the latest bells and whistles about favicons. Deep distress, really...

[aaFn]

In one of your cycles, once BSP2 is started just after install, can you do what I mentioned above, in that specific order:

1. go to the options page of BSP2
2. tick the Disable favicon fetching option
3. tick the Activate traces option
4. Open the browser console (Ctrl+Shift+J)
5. stop and restart BSP2 (like twice Ctrl-Q, or click twice on the button in the toolbar if you still have it)

and post here what you see in the trace window, and in the browser console ?

[vitaaliy]

Opening the browser console gives the following endless sequence:

code:1:1104 Console was cleared.code:1 code:1:1104 Console was cleared.code:1 code:1:1104 Console was cleared.code:1 code:1:1104 … 5. Stopping and restarting BSP2 led to: PlatformOs: win Setting Windows variations Browser Console was going on printing that same dull sequence. Though other BC windows were displaying a lot of diagnostic info. I attach the output at "Logging" stream. Where to find "Traces" results is not clear... [Logging.txt](https://github.com/aaFn/Bookmark-search-plus-2/files/2041562/Logging.txt)

[aaFn]

Thank you. Strange, first time I ever see that sort of error in the browser console, it doesn't seem to make any interpretable sense.

The only thing I can deduce from the above is that it appears the load from local store is not completing when restarting .. and FF API is not returning to BSP2. This is very early in the BSP2 start process. I wonder if you do not have a cycle in your bookmark tree ...

Can I ask one more thing then:

1. Recycle again by uninstall / reinstall
2. Open the browser console (Ctrl+Shift+J)
3. Make sure that in the browser console, the following traces are activated (click on the little down arrow at right of each label to verify / set the checkmarks):
   • Net: Errors and Warnings (not XHR, not Log)
   • CSS: Errors only (not Warnings, not Reflows)
   • JS: all checked
   • Security: all checked
   • Logging: all checked
   • Server: all checked
4. Do a right click on a bookmark in BSP2, and click on "Refresh favicon"
5. Add a new bookmark anywhere in the tree with BSP2 (like right click + copy / paste an existing one)
6. Do a folder Open / folder Close in BSP2
7. Stop BSP2 by going back to native bookmarks

Send me the result of the browser console. If there is a cycle in the bookmarks tree, I believe when saving the things to local storage something should yell .. and the three actions above from 4 to 6 should each trigger a save.

....... Note: I have a Win10 VM with FF 60.0.1 and about 75000 bookmarks, and all goes fine (just slow because this is a VM - do not pay attention to the load duration, I froze the VM that night and restarted it this morning, its more like 20 minutes usually) .. Here is the trace for reference .. Between "**" the first line that is missing from your trace, and which appears when local storage load is complete:

PlatformOs: win
Setting Windows variations
**structureVersion: -img16**
disablefavicons_option: false
Load local store duration: 828 ms
BSP2 version: 2.0.20
Tree load duration: 46323053 ms
Tree build duration: 6429 ms
Display duration: 43092 ms
fontFamily = '"Segoe UI"'
fontSize   = '12px'
Stats:
------
Bookmarks:  59222
Folders:    15184
Separators: 1
Oddities:   0
--------------------
Save duration: 71014 ms

[vitaaliy]

No wonder. All my life I stumble upon exotic software system errors. That is my karma. Probably I had to inform you in advance ;). Now to your prescriptions. Experiment May, 26.zip

[aaFn]

Thank you @vitaaliy . Nothing really significant in all this :-(

Let's go the detailed way if you agree .. I will make the bet there is something wrong in your bookmark structure .. and this is not showing until we reload it from local storage when BSP2 is starting a 2nd time. The only thing I believe can make things wrong is if you have "duplicate" bookmarks, so I will look for it.

Here below you will find a zip file of a debug version. Traces are force-activated from start, more traces have been added, and favicons are force-disabled.

Here are instructions of what I propose you to do, if you can run that on your side in that exact order, and give me the results:

1. In FF, uninstall BSP2 if you have it installed.
2. Stop FF, wait for the process to disappear.
3. Download the file below and unzip its contents to some temporary directory = it contains the debug version of BSP2 I mentioned above.
4. Start FF
5. Open the Browser console (Ctrl+Shift+J)
6. Make sure as before the following traces are activated by clicking on the little down arrow at right of each label to verify / set the checkmarks:
   • Net: Errors and Warnings checked (but not XHR, not Log)
   • CSS: Errors only checked (but not Warnings, not Reflows)
   • JS: all checked
   • Security: all checked
   • Logging: all checked
   • Server: all checked
7. Empty the browser console (click on the trash bin)
8. Open the Add-ons manager
9. Click on the Parameter/config wheel ("Tools for all add-ons" when you hover the mouse on it), and inside that, click on "Debug add-ons"
10. A new window opens, with a "Load Temporary Add-on" button at top left
11. Click on that button, a dialog appears .. navigate to the directory where you unziped the trace version of BSP2, select "manifest.json" and press "Open"
12. The trace version of BSP2 will be loaded and will start
13. Wait for it to load the bookmarks and show you the bookmarks tree.
14. There should not be any memory phenomenon since favicon retrieval is disabled.
15. Look at the trace textarea, copy its contents (Ctrl-A to select all + Ctrl-C to copy) and send it back here.
16. Look at the browser console, copy its contents and send it back here.
17. Empty the browser console (trash bin ..)
18. Press Ctrl-Q to remove the add-on
19. Press Ctrl-Q to make the add-on reappear
20. Wait for it to restart and provide the bookmarks tree (or maybe hang .. we'll see)
21. Then look at the trace textarea, copy its contents (Ctrl-A to select all + Ctrl-C to copy) and send back here that second run trace.
22. Look at the browser console, copy its contents and send back here also that second run trace.

And let's see if we get something more meaningful from all that ... Thank you for your patience.

Here is the file -> BSP2-trace.zip

[vitaaliy]

The only thing I believe can make things wrong is if you have "duplicate" bookmarks, so I will look for it. Maybe it could save us a gigantic labor you propose. Yesssss... mea culpa, mea culpa, mea máxima culpa. I know that my structure does have duplicate bookmarks. I have done them with my own hands. Sometimes it looked more handy to have identical bookmarks at different branches of the tree structure. The native bookmarks FF machinery never objected it, Show Parent Folder 2.1.1 By Alice0775 - neither. Can it help you in pinpointing the problem? If no - I will follow the plan you proposed.

[aaFn]

:-) Yes, that explains .. Well, can you please still execute the protocol above and send the result ? That trace version should identify them and give us the extent of what you did. You should retrieve your own hands labor :-D

If you can paste the results here, that should then give us fuel on if there is a way to support it. I need to see the structure of what you have done ... and if you can explain a little more what was your intention in doing that ?

............... The problem is that in order to accelerate things and get around FF API limitations / slowness (see #5 and #2), I had to now since 2.0.18 build an internal tree structure representing exactly the bookmark tree.

So far so good, however, when saving that to disk to retrieve it later and accelerate loading in a future release (I am going step by step ...), this gets serialized in a json form by FF. Json serialization does not like cycles nor complex graphes ... which are bound to happen if there are duplicates, i.e. the same unique id appearing twice or more. Saving is ok as I observed, but reloading them .. wrecks things up in FF or in the add-on memory at least.

............... Also, I need to warn you ... in FF, the bookmark id is really meant to be unique. If it appears several times, some operations should fail or work awkwardly .. which could explain the parts of tree "disappearing" as I believe you were explaining in one post in the past.

For example, in FF API, there is only one parentId for each bookmark, see https://developer.mozilla.org/en-US/Add-ons/WebExtensions/API/bookmarks/BookmarkTreeNode. It also only has one index.

So if the same bookmark (i.e. same unique id) appears several times under different parents / different places (same parent or not), this can only have "unpredictible" effects ... working fine in display, but when you do operations on them, like moving them around or deleting (or searching ...), you get at best partial results, at worst things which become orphaned (no parent) or even worse, cannot be manipulated anymore.

[aaFn]

Also, if you can explain to me how you created those duplicates by hand ?

[vitaaliy]

Hi aaFn, "Also, if you can explain to me how you created those duplicates by hand ?"

• Very simple: I drag the current link from the FF address field and drop it straight into the chosen folder in the bookmarks sidebar. Due the large and complex structure of my B-tree it takes too long and needs many clicks if I use the routine procedure following all parent edges of the chosen node.

"and if you can explain a little more what was your intention in doing that ?"

• Sure. Hierarchical natural classification is oriented strictly at fixed viewpoint. Consider two hierarchies:
  1. Bob IS male, Alice IS female, male IS human, female IS human
  2. Bob IS engineer, Alice IS programmer, engineer IS profession, programmer IS profession.

If I need to fetch all attributes of Bob - in the given example I should scrutinize all present classification trees. It is too unwieldy. The best solution would be the complete graph (semantic net - in Artificial Intelligence) where each object would have unic representation - its designation. All properties and relations of this object would be localized around it. FF does not work with semantic nets - it gives the primitive bookmarks tree. That is why one has to plunge both given classifications into one tree - so our heroes: Bob and Alice would be represented more than once.

FF authors gave us the tag mechanism. It is very unwieldy for large and complex graph structures. It takes too much efforts to classify with tags at each object insertion. But logical expressions (and, or, not) at retrieval stage are not provided.

Yes... I remember strange behavior of my bookmarks - when some nodes from time to time changed their location. Not disappeared - but changed - without any evident reason. It is the situation where I use BSP2 to find the escapee.

If my practice of nodes' proliferation is against the rules and is the source of strange effects and errors - I would repair my structure if some method would help to find these nodes.

Now to your scenario. May'27.zip

[aaFn]

Thank you.

So good news :-) = you do not have "duplicates" in the meaning I was intending, i.e. the same true structural object with same internal unique id appearing twice or more. Your "duplicates" are simply textual duplicates, they have no structural impact on the tree, it remains a tree .. even though their contents is the same, they are different object in the tree.

Bad news = I do not understand what is making your second run not working on 2.0.19 .. all the more that here with my trace version, it appears the 2nd run is working .. do you confirm ?

If yes, the only difference being the disable favicons forced from start, then once you disabled favicons, your 2.0.19 version should also work well ... I can think of more spacing and a delay mechanism in 2.0.20 to reduce more the effects, or give you time to switch off the option and recycle ..

[vitaaliy]

You are right - my "duplicates" really are textual clones. "Bad news = I do not understand what is making your second run not working on 2.0.19 .. all the more that here with my trace version, it appears the 2nd run is working .. do you confirm ?"

• Alas... I was acting strictly according to your prescriptions without deep understanding of all peculiarities. That is why I copied your commands one after another and gave my confirmations. If you need to check anything - let us use the same method: You give the new plan, and I strictly follow it.

As for me - I made the following experiment. I deleted your debug-version and loaded current BSP2 2.0.19. It began to work right away while "Disable favicon fetching" field being not marked. I let it build the B-tree, then switched to the native FF B-marks, marked "Disable favicon fetching" field, closed and opened FF, then switched BSP2 on (with marked DFF field) - and it started that same endless Loading bookmarks process. What is interfering? Does favicons fetching interferes? I can endure the slowness of its reaction, but periodic deleting and reinstalling is really tiresome...

[aaFn]

Ok, from the traces, I can see that the trace version is working fine the second time. Not sure why 2.0.19 is not working fine on the second time .. All the more if DFF is set, favicon fetching shouldn't interfere.

So one more experiment:

• You'll find a new trace version at bottom of this comment.
• It does not enable DFF from start, but it keeps the traces, and it introduces an hysteresis of 1 minute after end of tree load before starting to fetch favicons. And it also has more spacing between the saves to local store to try to decrease further the memory saw phenomenon.
• Can you re-try your experiment above with that version instead (so load temporary add-on each time ..), giving me the contents of the trace textarea of each run ? Make sure you switch to native FF Bookmarks within less than a minute after B-tree is displayed.

Then, uninstall the trace version, stop FF ..; restart FF

• Re-load the trace version below (temporary) ... but let it go ... I.e. do not touch DFF, do not stop it, wait 1 minute after the B-tree is displayed, watch the memory while favicons are fetched, and let me know if the memory saw effect is getting better or not ..
• When you're tired of watching memory, get the trace textarea to send it here, switch to native bookmarks, close / reopen FF, reload that trace version (temporary), and verify that B-tree is coming after 25s or so.
• Again get textarea traces, send here .. and you can uninstall the trace version

Let's see what you get on those 4 traces .. if this is working better, I will publish it offcially as 2.0.20 (without the traces) so that your 2.0.19 gets upgraded and hopefully you can work better.

Thank you. BSP2-trace2.zip

[aaFn]

By the way, how many FF windows do you have open ? If more than one, can you make sure that only one has BSP2, and all the others have the native bookmarks ?

[vitaaliy]

I have only one FF window opened. Though, the number of tabs is high. Now it is 40. The protocol of my efforts is attached. God bless us! May'28.zip

[aaFn]

Hello @vitaaliy , 40 tabs is nothing, I have 300 :-) ....

Thank you for your efforts, confirmations and tests. It looks like we have reached a a balance which should let you work again. I will then publish 2.0.20 tonight and leave that issue open for some time to let you time to come back and tell me if this is working ok or if we need to dig further.

Note that I made plenty of tests on my side trying to break the things by interrupting in the middle of saves, with switch to native bookmark, switch back very quickly, or simply disblaing by the button in the middle of favicon fetching, or stopping FF again in the middle of things ..

On the numerous tries, I could get only one case which resembles somewhat to what you experimented in 2.0.19 = the save was corrupted, and when loading next time, BSP2 was in fact not loading anything from storage, and nothing was triggering behind by the API, not even an error or a trap that could be detected. Therefore it was looking like your case with "never ending loading" (in fact no load at all, cpu was flat at 0).

If that happens again to you, then my workaround will be to have flip-flop save on two versions, changing slot at each save. Therefore, there will be always a good version, and with a timeout on load from local storage at next restart, I will be able to detect that something is going wrong and to load from the N-1 version.

Lert's see ..

Note: poor goldfish who is probably not mortal and then cannot be fine .. :-D And thank you for the bits of Russian culture and humor :-)

[vitaaliy]

Today I installed BSP2 2.0.20. As usual it immediately started to work with favicon fetching enabled. After 1 min delay it displayed BM-tree in the sidebar. Okay. Then I switched to the native BMs and after that tried to return to BSP2. The same situation as before: endless Loading bookmarks message, processor doing nothing. Thus, as before, I can use BSP2 only once after the fresh installation. After that to be in the ready state it should be deleted and reinstalled. Maybe setting initial value: DFF checked could be of any help?

[aaFn]

From what I saw in the single error case I could recrate on my side, this means storage was corrupted, and this is why the load is stuck (FF API does not send the result of load, and no error, so nothing happens). Not sure why things were ok with the trace version and not here ... this is the same except traces.

When you switched back to native bookmarks, was favicon fetching going on ? (memory saw ....) With your numbers, you have basically 35s or more after the bookmark tree is deplayed before favicon fetching really starts.

The theory would be that when you switch back to native bookmarks, you are interrupting the save, and this is corrupting it ... should not be systematic though, as on my side I could only reproduce that once on about 20 attempts ..

If you switch back to native before the 35s after display (like a few seconds after), things should be ok when you come back to BSP2 .. Can you try and let me know if not the case ?

To be on the safe side, then you can use the time during loading bookmarks on first install to set DFF in options, and then when the tree appears, you switch back to native within the 35s. And then when you start BSP2 again, all should be fine, no more chances of storage corruption because saves are then very small and rare.

On my side, I will look a doing a dual state save then to recover from corruptions ...

[vitaaliy]

1. Now I have BSP2 with DFF marked. Native BMs at the side bar. All is quite and steady. Processor: 0-3%, memory: 4.67 - pure horisontal line.
2. Switching on BSP2 leads to nothing. Still horisontal line for memory. The same level. Processor also remains practically quite. Loading bookmarks endless indication...
3. Now I remove BSP2. Restart FF. It starts very slowly and lazily. Memory consumption monotonically grows up to 5.9 Gb, then returns to 3.75 Gb. FF becomes functional maybe after 3-4 min. after its start.
4. Then I Add BSP2 to FF. It starts Loading BMs. Memory level slowly grows. Ready! Switched to the natives... Memory level returned to 3.69. No saw. Comp doing nothing. I switch BSP2 on... OK! Success! The tree - in place. Returned to natives. Everything fine. Steady memory. I completely forgot about DFF - went there and marked it...
5. Tried again: switched to BSP2 and after fetching the BM tree - returned to natives. Flight normal.
6. Once more - that same exercise - normal.
7. Now I will fetch BSP2 and stay with it for more time than few seconds... Nothing extraordinary happens. Memory - steady. No saw. Processor is quiet. Switched to natives... After, say, 5 min returned to BSP2. Flight normal.

The victory of the mind over soulless matter! Well... I think, good result! No saw, no processor overload. Switching from natives and back passes smooth. The only drawback remains long time for loading BSP2 tree - up to 55 sec... If something goes wrong - I will yell here ;)

[aaFn]

Good .. :-) Switch on should go down to 25s or so (display time) per your numbers when I can implement next steps of #5 and #2. I will also work on that save corruption problem and a recovery mechanism in case you are not the only one experimenting it... but now that you are in DFF, you shouldn't see it. Note: the other alternative to setting DFF is very simple = remove DFF, start BSP2 just before going to bed, and let it do favicon fetching during night .. and then all should flat again in the morning.

Your system is strange though .. 3 minutes to start FF .. does it do that each time you start FF ? Since when ?

[vitaaliy]

"3 minutes to start FF .. does it do that each time you start FF ? Since when ?" Each time. Yes. I think from FF 60.0.1 or even from 60.0.0. What we had before? 57? At that version it was quite reasonable. But now various FF processes switch on, memory consumption slowly goes on... But after that initial transition stage - all looks normal.

[vitaaliy]

Alas... Yesterday I switched my comp off while using native BMs, DFF marked. Today I switched it on with natives. Then I chose BSP2 - and again - endless Loading bookmarks, CPU doing nothing, memory - steady level… ç'est la vie... One more experiment. During this endless loading I removed BSP2 and then downloaded it anew without FF restarting. Nothing good. This beast remembers its parameters and just goes on its endless happening. So only pure restart of FF can be of help.

[aaFn]

Hello @vitaaliy , I will publish 2.0.21 later today which will be finer in decomposing the messages of where this is blocking. That will allow you to tell me where this is really blocking on your setup, and if that confirms what I could experiment on corrupted save, I will implement the flip-flop save feature to always have a good save somewhere.

[aaFn]

2.0.21 is out .. next time things are hanging, let me know which step is displayed ...

But I wonder if the problem is not related to your FF instance .. Maybe you should export your bookmarks, and then try the whole with another FF ?

A portable one for example so that it does not perturb your current install / profile (it will run on its full own, the only constraint is you cannot run both FF at same time) = https://sourceforge.net/projects/portableapps/files/Mozilla%20Firefox%2C%20Portable%20Ed./Mozilla%20Firefox%2C%20Portable%20Edition%2060.0.1/)

[vitaaliy]

Here is my memo. First I disabled all add-ons, beside BSP2. The same result: first installation works OK - both loading stages: Load saved state.., then: FF API load bookmarks - pass, but restarting FF brings that same endless loop: Load saved state.. So if the first stage pass successfully it never stops on the second stage. Then I installed the portable version - as you proposed. No extra add-ons, tiny initial BMs tree - works fine and very fast ;). Then I imported my famous gigantic BMs file - and obtained exactly the same result. So, the problem is provoked either by dimension, or by structure, or maybe by some extraordinary symbol or construction in it. I have got one more idea: experiment with FF in Ubuntu... but became somewhat tired fighting with these windmills… ;)

[aaFn]

Hello @vitaaliy , thank you, this is what I was suspecting and corresponding to what I could reproduce only once: the FF API loading from local store is not returning a result. That means it is corrupted at save.

On my side, I could fall on a case different from you where the save is not happening (and the value in local store becomes empty). It is due to an out of memory situation, because the Garbage Collector is not acting fast enough, and I suppose this is also what happens to you:

[Exception... "Out of Memory"  nsresult: "0x8007000e (NS_ERROR_OUT_OF_MEMORY)"  location: "JS frame :: 
resource://gre/modules/JSONFile.jsm :: _save :: line 306"  data: no]  (unknown)
DataCloneError: The object could not be cloned.  (unknown)
Failed to serialize browser.storage key "savedBNList": out of memory  ExtensionStorage.jsm:56
    toJSONSafe resource://gre/modules/ExtensionStorage.jsm:56:24
    _save resource://gre/modules/JSONFile.jsm:299:31
    InterpretGeneratorResume self-hosted:1256:8
    next self-hosted:1211:9
    JSONFile/this._saver< resource://gre/modules/JSONFile.jsm:108:40
    idleDispatch/</< resource://gre/modules/PromiseUtils.jsm:40:21
uncaught exception: out of memory  (unknown)

This is related to the memory saw phenomenon ... From my current tries, the most contributor to that is the save to local store, and I believe the jsonify() function they provide to save objects as strings. I can see that the fetch() function is also contributing to the phenomenon, but to a lesser degree.

So I will see if I can write my own jsonify() ... and if this can have an effect. I will keep you posted here .. meanwhile, I hope you will have the patience to cope with this "feature" of FF (as you guys says in Russian, from what a Belarus girl was saying to me once ..).

[vitaaliy]

Okay. From one side, I feel somewhat awkward inducing you to handle my specific case. But from the other side it probably help you to make your BSP2 more universal and reliable. As Friedrich Nietzsche said: That which does not kill us, makes us stronger. Russians say: For one beaten give two unbeaten. Good luck!

[aaFn]

Hello @vitaaliy , no problem. I could robustify a number of things by adapting the proc for serializing the object to json. I could also create a dual state save to make things more resilient to memory problems, which reproduce somewhat what you experience. It is also taking less space on local storage, and faster to read / save. And last I dare to believe I could reduce again the memory saw phenomenon.

However, there's one things I can't do much about = json is very verbose and is taking a lot of space. My simulations show that in your case, the detailed save of bookmarks tree, which I have now put in place in BSP2 to try to get around FF API slowness in next optimization round, is taking up to 220 MB of text file on your system !! (a test with 8000 bookmarks with the above setup is taking around 22 MB). And if I do not do the double save, which is less resilient, that will still be around 110 MB ..

So it might well be that while saving the bookmarks to local storage on first BSP2 run after install, FF is hitting disk space limitations on the partition hosting your FF profile .. and when BSP2 restarts, the FF API cannot read it properly and fails silently without gving anything back to BSP2, which makes it hanging there waiting for the data eternally .. Would you be able to check ?

If you have the default install, your FF profile is a directory in C:\Users\\AppData\Roaming\Mozilla\Firefox\Profiles Its name is of the form .default...

If it is somewhere else (non default installation like I do at home), you will find a profiles.ini in the ...\Firefox directory, telling you where to find your FF profile directory.

Could you verify that you have plenty of free disk space there ?

And then, if you go to the browser-extension-data directory in your FF profile, you will find a bookmarksearchplus2@aafn.org directory for BSP2, among the other ones for your other add-ons. Inside it, you will find a storage.jsfile containing the local store save. It would be interesting to track its size with your 80000 or so bookmarks ..

[vitaaliy]

The size of storage.js file is now 250 Mb. The free space on the system disk is more than 41 Gb.

[aaFn]

Ok, I was not bad at estimating :-)

I should send you tonight a zip version to temporary load and test as we did before ..

[aaFn]

Here is the temporary one to test (pre-2.0.22) BSP2-trace3.zip

Can you let me know:

• If this is going better ?
• The contents of both the trace window and the Browser console for first start, and then at least 2 restarts
• The size of the storage.js file in the browser-extension-data\bookmarksearchplus2evol@aafn.org directory ?

Hope this will go better on your system ... thanks, aaFn.

[vitaaliy]

Hi aaFn, "Hope this will go better on your system ..."

• Yesss... you are right. It works! All switches from natives to BSP2 and back go smooth. Of course, the loading time remains too large - about 60 sec... But the main functionality which we tried to achieve is reached! I am very glad. And thank you again for the great labor and all the efforts in polishing BSP2 to its current state. Let me consider it as my today's birthday present ;)

Vitaliy

June'2-Trace3.zip

[aaFn]

Excellent news and happy birthday :-)

Thanks for the detailed output. Yes, the size of storage.js is doubling on second save because of the dual save feature for resiliency to memory problems and corruptions. I am toying with the idea of making it an option .. however that is much much less than before so maybe I should leave it in standard ..

One thing: since you left DFF to its default disabled value, it is now in the process of getting favicons in the background. With 80000 bookmarks, and favicons being 16x16*4 bytes, I estimate this would make 80 MB additional max per save slot (if no compression, but as their encoding is usually compressing things, that should be less, like maybe 20 MB additional), so around 160 MB additional (or 40 MB if compressed), which would make your storage.js at around 230 MB ultimately (or around 110 MB) .. and longer to load.

Maybe you should consider setting the DFF option on, when I issue 2.0.22 (by the way, when 2.0.22 is out, I suggest you recycle one more last time by uninstalling and reinstalling, to make sure there's nothing left from a previous corruption).

At any rate, in a few days after you get 2.0.22, it would be good that you look at the storage.js of browser-extension-data\bookmarksearchplus2@aafn.org and let me know at which size it is .. I have the feeling that the problems you have been hitting may be related to an integer limit somewhere as one of your posts above was suggesting, and that 256 MB could be the max supported size ...

[vitaaliy]

My style is as follows. As a rule, I use natives. It is somewhat more handy and faster in any manipulations with bookmarking. From time to time a problem arises to find the real location(s) of some identifier. This problem needs the BSP2 help. After the needed location is found I switch back to natives - and until the new problem to be solved with BSP2. The question is: I will keep DFF unmarked - let the favicons live their natural life. But does the process of their fetching still goes on in the background when I am in the stage of using natives? Or I should have BSP2 steadily on - until the favicons fetching is done? Or this process saves its current state and can resume fetching when BSP2 is on - next time?

[aaFn]

Looks like a good style of doing things :-)

To answer your question, currently, the process is going only in background when BSP2 is active in the sidebar. So this is safe to leave DFF unmarked .. I would just be interested in the size of your storage.js in some time, if you come back here from time to time to report it ..

You are my special user with 80000 bookmarks and I like the challenge of making your life easier while preserving the life of others at the same time (of course preference will go to them when this is not compatible as they are more numerous) :-) Also, as you rightfully say, making your extreme case work ensures me that it will work for all or most others, so I see that also as a highly required exercise of robustness : "what doesn't kill you .. etc ..." :-D

In brief, your case is a wonderful case making me get a better add-on for all.

Now, in one of the next steps of optimization when they happen (see #2 and #5), the favicon loading will have to occur in background even when BSP2 is not active in the sidebar. Indeed, that's the only place it will be able to run = in order to load the bookmark tree from memory and to persist it in memory across sidebar activation/deactivation cycles, and not from FF API each time the sidebar is activated, all that process will go to the background page. Benefit for you will be a faster load of the sidebar, as it should reduce your 60s before you get the tree to around 30s, We'll see when we get there ...

It will still support being interrupted at any time like today (e.g. an FF stop), and will resume when FF restarts.

[vitaaliy]

I left BSP2 in action - doing what it please for about 4 hours, DFF unmarked. Processor is doing almost nothing, memory is on the constant level, storage.js is 77 Mb. As I suspect, it ought to use time to fetch favicons. But I noticed noone. Where are they?

[aaFn]

You might have hit an out of memory error, and in such a case, when it is detected by BSP2, it stops to not worsen things. If you recycle it (click on the button in the toolbar twice, or hit twice Ctrl-Q), it should restart fetching them roughly 20 seconds after your tree is displayed.

In the trace version you have, it is visible in the Browser Console with the

json.length = 30074025
savedBNList
saveFIndex: 0

messages appearing every 20 seconds or so

[aaFn]

2.0.22 is out. You can remove the temporary version and upgrade (well as I said, uninstall 2.0.21 if you still have it, and reinstall 2.0.22) ..

Let me know how that is going. (note: I eventually chose to make the dual save feature an option for people who have trouble with saves, to not grow up on local storage where not needed)

[aaFn]

I guess we can close the issue now, things seems stablized :-) Thanks, aaFn.