search

wikitree / wikitree-browser-extension

Browser extension that adds advanced features to WikiTree.com.
MIT License
24 stars 20 forks source link

Improve Backup/Restore #558

Closed udjeni closed 4 months ago

udjeni commented 4 months ago
jonathanduke commented 4 months ago

I was also working on some improvements to the cross-tab messaging so that it will skip over old tabs where the context has been lost. I'll see if I can merge mine into yours.

udjeni commented 4 months ago

Oh yes, forgot to mention that we now get a popup telling the user to restart the browser or update all the WT tabs for backup/save to work

jonathanduke commented 4 months ago

I am working on merging, but now with the new format, it seems to be including way too much data in the backup file. I'm getting an error now that says "Uncaught (in promise) TypeError: Message length exceeded maximum allowed length." when it tries to call sendMessage to send the data back to WBE. It looks like this is probably related to the cache tables such as ConnectionFinderWTE (146 kB) and RelationshipFinderWTE (77.5 MB). CC7 as well has 79.4 kB in CC7 and another 55.1 kB in cc7Deltas. I don't think cached data should be included.

udjeni commented 4 months ago

Mmm. It was a specific request that the cached data be included. My CC7 is reasonably large, so I thought if mine passes, it would still be fine, but obviously not. (My backup file is currently 48.5MB)

jonathanduke commented 4 months ago

Looks like the hard limit is 128 MB, although it probably isn't good to be anywhere near that size for cross-tab messaging:

udjeni commented 4 months ago

If it is a strong requirement to have the other indexedDB tables included (because the data from those tables are used in the extra columns added to tables by the Table Filters feature), then perhaps we should remove the save/restore of the data from the settings tab and just go with the current save/restore buttons on the home screen?

jonathanduke commented 4 months ago

I don't understand what you're trying to accomplish. What buttons are you talking about? Why do you need a cache table in the data backup? Ian's features have been using this for a long time, and it works well enough other than some browser quirks. Why should we remove it from the settings dialog?

udjeni commented 4 months ago

Ian asked me to include those extra tables in the backup. That was the main purpose of this change. They are not only cache tables. They are also used to provide extra columns to other tables. If these so-called cache tables need to be in the backup, we obviously can't use the cross-tab messages any more. Ian has already added backup/restore buttons to the WT home page (My Navigation page), so I am suggesting maybe people should use those rather than the two at the bottom of the options/settings dialog.

jonathanduke commented 4 months ago

I found them on https://www.wikitree.com/wiki/Special:Home but nobody would ever know they were there. That's not where they would look to backup their extension data.

udjeni commented 4 months ago

The help document can be updated and the 2 buttons on the settings dialog can be replaced with a sentence telling people where they are, maybe ven include a url

jonathanduke commented 4 months ago

I believe this is ready now. The issue with common.js loading twice on the page has been fixed, so the semaphore should no longer be necessary. The backup logic should match what was already in this PR, except that when triggering it from the WBE settings dialog, the large tables are excluded at this time (see the compactMode argument).

Since WBE will attempt to notify each tab individually until it finds one that works, the user should not get an error unless all of their open tabs are stale. If at least one has been opened or refreshed since the extension's context changed, it should work fine.

udjeni commented 4 months ago

Good job! I think it might be worthwhile to consider not excluding CC7Database in compact mode. My CC7 is currently 9369 and the backup size of my CC7Database is 768KB, so Jim Loden's should be less than 53MB, about half the max size. :)

jonathanduke commented 4 months ago

Yes, I don't consider it a full solution yet, but for now I didn't want to alter the existing functionality of what the settings dialog already did. The limit seems to be lower than the 128 MB in the source code, as mine is only 77 MB but still throws the error, so I'm not sure what the practical limit of the size actually would be.

Ideally, I'd like to combine options and data together into a single WBE backup file with no size limit. But also as Ian pointed out to me, if we're including all of those internal database tables in the file, that means the backup would contain potentially private data about private profiles and relationships, so we're kind of limited on where and how we can generate such a file.