search

kwaschny / unwanted-twitch

Hide unwanted streams, games, categories and channels on: twitch.tv
https://kwaschny.net
MIT License
102 stars 13 forks source link

Maximum blacklisted channels? #28

Closed djcline23 closed 4 years ago

djcline23 commented 4 years ago

It seems new items I add to the blacklist are not sticking. I am at ~16K blocked channels, so that makes me suspicious that is the maximum number of blacklisted channels? Seems to be the same with and without using cloud storage. Is that indeed the max, or is something else going on maybe? If so, is it possible to increase the max?

kwaschny commented 4 years ago

There is a hard limit of around 5 MB. The storage fragment splitting required for the sync storage (as this one has more restrictions than the local storage) causes quite some overhead due to repeating keys in the JSON, reducing the number of maximum items even further. I was surprised back when someone told me he needs more than 1,000 items on the blacklist. So I implemented said storage fragment splitting to allow for more than 10,000 items. Kinda funny it's happening again. 🙂

The solution is asking for the unlimitedStorage permission. It only applies to the local storage mode, but will allow an unlimited number of items in the blacklist.

djcline23 commented 4 years ago

I added unlimitedStorage to the permissions in the manifest (and added unsigned in Firefox Developer) and it still gets stuck at ~16K. Does that surprise you? Or there are also some code changes or something that would need to be made to use the extra storage? Maybe something else is going on?

kwaschny commented 4 years ago

Found the issue. The before-mentioned fragment splitting was also applied to the local storage mode, hitting the maximum key count limit (chrome.storage.sync.MAX_ITEMS) when having lots of items, i.e. high number of fragments. There are no such limits for the local storage, so that's gone now.

Thus unlimitedStorage permission is not needed at all, considering that 5 MB allow almost 400,000 items in the local storage mode now.

Thanks for reporting. You can pull 140fe75ac07b8f13dc010b65d72eeeae29968c95 and sideload the extension if you don't want to wait for the next patch to roll out. Just make sure to use the local storage mode (disable cloud sync) - and backup/export your current blacklist, just in case.

djcline23 commented 4 years ago

Thanks, I've been using it in Firefox Developer and I'm currently at ~21K entries. So now I'm trying to transfer my list into normal Firefox since the add on has officially been updated. And the import doesn't seem to stick. It takes a couple minutes, then says the import completed and I click Save. But then when I look at the blacklist I still see the previous number ~16K. (Not using cloud sync still.) I see that I have version 20.1.20.

Something still off for import?

kwaschny commented 4 years ago

Do you mind sharing the list with ~21k entries, so I can debug with it? You can send it via e-mail, if you want.

djcline23 commented 4 years ago

Oh sorry, when I was grabbing the file I realized I was importing the old export with 16K, not the new one with 21K. It worked fine when I used the right file!