Revamp manga list syncing method

[fuzetsu]

The current method of syncing through a bookmark is less than ideal and many people have had trouble getting it to work well for them. There is also the issue of storage space to keep in mind.

Possible Solutions

• chrome.storage
• chrome.syncFileSytem
• Continue using bookmark method with some revisions

[KriPet]

Pros and cons as far as I see it:

==chrome.storage:== =Pros= -Comparatively easy to implement. -Can get old value as well as new value. This will simplify ensuring that new data is not overwritten. =Cons= -Limited space, would need to break up the data or find a more compact way to save it

==chrome.syncFileSystem== =Pros= -Almost infinite size -Got some conflict resolution. Not sure how this works though =Cons= -Not finalized -Isn't it only for apps, not extensions. -Dependent on google account.

==Bookmark method== =Pros= -None that I can think of =Cons= -Doesn't always work -Can't sync on demand -Hard to debug (I'd think it is, can anyone confirm this?)

It seems to me that in all cases we need to store a serialized json object, so that won't be a deciding factor. The chrome.syncFileSystem seems to be a good solution, if it is available; but the simplicity of storage,sync, if we get past the size problem, makes it the best solution, imo.

[fuzetsu]

Isn't it only for apps, not extensions.

You're absolutely correct about that... (just tested it to make sure) Teaches me to read more carefully...

Good breakdown though, certainly storage.sync would be the simplest to implement since it really is just syncable localstorage with more severe storage limitations.

[KriPet]

I feel the best to do for now is to find out why bookmark sync isn't working and wait to implement a new sync until the new storage solution has been finalized. Replacing manga names with unique ids would shrink the json string, so starting to implement storage.sync now might lead to us making doing a lot of unnececcary work.

I'll look at the bookmark sync this weekend if I got time.

Also, do you know if a lot of people use this feature? I've never been able to, but I'd love it if I were.

Another thing to consider, even if the syncfilesystem api requires a google account, so does the bookmark sync (if you aren't using other bookmark syncing software) so it wouldn't really be an issue.

[fuzetsu]

Another thing to consider, even if the syncfilesystem api requires a google account, so does the bookmark sync (if you aren't using other bookmark syncing software) so it wouldn't really be an issue.

That's true, and come to think of it chrome.storage must require a google account too, otherwise how would the data know where to sync to. I guess that means that one pro for the bookmark sync is that it doesn't necessarily require a google account.

Also, do you know if a lot of people use this feature? I've never been able to, but I'd love it if I were.

Some people must have it working since you don't hear too many complaints about it. Could just be that not many people have the need to sync. I used to have it working, but some AMR version came along and broke it for me. Might be some encoding error like you mentioned earlier.

[braiam]

Some people must have it working since you don't hear too many complaints about it. Could just be that not many people have the need to sync. I used to have it working, but some AMR version came along and broke it for me. Might be some encoding error like you mentioned earlier.

In reality was chrome that broke it first, and we didn't touch it when we ported the extension to manifest version 2. So, is to be expected that something doesn't work, but funny enough the export function that use almost the same method works like charm. Also, after 1.5.0 a guy came to forums asking for a method to recover his list because he activated the bookmark syncing backwards (first were he didn't had the list, then in his main) so his list got wiped, so I assumed it does work...

About the storage.sync limitations, I've calculate how much data I need to store my stuff, the way I'm thinking about the implementation. My current export json string has about 63k characters, but some of that is easily cut down (like the name of the chapter or the chapter list url), so I was estimating about 156 bytes for the value (the last read chapter link + metadata like categories and stuff) and 13 for the key (unique chapter + mirror numerical value cut till the 13th value, or whatever), way down of the 4,096 bytes per item limit and multiplied for 512... (156+13)_512=169_512=86,528 so we still have space left for more stuff! Bookmarks and preferences... or so I would like it to be. Also, we need to think about the max writes/hour and operations/minute. We might think about a limiter so users don't abuse of this (like deactivating/activating a function several times).

We could compress the data using some shady library I saw like 2 month ago whenever we exceed the max value. The compression wasn't binary, if memory serves me right, so we might store it like any other string. Found the library as I was writing my response (thanks again SO) http://stackoverflow.com/a/294421/792066

[mexicano21]

I have a javascript zip implementation that works like a charm. I almost bundled it in AMR in a "download this chapter" function, but this didn't work because the browser crashes or freezes when javascript sucks too much memory loading a lot of images. Also had problems with same domain policy in a lot of mirrors. So I never submitted it.

But for a JSON string it would work perfectly. Also if the binary encoding turns out to be a problem it is easy to convert it to Base64 (and gaining some bytes back in the process).

2013/4/25 Braiam Peguero notifications@github.com

Some people must have it working since you don't hear too many complaints about it. Could just be that not many people have the need to sync. I used to have it working, but some AMR version came along and broke it for me. Might be some encoding error like you mentioned earlier.

In reality was chrome that broke it first, and we didn't touch it when we ported the extension to manifest version 2. So, is to be expected that something doesn't work, but funny enough the export function that use almost the same method works like charm. Also, after 1.5.0 a guy came to forums asking for a method to recover his list because he activated the bookmark syncing backwards (first were he didn't had the list, then in his main) so his list got wiped, so I assumed it does work...

About the storage.sync limitations, I've calculate how much data I need to store my stuff, the way I'm thinking about the implementation. My current export json string has about 63k characters, but some of that is easily cut down (like the name of the chapter or the chapter list url), so I was estimating about 156 bytes for the value (the last read chapter link

• metadata like categories and stuff) and 13 for the key (unique chapter + mirror numerical value cut till the 13th value, or whatever), way down of the 4,096 bytes per item limit and multiplied for 512... (156+13)_512=169_512=86,528 so we still have space left for more stuff! Bookmarks and preferences... or so I would like it to be. Also, we need to think about the max writes/hour and operations/minute. We might think about a limiter so users don't abuse of this (like deactivating/activating a function several times).

We could compress the data using some shady library I saw like 2 month ago whenever we exceed the max value. The compression wasn't binary, if memory serves me right, so we might store it like any other string. Found the library as I was writing my response (thanks again SO) http://stackoverflow.com/a/294421/792066

— Reply to this email directly or view it on GitHubhttps://github.com/AllMangasReader-dev/AMR/issues/75#issuecomment-17052937 .

Fábio de Godoy http://www.animenewsnetwork.com/MyManga/?user=mexicano21

[fuzetsu]

@mexicano21 I'd be curious to see the implementation, I was actually taking a look at the link that @braiam just posted, I found the same function a few days ago. Just made a little test page for it. I manage to reduce my manga list json to about 48% of its size using it 9ms to encode 1ms to decode 24KB file. Although I'm not sure if I was calculating the savings correctly...

[KriPet]

Made an issue about the bug in BSync:

https://github.com/AllMangasReader-dev/AMR/issues/76

[braiam]

@fuzetsu I've just tried it but seems that it don't like the ampersand character cuz it became gibberish when I tried.

[fuzetsu]

@braiam what kind of gibberish? The way it compresses it is by converting it to those unicode characters. Is that what you're seeing?

[braiam]

Ah, ok, yeah, was a little sleepy and saw like your code shall return the same value. But in any case let's consentrate in the issue at hand.

[fuzetsu]

Haha, yeah I guess I should have probably put another field to decode the encoded data to prove that it worked.