Ability to sync across devices (please contribute ideas!)

[CristianHenzel]

Not sure if this is even possible, because I don't quite understand how the playlists (watched videos, etc) are stored, but I would very much appreciate a way to sync the playlist between multiple devices. For example if I watch a video on my work computer, it should also be marked as watched on all other devices that I access the playlist from (home computer, phone, etc). Is this somehow possible right now? If not, do you think a feature like this could be implemented?

[daviddeutsch]

Right now, all data is only stored locally in the browser. I'm currently looking into several options to improve data storage and in particular cross-device storage:

• Try to figure out whether we can let the browser sync stuff. I think Firefox for instance can already do something like this with its sync.
• Check on IndexedDB for general improvements. I think this might also help with sync related to the above.
• Use an online service like:
  • Various "cloud database" hosts have free plans, although a lot of them limit use so much that we cannot use it for the hundred fifty or so people who use sanegrid
  • Dropbox can handle data storage and most people have a Dropbox account already
  • Roll my own solution, but then we're moving into "register on this site" territory and I'm not sure whether people would like that. That would also move things into "this is more than just a hobby" territory
  • With "log into your YouTube account", we might be able to store data right with the YT account, but I'm not sure whether I trust them with that data

In general, I think I could offer a number of different services at the same time, but I'm having a hard time estimating what people would go for. All input is certainly welcome!

[CristianHenzel]

I think the "let the browser sync stuff" shouldn't be considered, since you might not always be on your computer when you access sanegrid. For example someone might be on a work computer and not be logged into their browser to be able to sync the browser data, so then they would lose this functionality.

Regarding the other options, there would be a couple of things to take into consideration. First, IMHO, would be privacy. That means, if you decide to store all of the data in a single location, you would need to somehow allow encryption, so that nobody could gain access to it, and encryption is fairly hard to do in an open source application, methinks. Personally, I couldn't care less if someone sees my watched videos history, or the likes, but I'm fairly sure there are people that care, and they wouldn't be happy if this information was easily accessible for others. Second would be maintainability. If you use Dropbox or Google Drive for example, you would need to code an API interface for that, since you also need to write to the cache file, don't you? That would add a bit to the maintainability of the application. Third would be running cost. Ideally, the service used should be free, else you would need to charge people to use the app, which I think would be bad. Also, a fairly big factor, would be size needs. If the data, that needs to be stored for this isn't very big, you could for example store it in an MySQL database. This would also have the advantage of being fairly fast, probably in order of magnitudes faster than Dropbox or similar services, which is a plus. The downside would be the fact that you would need to store it all in a single location, and that would probably raise privacy concerns.

What's your thought on these? Am I overthinking thinks here?

[daviddeutsch]

Encryption and Open Source are absolutely not opposite concepts, actually, the more open your encryption, the better. Put in other terms: Don't trust encryption if you cannot see the code that supposedly encrypts.

I do agree that privacy is a central consideration here, this is also why I'm somewhat opposed to using stuff like Dropbox. Then again, if you use Dropbox, you might as well store this data there. Also: Dropbox has some pretty nice APIs, even for Javascript, so integrating their service wouldn't be that hard.

I mostly noted browser sync because in html5, with IndexedDB, browsers to have a nice way of storing data for fast access - maybe not as fast as running MySQL, but still a lot faster than I have right now. Definitely fast enough for a small application like yt-sanegrid. What I have not figured out so far is how well this mixes with built-in sync. I do agree though that it would be kind of hard to make this work i.e. on a work computer, but then again - no personal data storage works well with that. The only thing that works there is some kind of central server or cloud service, which again kills privacy (at least if you factor in three letter agencies).

The most likely scenario is that I will implement and maintain a number of ways to sync and manage data. But you're right - we're running into a cost problem. Not "paying for server" cost, but "what my time is worth" cost. So far, I'm not paying for hosting since it runs straight on GitHub and the data that is served isn't really notable anyways. But I do have to carefully plan where and how I spend my time with this "hobby" project.

I think there is another site that does a similar thing as yt-sanegrid, just that they use a central server where your data is stored. But they want you to log in with your YT account straight away - that and the fact that it's stored with them turned me off to the idea. So I definitely think the guerilla nature of yt-sanegrid is a big factor worth being protected.

In the end, we really aren't talking about that much data. For most people, especially if I include some kind of archiving, you would end up with a few megs tops. Depending on how far people are willing to go for sync, you could save that to an external file on a usb stick and just carry that with you. So a solution might be that even on your work computer, you load yt-sanegrid, tell it to load your file on a usb stick (or dropbox folder or whatever) and you have your data.

Hmm, mostly rambling through ideas here. I really have no good final solution here. Mainly wanted to say that I definitely agree with the spirit :wink:

[CristianHenzel]

Well, I guess you could make 2 ways then, to cope for every use case and type of user. A "local file", which would allow the user to store the playlist information into a file saved to his machine, maybe even encrypted somehow, for added security (which he could then put on a USB stick, on Dropbox, or whatever else), and a MySQL Database, for the users that don't care that much about who sees their watched videos list. The thing with MySQL though, if you store data for usernames, is it could easily be misused by someone, if you don't implement an authentication somehow, couldn't it? Like for example someone could enter my username and mark all videos as watched, and if it saves this just according to my username, I wouldn't see those videos anymore (or anyone else who enters my username for that matter).

[daviddeutsch]

Yeah, if I would do a central database, that would be one of the issues. However, you could get around that by tying the data to logging in to your YouTube account. But again - not sure whether I like a centralized solution like that.

[CristianHenzel]

Well, if you could figure out some decent encryption and everything would be stored securely, I for one wouldn't have any problem with a central database storing all the data.

[Vankog]

Just to give this a notch. Some kind of (optional) sync would be absolutely awesome. Viewing habits on several computers could be greatly improved this way.

This does not only include what was watched, but also which videos are in the storage overall. Because on new browser profiles sanegrid can only load a finite number of initial videos. For instance: I just changed to a new PC after having one or two weeks of youtube absence. On the new PC my unseen subscriptions could not be loaded completely by sanegrid. I had to head over to youtube.com/feed/subscriptions to get further back in time. btw: It seems they lately have greatly increased the max. number of videos in the subscription feed. I don't know if it is still limited at all.

(Maybe some kind of export/import might be a first easy solution. Or I could imagine putting some kind of data(base) file in my Google Drive or Dropbox sync folder to auto sync the storage by myself.)

[daviddeutsch]

Long winded post mode engaged

There are two main constraints to working on this right now. The first one is that data is still stored verbatim in the localStorage of the browser, which not a particularly structured method and thus tough to latch onto for syncing. The other is the "where would we store this?".

What I've been looking into for the first is, of course, PouchDB - it works across browsers and should give me a much better control over what is stored and how. I will probably work on that next, after the v3 switch.

Where to store that data is a little tougher. Google Drive and Dropbox are definitely options, particularly Dropbox has a very nice API for doing that. Other options would be cloud Database providers like Firebase and such. Finally, I have been working on a bunch of syncing methods myself lately for my "work" work stuff.

My main concern there is that these things are, slowly but surely, stepping out of "this is a hobby project" territory. I have been thinking about this for quite a while now and these next bunch of features are really more features that you would expect in a for-pay project, particularly the sync which deals with the concept of user identity and for which I would probably have to implement not just one but a handful of options so that users can have choice.

Now - I have accepted donations for quite a while. Sanegrid is nearing its first year in existence and over those 8 months, I have collected a total of around 50 Euro. I would guess that I have put about 150 to 200 hours into this project so far. As long as the work was simple, I could justify it with learning new things and using it myself. But these days are certainly numbered with this latest v3 switch - this is cutting into my other work time and the cost won't match up for much longer.

I think I will make Sanegrid into an actual website that offers these extra features and a way to migrate from the current github url. I would decide on a final set of features that the github page will have so that people can stick with it if they want to and then release the more advanced features on the new commercial URL.

I have no clue what I would charge, though. Hosting would cost close to nothing because the traffic and database load is negligible, so there aren't any metrics that I could go by there. Developing more features will definitely take more time in the future. I would estimate that I would spend around 10-20 hours a month, which would put the hourly total (in order to compete with my other work) at €500-1k.

Reddit peak times aside, Sanegrid gets a steady audience of 300-350 users at the moment. To cover the cost that would mean about 3 Euro per user per month.

It's funny, really, I see that I spend so much time in Sanegrid myself - switching to it, updating it, sorting through and filtering my list - and from the numbers that I see on my analytics page, a lot of users do the same. I can definitely see that it has commercial potential that way, even though it's "just a youtube subs inbox".

Finally, finally - The other option I have been pondering is whether it might make sense to have my own Patreon page. That would allow me to keep the personal vibe that I have going with this project. I don't think patreon is being used for programmers so far and I'm not sure whether the "x Dollars per work" makes sense - The only thing I could map that to would be something like new version releases, but that would be weird and arbitrary.

Maybe the other project I want to be working on is a patreon for programmers :wink:

Anyhow - this has now grown into a proper rant. Thanks for reading... everybody who read this. Let me know what your thoughts are on this!

[Vankog]

While I was reading this the Subbable/Patreon thing came to my mind, too. Making me smile at the moment you mentioned it. ^^ In my eyes a Subbable project could be more suitable for such a project than Patreon, because you can pay per month instead of per work and could be indeed a great kind of recurring donation.