Synchronize subscribers and watch data between NewPipe and Freetube?

mattfj10 commented 3 years ago

I just downloaded the application onto my Linux Distro and I absolutely LOVE it so far. However, one thing that has kind of annoyed me is the idea of having to constantly keep my NewPipe and FreeTube applications in sync in terms of subscribers and data. I think adding some form of data synchronization (perhaps over a local network? Or in a style close to how brave or signal handles syncing but for NewPipe and FreeTube) would:

Reduce the nuisance of having to manually synchronize subscribers and channel profiles/feeds
Increase ease of use and positive user experience as having watch data synced leads to an overall more pleasant experience.

Perhaps it could be implemented by having a button on Freetube that opens a QR code, or something that NewPipe could scan to synchronize watch data? And everytime you want to re-synchronize you can just do that again, or perhaps a sync chain style setup like Brave? I realize implementing this would require the cooperation of NewPipe, but I thought it was an idea worth just throwing out there.

PrestonN commented 3 years ago

This is something I'd be interested in doing at some point. As you mentioned though, it would take a lot of coordination with the NewPipe team to get something like this setup. There's also some relevant discussion over on the NewPipe Repo.

There's several ways that it could be done so both sides would need to come into agreement on how this would be accomplished. I'd personally like for it to be a bit more automatic than what you're suggesting. Any sort of server involved for syncing would have to be something that the user controls and can be self hosted. My past suggestions have been to sync via WebDAV and I'd still be interested in taking that approach.

I'm not quite ready to discuss too much just yet on getting started on this, so it may be a while. I'd like to focus on other aspects of FreeTube before I tackle this as this is going to be a huge addition and there's a lot of smaller features I'd like to implement first.

mattfj10 commented 3 years ago

@PrestonN Good point, I just think it would be really great, especially for someone like me (I have a multi-boot system, and switch between OS's regularly so being able to manually sync up my freetube instances by hand perhaps at the end of each day would be AMAZING.)

Also while i'm here if you don't mind me asking, when I export data from one freetube instance to another, does it carry over the custom feed channels I've made for my subscriptions?

PrestonN commented 3 years ago

Also while i'm here if you don't mind me asking, when I export data from one freetube instance to another, does it carry over the custom feed channels I've made for my subscriptions?

It does as long as you're exporting / importing FreeTube specific files. If you're importing from YouTube / Invidious / NewPipe then all channels will be imported into the default profile only.

Syncing between FreeTube instances is something that will be much more realistic and is something I'm hoping to do. Even the simple feature of getting to set where your data is stored (which has been highly requested) would allow you to sync up between FreeTube instances using something like Syncthing. I'm hoping to put that together at some point though I won't be giving any ETA for now.

mattfj10 commented 3 years ago

Also while i'm here if you don't mind me asking, when I export data from one freetube instance to another, does it carry over the custom feed channels I've made for my subscriptions?

It does as long as you're exporting / importing FreeTube specific files. If you're importing from YouTube / Invidious / NewPipe then all channels will be imported into the default profile only.

Syncing between FreeTube instances is something that will be much more realistic and is something I'm hoping to do. Even the simple feature of getting to set where your data is stored (which has been highly requested) would allow you to sync up between FreeTube instances using something like Syncthing. I'm hoping to put that together at some point though I won't be giving any ETA for now.

Fair enough, well since this is open source and if you're open to contributions, I'd love to give a hand where possible, as I've been wanting to find a decent project to contribute to in my spare time for awhile. So if you have any advice for getting started on that as it pertains to freetube if you're open to help from others in the OS community, I'd be grateful.

PrestonN commented 3 years ago

Absolutely! We're more than happy to have more contributors helping out in any way possible. There's already a handful of pull requests that I still need to look through, however I'll accept most features even if it isn't one of my priorities. I can't guarantee that I'll look at a PR right away but I promise that I'll eventually get to it.

We keep a very basic Getting Started over on the Wiki. That should get your environment up and running. As you probably know, we use a combination of Electron / Node / Vue so having knowledge with those technologies are helpful. If you need any further help, myself and others are fairly active over in our Matrix Community, so feel free to jump in there and chat if you get stuck.

As for "things to work on", that's entirely up to you. There's a large pool of issues here to look through and you're welcome to pick up anything that's currently open. Make a comment in the issue that you'd like to work on it and I can assign it to you so that myself and others won't also work on it. I keep a project board active that states the issues that I plan on focusing on for the next major release. You're welcome to pick something from there if you want a more finite list. Finally, there's several upstream projects that may be worth contributing to which in turn comes back and helps out FreeTube which includes Invidious and the Local API, which each has their own maintainer / standards for coding.

Hope this helps, looking forward to any future contributions! :)

d4v1dv00 commented 3 years ago

How about allow user to specify the location of the "Saved" playlist (example user own Google Drive) so that no matter which device user uses, they always have the updated playlist.

Drasila commented 3 years ago

That is a feature I would like to see implemented.

citizenserious commented 2 years ago

Sync over nextcloud would be awesome though (: An option to replace subscriptions with the new file added (for me from newpipe) would be great so far, sometimes I unsubscribe channels :D

dtbj commented 2 years ago

WOW this feature is amazing I really want it.

therealrobster commented 2 years ago

Just wanted to check in to see if there's any plans for movement on this? I have my laptop and work machine that I'd like to sync. So even something like Syncthing shared folder or a SMB share or similar would be great.

However, to sync with NewPipe would be a dream. Have both teams been in talks as yet? Does anyone know?

therealrobster commented 2 years ago

An update on sync in general. I have found I can use syncthing on my linux machines and sync the ~/.config/FreeTube folders between them and they all work. I can use on one, close, open on another and it's all there. Hope that helps.

hilburger commented 2 years ago

An update on sync in general. I have found I can use syncthing on my linux machines and sync the ~/.config/FreeTube folders between them and they all work. I can use on one, close, open on another and it's all there. Hope that helps.

Syncthing is a cool idea! This won't work between Linux, Win and Mac machines, wouldn't it?

therealrobster commented 2 years ago

I'm not sure but I can't see why not. The programming team would have to verify, but if the files are all the same, all you need to do is have syncthing on your mac point to the mac freetube folder, on windows, to the windows freetube folder and on linux, to the linux freetube folder. Assuming all files within those folders are named the same and operate the same way it should work. I do note some conflicted files that syncthing leaves behind, so be aware of that, however it seems minimal and it appears to work (better than nothing). For me right now, very happy with this solution.

PrestonN commented 2 years ago

The required files with user data are setup the same way across all operating systems, so using something like Syncthing should be possible. As long as the files located here are being synced between your machines you should be able to have seem synced between platforms. I do personally use Syncthing but not for FreeTube data so I'm not sure how well it would sync if you tried having two machines make different changes to the files at the same time. This would likely be something we solve when we do a proper sync solution if this does become a problem. Either way using Syncthing should work fairly well if you're willing to set it up.

As for other syncing methods, it's still something I'd like to do, but it's going to be an after v1.0 type of feature, so it may be a while.

mattfj10 commented 2 years ago

The required files with user data are setup the same way across all operating systems, so using something like Syncthing should be possible. As long as the files located here are being synced between your machines you should be able to have seem synced between platforms. I do personally use Syncthing but not for FreeTube data so I'm not sure how well it would sync if you tried having two machines make different changes to the files at the same time. This would likely be something we solve when we do a proper sync solution if this does become a problem. Either way using Syncthing should work fairly well if you're willing to set it up.

As for other syncing methods, it's still something I'd like to do, but it's going to be an after v1.0 type of feature, so it may be a while.

I think we're all looking forward to this. One of the biggest hassles has been syncing up freetube across my multiple devices (On top of syncing it to NewPipe as well). It's just overwhelming to have to deal with it. I'm excited for the day I see it happen though.

BeatLink commented 2 years ago

Have both teams considered https://github.com/39aldo39/DecSync?

mattfj10 commented 2 years ago

Have both teams considered https://github.com/39aldo39/DecSync?

Hmm, I've actually never heard of those, I'm going to star this repo and keep up with it!

stfnx commented 2 years ago

I just wanted to mention that I'm also interested in having this functionality.

However rather than trying to coordinate a sync function between FreeTube and NewPipe I'd create a kind of "standardized" file format that can be exported and imported in various applications. Then also other applications could integrate it easily. This could be a json file with pre-defined structure. Every YouTube client/whatever could implement it and everybody would be happy.

If its a simple file that can be exported and imported, it could also easily be synced with Syncthing, Nextcloud or any other file sync solution.

GlowingUmbreon commented 2 years ago

It looks like the only stuff either application requires is the following:

Playlists

playlist name
playlist children [
  video name
  video id
  video duration
  channel name
  channel id
  video view count* (Does not seem like a necessity, it would be inaccurate anyways)
]

Subscriptions

channel name
channel id
channel thumbnail* (unless there is a way to directly get thumbnail based on id)

Potential design

playlist_name.csv

id, name, duration, channel_name, channel_id
HnqznXKPqSE, [Hardcore] Sunhiausa - Nothingness, 335, Sunhiausa, UCCCULnxnWjBMhYP6-x4NvwA

subscriptions.csv

id, name, thumbnail
UCXuqSBlHAE6Xw-yeJA0Tunw, Linus Tech Tips, https://yt3.ggpht.com/ytc/AKedOLSZnNA3eg_fDwUgsCxpqMnVWu3UGo-Rln4621ncIQ=s176-c-k-c0x00ffffff-no-rj

Design choices:

Uses csv as the file format, csv is very easy to parse and can be done easily by both applications and external scripts, csv also uses a newline to separate elements this makes it easier for syncing tools to detect changes since they can just assume a line that got added means a new subscription or a playlist was added.
Every playlist uses a separate file, Having every playlist as a separate file makes it simpler to parse and is nicer for people using it, lets say you want to share a playlist of songs to a friend you can just send them the file instead of needing to go through it and removing songs from other playlists. _'s can signify a space so Playlist_name.csv becomes Playlist name.
Exporting both playlists and subscriptions, I can see possibilities for a "Full export" 1) Just put all the files into a zip file, the advantage of this is that the contents are compressed and mean a smaller file size but the disadvantage of this is that its a binary file and would be alot more complex to parse for either tool. 2) Combining all the files into 1 file, something like this: export.db
```
playlist_name.csv
id, name, duration, channel_name, channel_id
HnqznXKPqSE, [Hardcore] Sunhiausa - Nothingness, 335, Sunhiausa, UCCCULnxnWjBMhYP6-x4NvwA
-----
subscriptions.csv
id, name, thumbnail
UCXuqSBlHAE6Xw-yeJA0Tunw, Linus Tech Tips, https://yt3.ggpht.com/ytc/AKedOLSZnNA3eg_fDwUgsCxpqMnVWu3UGo-Rln4621ncIQ=s176-c-k-c0x00ffffff-no-rj
-----
```
```
file_name
file_contents
-----
```
the advantage of this method is that it is a plaintext file and is easier to parse and keeps complexity low, the disadvantage of it is that it has no compression and will have a much larger file size.

radasbona commented 2 years ago

Hi, I'm not really familiar with this kind of thing. But maybe this would be worth a look. I use it (+ my Nextcloud) to sync my podcasts between my devices. https://apps.nextcloud.com/apps/gpoddersync Source Code (https://github.com/thrillfall/nextcloud-gpodder)

Subscriptions + viewed status + play position. Sounds like everything we need here.

Nemo157 commented 1 year ago

I was just looking at using syncthing to synchronize the existing directories, and the major issue I notice is that both the persistent data and caches are put in the same place. If the caches were moved to the correct platform specific locations this would seem much more feasible to do with simple file synchronization. (I have tried setting up ignore rules to only sync the .db files within the folder, which does seem to be working so far).

therealrobster commented 1 year ago

Just checking in to see if there's been any talk with the NewPipe people on this? I'd like to contribute, but not sure if my skills are high enough?

cRaZy-bisCuiT commented 11 months ago

Are there any updates on the matter? I'd like to share my bookmarks with NewPipe as well or at least synchronize them with my several desktop FreeTube clients. No matter if it's syncthing, git, Nextcloud...

Having a standard file type for subscriptions, playlists etc. instead of a db would help a lot here.

SamuelBMartins commented 11 months ago

I would like to let you know of the solid project : https://solidproject.org/. I believe it could be used to store the subscription and playlist data.

tkbremnes commented 7 months ago

If both apps used a common format for export/import, such as OPML, syncing could be implemented using third party tools.

gituser823 commented 7 months ago

Any updates on this?

therealrobster commented 5 months ago

They don't need to have the same format, they just need to be open, and someone have the skills and time to write a tool that can transform the data between formats. I unfortunately don't think I have the skill.

However, if I have it a crack, does anyone have a tip on where to look for both formats?

I do know a bit of python. Hmmm

Pentaphon commented 1 week ago

I think it might be possible to do a local sync with other Freetube clients on the same network (especially now that we have Freetube on Android) but it probably won't be possible for Newpipe clients. I'd be fine with just Freetube sync. That would be a huge step and we wouldn't have to depend on Piped or Invidious.

FreeTubeApp / FreeTube