[Feature Request] save your playlists, subscriptions and settings in the cloud

[ghost]

• [x] I carefully read the contribution guidelines and agree to them.
• [x] I checked if the issue/feature exists in the latest version.

I'm not sure if this goes a bit against the policy of Newpipe, but I think it would be a very good idea to make a server in the style "firefox-sync" in which you can register and store your data there to be able to synchronize them more easily in other phones, tablets and AmazonTvs. maybe even this idea works also for when Newpipe is ported to computers

[theScrabi]

Well im currently working with ownCloud on their android app, lets se what the future will bring ;D

[ghost]

great, I'll wait for news :grimacing:

[gabriellluz]

My suggestion is to integrate Google Drive and OneDrive so that NewPipe syncs subscriptions, history in an encrypted file so that we can access it anywhere. Right now I use OneDrive but I have download it and upload once a week. It's kinda boring

[theScrabi]

Well an idea would be some sort of intent interface for the owncloud app, so newpipe could trigger it to sync its database. I'm just not sure if ownCloud would like that 😅

[gabriellluz]

It'd be like an OwnCloud app or extension? something like that? it would be cool BUT I don't use Owncloud anymore because they collect your data. So your best best is to create an encrypted file (end to end). Otherwise, OwnCloud exposes our data so that NSA can create a simple tunnel/scraper in order to grab our data. They do it currently with GMail, Outlook, GDrive and so many other providers according to Edward Snowden. There's a book I read about it.

[theScrabi]

@gabriellluz I am not going to discuss anything regarding ownCloud here at the NewPipe project. But if you have doubts or questions feel free to ask over at owncloud central or write me a mail.

[trymeouteh]

I would love this feature since I would not have to worry about losing data if my phone is stolen, lost or stops working since it will constantly sync itself to a cloud and everytime I add a subscription, a bookmark or create a playlists it will be backed up.

Same goes for my Tablet. I would like to have my bookmarks and subscriptions synced between them for easy use instead of having to export and import the data all the time.

The only issue with cloud syncing is there are so many providers and I find that not many are using WebDav and others like myself have their own trusted cloud providers since many cloud providers are not privacy friendly like Google Drive, OneDrive, etc.

Some people host their own NAS and use ownCloud or NextCloud but not everyone has this nor wants this setup for privacy. I personally like using MEGA and Sync.com. But to ensure privacy and to also ensure everyone gets to choose their favorite cloud with ease I think the following needs to be done especially for cloud storage providers that do not support webdav.

-File Files are the easiest way to contain data and have then sync to various clouds. Every cloud storage system supports files and many cloud storage apps support syncing files from a directory on a device to the cloud. And even if your cloud storage provider does not do this there are apps that do just this. Maybe users want to use Syncthing and this will make things easy. But for the file to work the user must choose where on the device the file is stored and the name of the file

-Files Same as the suggestion above except you do not sync the entire file between the cloud and devices since a file can get large overtime take longer and longer to sync. Instead you sync files with updates that say (Unsubscribe from PewDiePie 20181020-204455) or (Subscribed to Halo 20181020-204500) and when another device recieves the file it will do as it says and keep the file for sometime.

-Encryption This can be optional but the file can have a password set to it so to access it in NewPipe you must enter the password to read/write to the file. This is needed for those who use non-private cloud storage systems so the cloud storage provider cannot read the NewPipe data nor can any eavesdroppers see what the data is when being send to/from the cloud.

Personally I would prefer to use MEGA with an encrypted file since MEGA is pretty good on privacy but if someone does get my NewPipe file they cannot see my bookmarks and my subscriptions.

The reason to keep NewPipe data private and secure is because YouTube and your bookmarks, subscriptions could tell a lot about you just like a years worth of search results. NewPipe is designed with privacy in mind since it does not use a Google Account but NewPipe.

That is my opinion on syncing data to a cloud or between devices. This method above I have explained can allow NewPipe to support many cloud storage provides webdav or not webdav, self hosting, payed cloud storage, syncthing, etc. There are so many cloud storage providers and ways people like to manage their data and it will be a daunting tasks to add support to every cloud storage provider and every self hosting method.

[kroeliebuschie]

here's an idea: newpipe automatically makes a local history/subscription backup in a certain directory, you install the nextcloud app and select this dir. for autouploading. This keeps the 2 apps separated. You could a description of the process in the settings menu under something like "cloud backup"

[steveej]

here's an idea: newpipe automatically makes a local history/subscription backup in a certain directory, you install the nextcloud app and select this dir. for autouploading. This keeps the 2 apps separated. You could a description of the process in the settings menu under something like "cloud backup"

I second this simple solution! Pair that with an auto-import setting and make them mutually exclusive to avoid conflicts.

[Richard-Payne]

Any news or progress on this? Cloud sync (NextCloud would be my preference) is the major reason why I don't use NewPipe atm.

[theScrabi]

1. Not yet, but if so we would need to find a way to sync with any cloud system. Maybe this was possible with SAF, but for that we first need to switch to SAF with saving/restoring the database.
2. Since this discussion was started before Invidio.us came on the plan. Now maybe we should build a backend to invidio.us instead. This would even be better as syncing events like "watched this video" is better than scncing the whole db, because what would you do if you produced a colission? @omarroth is there a REST API for invidio.us to which we could connect to and sync account data with?

[Richard-Payne]

2. Since this discussion was started before Invidio.us came on the plan. Now maybe we should build a backend to invidio.us instead. This would even be better as syncing events like "watched this video" is better than scncing the whole db, because what would you do if you produced a colission? @omarroth is there a REST API for invidio.us to which we could connect to and sync account data with?

I'm not familiar with invidio.us but it appears to be a competing video site. I'm not sure how supporting that is relevant to syncing youtube data.

[theScrabi]

Its selfhosted free and we work with @omarroth.

[omarroth]

I expect @PrestonN and @cloudrac3r will also be interested and have some thoughts on this.

is there a REST API for invidio.us to which we could connect to and sync account data with?

Yes, see omarroth/invidious#473. It's currently not very well documented unfortunately, but it is absolutely possible to sync subscriptions and user preferences. Watch history will likely be available in the API once omarroth/invidious#673 is merged. Are there any other endpoints that would need to be added?

[Gymcap]

@Richard-Payne Invido.us is a bit like a self hosted mirror of youtube, not a YouTube competitor exactly. You can watch any YouTube video on it by switching invidio.us for youtube.com

It's a project trying to FOSS YouTube as much as possible like NewPipe, except on the web.

That being said, if I could host my own invidious instance that feeds all my devices with their subscriptions and watch history, that would be extremely nice. Even better if we could use the Invidious instance for browsing instead of YouTube.

[PrestonN]

Obviously at the end of the day it's your decision on what you end up doing, but I'd like to throw WebDAV in here as an option since it would allow synchronization with Nextcloud, Owncloud, and any other services that support the protocol (There are even plugins to make it work with larger services like DropBox). I will personally be attempting to support this in some future update of FreeTube though it might be a little ways from now before I get there.

Also this is just a bit of bias, but it'd be pretty neat to be able to sync up subscriptions, history, and such between NewPipe and FreeTube. This would require more communication of course but I may as well throw in the idea. I'd expect that there would have to be changes outside of just cloud sync on both sides in order to make that happen but that's discussion for a different day I suppose.

[Gymcap]

+1 for WebDAV

[cloudrac3r]

With so many different alternative YouTube apps and websites I think it's important to create a solution that all of these apps and websites will be able to implement if they want to.

While websites can obviously provide an sync API to POST and GET from, apps can't. This would mean data would have to be sent to some sort of intermediary service by the first app, and fetched from that same service by the second app.

I don't know enough about WebDAV to complete this thought, but I do agree that being able to use a regular file cloud as a synchronisation store sounds like a good idea.

[cloudrac3r]

The other main option would be to create a brand new selfhostable service for storing sync data. It could have an API that all apps and websites can POST data to and GET data from.

Designing a brand new service would allow for API endpoints like "what has changed I last asked", which would be more convenient than a flat file stored in the cloud. On the other hand, requiring a custom service like this one rather than a file cloud would make it a little more inconvenient for users who want to selfhost.

I think that this service should be separate from Invidious, because having to run a full Invidious node just to synchronise data between NewPipe and FreeTube would be silly.

[theScrabi]

Well if someone writes this service it would be ok, but since invidio.us is already written I'd focuse on that.

[poperigby]

I think being able to login with your invidio.us account to sync would be great!

[yausername]

Related PR - https://github.com/TeamNewPipe/NewPipe/pull/2668

[ushername]

With great interest I have been following the discussion lately. What is the current status on deciding/implementing WebDAV as a sync option? I agree with @PrestonN this has a wide range of applications.

Mind my enthousiasm. It would be fantastic to be able to sync invidio.us with freetube, newpipe, etc. all with WebDAV. The end user is in control of its own history subscriptions, etc. so the legal aspect isn't a big thing to worry about. Because people host their own data.

The difficulty to implement & maintain it this way is also relatively easy I suppose, WebDAV is known for some time.

[rakshazi]

Hello, invidious user here.

Right now, invidious allows to import data from NewPipe (both zip and json), so it will be very good to use it as "synchronization backend"

[smnthermes]

It would also be good an option to sync with Syncthing.

[BeatLink]

Are there any updates on this? I think the best solution would be to build a syncing api for invidio.us. Theyre floss, free, already can do import and export with newpipe and can be self hosted. its a win win!

[T-vK]

I'd really like to have syncing between NewPipe and FreeTube. I personally vote for WebDAV because it would allow to sync arbitrary settings or databases, not just subscriptions, playlists and watch history...

I wrote a proof-of-concept for WebDAV synchronization for FreeTube earlier today and it was really trivial. https://github.com/FreeTubeApp/FreeTube-Vue/issues/44 I don't know much about Java, but I'd imagine it to be equally trivial to implement in Java.

Maybe we could agree to save the subscriptions, playlists, watch history and whatever else FreeTube and NewPipe have in common in a specific format that everyone has to support.

If you really prefer invidious over WebDAV I'd be okay with ditching my WebDAV poc and writing a new one for the invidious API if I can find documentation for it.

[cocoonkid]

Yes please! webdav FTW

[T-vK]

Very unfortunate that the devs aren't replying.

[Rik44444]

hello all! Thumbs up for webdav sync! Also a solution like this like Antennapod does might be possible https://gpodder.net

[T-vK]

I have integrated my proof of concept for WebDAV synchronization into FreeTube some time ago, I think the only thing that was missing was automatic synchronization (as in you had to manually click the synchronize button). When I'll have some more time I'll get this merged....

I would really love to see someone do the same for NewPipe. But it seems the NewPipe developers don't care. I still haven't heard a single word from them.

[opusforlife2]

But it seems the NewPipe developers don't care. I still haven't heard a single word from them.

The one you were interacting with was Schabi, who was the lead dev at the time. So you did read several words from a Newpipe developer, though you're correct that you didn't hear them. :P

But Schabi has moved on to other projects, so all these issues which are pending have to be brought to our attention one by one. That's just the way it is, sorry.

Anyway, for a feature like this, we need a lot of discussion, which sometimes happens in Github issues, but sometimes happens mainly on IRC. Could you join the Newpipe IRC channel to talk about it?

For example, there is also #3249 to consider.

[Rik44444]

I have integrated my proof of concept for WebDAV synchronization into FreeTube some time ago, I think the only thing that was missing was automatic synchronization (as in you had to manually click the synchronize button). When I'll have some more time I'll get this merged....

I would really love to see someone do the same for NewPipe. But it seems the NewPipe developers don't care. I still haven't heard a single word from them.

cool, is there a GIthub issue for this on Freetube? i could only find this https://github.com/FreeTubeApp/FreeTube/issues/370

[pawlosck]

https://github.com/FreeTubeApp/FreeTube-Vue/issues/147 - It's my issue for dev version of Freetube(Vue). Currently code was moved to official version of Freetube, somust be created new issue if doesn't exists. As you can read in my issue, author of Freetube know about this feature.

There's a member of the community already working on WebDAV support for FreeTube. If it's something that the NewPipe team would like to be involved in then they are more than welcome to be.

[Dangelman]

Would be nice to have NewPipe export the database automatically via E-Mail or Google Drive

[seniorm0ment]

Any updates with this?

[Feuerswut]

really the only thing NewPipe needs to do is auto-write newpipe.db.zip backup file recurringly in a predefined time interval (e.g. 1 day). The rest can be done with other apps.

[seniorm0ment]

It would be nice to be able to selfhost some sort of minimal api that it syncs with, then users can just input this url on any device or site like invidious and have everything constantly synced.

[emmatebibyte]

+1 for syncthing/decsync

[h4n23s]

As commented on #5325, the Nextcloud Bookmarks app could be a solution (apart from synchronizing settings) to this issue.

[blacklight]

I'm not sure if it's been considered/discarded already, but how about syncing playlists and subscriptions to a Piped instance that supports user registration?

I only see two downsides:

1. Not all the Piped instances allow user registration (but it's relatively easy to host an instance)
2. This method only works for YouTube content, unless Piped also implements support for other sources

Besides this, it sounds like the most straightforward solution to me. The app may optionally provide the user with the ability to log in to a Piped instance, and then:

1. Synchronizing the app subscriptions to Piped is just a matter of calling POST https://<instance>/subscriptions with a payload containing the channelId
2. Synchronizing the upstream subscriptions to the app is just a matter of polling the feed at https://<instance>/feed/rss?authToken=<token>
3. A similar strategies works for synchronizing playlists too

No extra apps nor integrations involved. I also thought of using Syncthing or Nextcloud as an intermediary, but if we already have upstream Piped instances that support subscriptions and playlists, why don't we just use those?

[h4n23s]

I'm not very familiar with Piped, but does it support platforms other than YouTube?

If so, this would probably be easier to implement, as you'd need to find a clever way for bookmarks to store advanced metadata (e.g. number of clicks, etc.).

Some information could be embedded in the bookmark's folder structure (e.g. the service type, like YouTube or PeerTube) or the actual URL (e.g. where to continue watching a particular video), but it's generally much more complicated.

Still, I like the idea of being able to sync your subscriptions and playlists with your browser's built-in bookmark store.

Edit: Sorry, I overlooked the part of your comment where you mentioned that Piped only supports YouTube. Bookmarks are, of course, much more flexible in this regard.

[QJKX]

Still, I like the idea of being able to sync your subscriptions and playlists with your browser's built-in bookmark store.

It's a nice idea.

Unfortunately the firefox sync design is a simple server and a complicated client, and acccess seems to be all or nothing - it doesn't seem possible to just give a client just access to synced bookmarks but not synced passwords. https://mozilla-services.readthedocs.io/en/latest/sync/index.html

[shvchk]

EteBase is a good FLOSS option, IMO.

[BeatLink]

The best solution IMO is to build the capability to log into and sync changes with Invidious instances and use that as the server back-end for syncing. Its free, open source and along with Piped is one of the most popular YouTube front-ends.

[derei]

What about plain old webdav (or specific cloud client implementations), and encrypted database with a passphrase? Most (if not all) users use some sort of cloud storage. And yes, some services have no regard for privacy... But an encrypted db would solve that issue.

I agree this is not the most elegant solution and definitely some users won't be able to benefit of it, but it's a working and not too difficult to implement solution. It could be implemented until a better, more seamlessly integrated solution arises, and after, it can remain as a secondary option.

[cocoonkid]

I commented here 4 years ago and was for WebDav but since then I've come a long way and now would like to throw Syncthing into the mix.

I use it for all my sync WeekToDo, Obisidian.md , Freetube and they all work like a charm.
Took a little bit to finetune syncthing settings to avoid collisions (left with 2-3 a year now at most and quickly fixed). Syncthing will not corrrupt any files, just create dupes.

Syncthing also supports android so that should actually work just like this.

Just had no reason anymore to test myself since I just got used to use my phone only for calls and banking and that's it. Flippin' waste of time and mind. Can recommend :-)

So basically just leaving this here for others to try out :-)