Request: Mirror federated storage locally [$20]

sunjam commented 6 years ago

Steps to reproduce

Mount remote federated share 2.take remote federated share offline 3.attempt to access remote federated share data locally

Expected behaviour

Federated data from remote host will still be accessible from the web client while the remote host is offline. Federated share owner can allow those with remote access to keep local copies of the shared data.

Actual behaviour

All access to federated remote shares is lost until host goes back online.

Server

Nextcloud 13 and Nextcloud 12.0.5

There is an open bounty on this issue. Add to the bounty at Bountysource

andreicociuba commented 6 years ago

wouldnt this be a privacy/security concern?

i mean, federated content can be taken down for legitimate reasons. how do you propose to ensure that any caches are taken down when that happens?

sunjam commented 6 years ago

What I’m requesting is persistent federated shares. The concern in this case is not security, but availability and uptime.

Example: most admins might very well have multiple federated instances of Nextcloud with no concern over internal content needing to be taken down. The issue in this case is when content needs to stay up for legitimate reasons, but having shares on a single instance becomes a point of failure by going offline.

MorrisJobke commented 6 years ago

cc @nextcloud/sharing

The last state here was that we don't want to implement the whole syncing logic on the server as well and that it then would also work against the "your data is on your server" principle we have right now.

e-alfred commented 6 years ago

@MorrisJobke Well, this stance is a bit of an "security by obscurity" thing because if I share data to another Nextcloud they can copy this data anyway (for example if they download it with the desktop client or simply copy it in the web interface let alone automatic snapshotting on Windows or Linux or just making screenshots of documents/images, I think there are 100s of ways). If the data is handed out to a foreign entity, there is no control over what happens with it. Everything boils down to a technical solution for a social problem (trust in those that the data is shared with) which mostly won't work except maybe for superagressive DRM (which also doesn't work at the end of the day because people go against it with enough determination and it is contrary to FLOSS principles).

Aside from that, allowing sync data between federated servers could be a very useful and easy backup solution and take away the problem of the desktop client showing errors all the time if a federated share is offline even though ones own server is fully functional.

MorrisJobke commented 6 years ago

@MorrisJobke Well, this stance is a bit of an "security by obscurity" thing because if I share data to another Nextcloud they can copy this data anyway (for example if they download it with the desktop client or simply copy it in the web interface let alone automatic snapshotting on Windows or Linux or just making screenshots of documents/images, I think there are 100s of ways). If the data is handed out to a foreign entity, there is no control over what happens with it. Everything boils down to a technical solution for a social problem (trust in those that the data is shared with) which mostly won't work except maybe for superagressive DRM (which also doesn't work at the end of the day because people go against it with enough determination and it is contrary to FLOSS principles).

True point. 👍

There is still the technical issue with keeping things in sync and how to deal with conflicts. It basically means we need to implement the sync engine of the desktop clients on the servers as well.

nachoparker commented 6 years ago

This would indeed be very interesting! +1

rullzer commented 6 years ago

The problem is also that we can't enforce a lot of things on the receiving end.

While it is true that once you have access you can just copy the files. But then of course there are more things at play. For example the data owner might have acces rules in place. Of have rules in palce that forbid the upload of certain files.

sunjam commented 6 years ago

Hi, I'd like to mention a couple scenarios I've found. Thanks for all the comments!

The single point of failure that plagues self hosters running a small Nextcloud server.
The individual points of failure that occur when a series of federated Nextclouds attempt to federate a folder between them... I've realized how I lack the infrastructure of Dropbox or Google by using a Raspberry Pi, but being able to distribute data with systems I choose can address this problem.
You spin up multiple Nextcloud instances and actually do own + admin all of them. Mirroring my data between them is the first thing I want to do.
You share a photo publicly via url with other Nextcloud servers who could help host it, but your computer goes offline.
You would like a secondary instance to have a copy of the federated data, even if it is not regularly synced. It might not be fully up to date, but it certainly beats everything having gone offline.
Getting to interact with files that are stored locally is easier in Nextcloud. :)

putt1ck commented 6 years ago

Some conversation around a related issue was had sometime back:

https://help.nextcloud.com/t/hybrid-on-prem-and-cloud-replication-of-storage/2773/24

Biased, me, but I think NC level replication is a really robust approach that is ideal for orgs with multiple offices; covers offsite backups and availability when connections are down. Would need further thought on how to replicate DB tables for apps (a table that apps can chose to write "and me!" notifiers into?), but otherwise it's not a problem that hasn't already been solved.

bpcurse commented 6 years ago

If mirroring was implemented, a "do not cache on federated instance" flag, that all nextclouds respect, should be considered. Meaning that either the sharing instances' admin can set this globally and/or the user as a simple checkbox "per share". As mentioned above this should not be seen as a security measure but more as a "control where the data is physically stored and who has internal access" feature for GDPR compliance.

jcklpe commented 5 years ago

I'm very much a newbie here so maybe what I'm fixing to suggest is gibberish, but I'm curious if this feature could be implemented using some kind of p2p technology similar to bitorrent?

Instead of distinct servers sharing access to data through federation, each one only storing their own data, you could have the servers in the federated network acting as a cluster, each of them maintaining redundant copies etc, so if one of the servers goes down, even the server that "owned" the original data would be able to recover that data from copies on the other servers. You could have it so that instead of downloading that data from a single server, it could actually download it from all the other servers (similar to how bittorrenting allows you to connect to thousands of other people "seeding" parts of the download to you).

You could perhaps even use this as a sort of rudimentary CDN system? So if you and 5 other friends who all live in different geographical locations were to federate, you could store the large binary assets for a website (images, video, etc) in the federated shares, and then the website could source from whatever particular server was nearest to it in order to get the best performance possible.

Currently the only thing that keeps me from self hosting my own website is because of how easy my host makes it to use cloudflare CDN, and that's an absolute must to make sure my website feels nice and snappy. But I have friends who live all over the world, and I could see it being possible for us to create our own ad hoc CDN through something like this.

I also have read that it's possible to self host your own CDN, but I'm pretty sure that's "self host" in the "control of the machine down to the root level" and not self host in the "own the physical machine and have it in your bedroom closet" kind of way.

Again though, I'm not the best when it comes to backend stuff so maybe I'm just talking nonsense. It feels like a p2p powered CDN would be a HUGE deal though. It would give NC a huge extra value added proposition. People could form federated alliances that provided distributed server space on the level of Google or AWS cdn etc. That would be a huge deal.

sunjam commented 5 years ago

@jcklpe Hi, what you are requesting is different from this request. I'm requesting that the Nextcloud web app optionally store federated data locally. This is to keep access to the data or could possibly act as a basic mirror if that federated data goes offline.

What you are requesting is interesting, but should be posted as a separate thread. See this issue #11653 requesting Zot protocol support (account mirror, merge, remove across many servers, aka Nomadic Identity) in php and this issue requesting greater integration with Syncthing #8384

sunjam commented 5 years ago

Bounty Added! Please consider donating to it.

QEDeD commented 4 years ago

The bounty is now 60$ @sunjam

sunjam commented 4 years ago

The bounty is now 60$ @sunjam

Looks like bounty was raised for Zot integration, which is awesome. Request here is basically to optionally allow admins to store copies of federated, remote files. Currently, they are only remotely accessible from the webui + literal copies are stored across desktop and mobiles devices.

plaxon commented 4 years ago

Bounty added (15$)

Spartachetto commented 4 years ago

I was thinking to the issue of nomadic identity and, searching for it I found just this issue. It seems to me that we have two potentially different aspects:

a local copy of federated data
moving my data from a server to another permanently, while maintaining everything (tags, comments,...)

This issue is on the first point but I did not find anything on the second. If you think that I have to open a new issue, just tell me.

The advantages of "moving my data" permanently are several:

as a single user I can easily change provider
as a "single install" I can buy bigger hardware, install a new Nextcloud instance and then transfer everything without having to deal with the database
as a big organization I could decide to move the data of a single user around the different instances if s/he changes office. In this last case probably I'd need a more granular approach (you can move this directory, set if contacts, ... but not this other...)

putt1ck commented 4 years ago

@Spartachetto seems to me that's a variant of the NC level replication idea - if that was built as master/master, then you up the new server, set up the replication, then break the connection and can dispose of the old one.

Spartachetto commented 4 years ago

@putt1ck technically you are right. Yet my second use case could benefit by a simpler interface. Ideally it should be so simple that a user could think of writing another account and pushing a button to move all your data...

sunjam commented 4 years ago

@Spartachetto that is exactly what Zot would provide. It is linked above. :)

radawson commented 3 years ago

+$15 for the bounty

Spaenny commented 2 years ago

+30$ for the bounty

nextcloud / server