LAN Sync [Migrated from owncloud/core]

[LukasReschke]

Dear all,

I am trying owncloud and it looks really great, but there is one feature DropBox has and that I cannot find in owncloud: the Lan Sync, i.e. the possibility for clients under the same Lan to synchronize directly between themselves, which limit the bandwidth.

For organizations working in remote areas with slow connection, it really makes a difference and I was wondering if the feature does exist and if not, if it would be possible to implement it.

Thanks and best regards, Pierre

Migrated from https://github.com/owncloud/core/issues/304

[RandolfCarter]

Isn't this the same as https://github.com/owncloud/mirall/issues/203 ?

[LukasReschke]

I don't think so. This issue seems to be about the LAN sync between clients like Dropbox has it.

https://www.dropbox.com/help/137/en

[RandolfCarter]

Ah yeah, this one is between two clients in the same LAN, while the other is for connecting differently between server and client according to current location of the client! Thanks for pointing it out!

[zatricky]

This feature's power is especially compelling when you have remote sites with poor connectivity. I've been using my laptop to get ease the bandwidth requirements with Dropbox - would give me less reason to keep Dropbox around if this was implemented in ownCloud. :)

I'm not sure how Dropbox implemented the feature. The KISS method I can think of is to report the private network and masks to the server - if there's a match, a simple cryptographic between clients, coordinated by the server, can confirm local connectivity.

[zatricky]

https://en.wikipedia.org/wiki/Local_Peer_Discovery (References refers to http://forum.utorrent.com/viewtopic.php?id=63567 which mentions some implementation detail used there)

The above is also a KISS approach - a simple cryptographic would "seal the deal".

[nloui]

FWIW - We setup an ownCloud server in-house and then a second one in the cloud where the data is sync'd via Bit Sync (a little convoluted) but achieves something similar. DNS within the network points to our in-house ownCloud and outside the network, it goes to the cloud-based server.

[etiess]

Hello,

I live and work in Africa, where bandwith is low and expensive: it would be a great point to have this "LAN sync". Is it part of the plans for owncloud?

Thanks!

Etienne

[danimo]

@etiess As you can see, this issue is not scheduled, so it's not on the near-term list.

[etiess]

Thank you @danimo for your feedback. We'll be patient!

[menelic]

+1 for LAN sync, this would really be an important feature in contexts of low bandwidth, high connection costs but not only there. In US, UK and GER, where internet is comparably cheap and fast, sync on LAN is still attractive for multiple device scenarios, eg a music or movies folder synced between laptops, tablets and phones.

As OwnCloud positions itself as a collaborative suite with teh documents app, sync on LAN becomes an important feature in that regard too...please consider giving it more priority

[Herve5]

+1 for LAN sync; for me it'd become a reason to switch an entire research lab here… (indeed, it's currently the reason they refuse to at this moment…)

[jancborchardt]

Is this related to the 1.6 »Sync Performance« milestone? @MTRichards @dragotin @danimo

[MTRichards]

It is likely far more complicated than we can put into 1.6. Peer to peer sync is of course interesting, but for me a feature request much further out. And, interestingly, there are those who don't want this enabled, so it has to be configurable.

[willie68]

+1 for LAN sync, too; I work with PC's, Netbook, Laptop, sometimes all within the same network. So a quick sync of a client over the LAN would be desirable. Actually i'm just waiting on my new laptop to sync all data. Just 7GB, but i think it will take 32hours... (The server has only a 384kbit upload rate) My netbook with all data in sync is just 20cm away.

[benedikt-buchert]

This seems to be an interesting solution to do this in case you are using your own server: http://adammatthews.co.uk/2013/05/owncloud-bit-torrent-sync-dropbox-clone/

[zatricky]

This brings across an interesting point: People have opted to bypass ownCloud's builtin "sync" entirely because it is inadequate? :-(

FWIW, I'm not experimenting with ownCloud any more. I realise ownCloud has uses other than file sync - but btSync has had me sorted for a while already.

[Asuquo12]

+1 LAN Sync, This is very key for my organisation and for Owncloud as well. LAN Sync is the single reason we are looking at Dropbox. Please lets get this pushed up the priority ladder.

[ghost]

+1 LAN Sync!!

[muetze-online]

Me too. Mostly my two synced Macs are at different locations, but often they are in the same network. If I sync 2 GB of new files, they will be uploaded to the internet - that must be - but then they will be downloaded for my second machine from the internet, allthough the data is present via a gigabit ethernet. When dropbox offered this feature, I was very happy. Now I'm on ownCloud - far beyond dropbox with some needed features (but better in an overall privacy relationship). You get the users with speed and simpleness.

[Herve5]

Muetze, I hope what we write here will be read (we have little else to ask since we are not developing ourselves indeed) but for regularly checking this feed, what I understand is, a guy name Deepdiver closed the action here referring to another thread, that he also closed the same day. He seems to behave like he really don't like the issue.

Will Owncloud die by lack of simplicity I don't know. I definitely continue to use it since it is, well, the only open alternative… The world is not so large after all :-(

[muetze-online]

This issue is still open - see on the top. The other issue is closed, because it is a duplicate of this issue.

[tflidd]

There is even an open-source alternative to bittorrent-sync: http://syncthing.net/

[step4wd]

+1 for LAN Sync but is there any news/update on the roadmap?

[Herve5]

Still definitely interested in LanSync -for my younger contacts that's definitely the thing preventing them to abandon Dropbox, for instance. Step4wd the last large update is quite recent IMHO (months), I didn't even switch myself. Let's hope :-)

[ghost]

Didn't see an answer here yet about the Dropbox way of LAN sync: Dropbox regularly sends broadcast packets to the network, announcing that it is running. What I don't know is how they communicate who has the most recent version of files... didn't inspect that yet, I only saw the announcement packets when wiresharking my network. LAN sync would be great!

[strugee]

Didn't see an answer here yet about the Dropbox way of LAN sync: Dropbox regularly sends broadcast packets to the network, announcing that it is running.

What kind of answer did you expect?

[zatricky]

@zeero4ever - from my comment on 27 Mar 2013:

https://en.wikipedia.org/wiki/Local_Peer_Discovery (References refers to http://forum.utorrent.com/viewtopic.php?id=63567 which mentions some implementation detail used there)

This isn't necessarily how Dropbox does it - but it is the most likely method since it is the simplest.

[oxivanisher]

Another question is, if the clients should transfer files without the knowledge of the server, or if the server even should orchestrate it. So that it tells clients A and B to transfer the file X between them. The fear a hacked client which can fetch all files from clients in the LAN could be ruled out this way.

[benedikt-buchert]

I think it would be nice to have peer to peer sync included into own cloud in general not just for lan. This could also result in a big speed increase for a folder shared with a group. If this could be switched off on client and server side would be amazing. I know that there is torrent sync to achieve this. But own cloud has some other advantages which i don’t want to be miss.

Am 08.01.2015 um 10:44 schrieb Marc Urben notifications@github.com:

Another question is, if the clients should transfer files without the knowledge of the server, or if the server even should orchestrate it. So that it tells clients A and B to transfer the file X between them. The fear a hacked client which can fetch all files from clients in the LAN could be ruled out this way.

— Reply to this email directly or view it on GitHub https://github.com/owncloud/client/issues/230#issuecomment-69156829.

[danimo]

@oxivanisher We want the server to have full control. That's why we can't implement this now. The server has to be prepared for it.

@eundd While pure peer-to-peer sync might be nice, it is not what we envision for ownCloud, as it puts the server operator out of control of the data. You may want to try bittorrent-sync or other solutions instead.

[Herve5]

The issue is not really peer-to-peer: it's the fact that, when two sync'd machines appear to be on the same LAN, Owncloud in this case only appears pathetic when compared to Dropbox, and this case happens very, very often both in research and industry. I know personally a dozen people that'd obviously chose Owncloud against Dropbox and renounced just because of that very feature. It's to the extent I don't dare advertising for us, or at least very prudently now :-(

[sharadmittal]

I have a suggestion to implement this in a relatively easy way: 1) the current csync based approach does not allow for secure peer-peer chunk transfers. 2) How about we add a 2nd download option based on bittorrent protocol. libtorrent (C++) can be used for this. 3) libtorrent supports SSL cert based auth for the tracker, web seed and peers. 4) The owncloud server can act like the tracker and web seed. It can also issue peer certs to every client. 5) In libtorrent, we can add an extension (great support for that) which exchanges a token after the SSL handshake is complete. This token could be issued by the server to each peer for authenticating that the peer has access to the particular file id. 6) This way server has control over who gets the file while the peers can exchange the pieces among themselves in a secure authenticated way.

Reference - http://blog.libtorrent.org/2012/01/bittorrent-over-ssl/

Overall, this will drastically reduce the load on the server in terms of cpu and bandwidth, and make owncloud much more useful in low internet connectivity situations.

In terms of code complexity, we could integrate with libtorrent and add an extension which validates a small token. I don't imagine it being a huge effort.

Any comments/questions on the approach?

[danimo]

@sharadmittal This conflicts with the requirement that the server needs to have the last word on which client is allowed to received a certain file.

[Herve5]

@sharadmittal I'm 100% favorable to this idea; in case the issue described by danimo is a concern one could perhaps add a specific setting for enabling/disabling this -but I must say I don't really understand that concern indeed, as long as the server does remain a central hub, to quote sharadmittal: "this way the server has control over who gets the file…"?

[sharadmittal]

Thanks @danimo for your feedback. How about the peers make a quick check with the server whether the connecting peer has access to that file (using the authenticated peer id)?

Even the server has no control after it gives out the file to a client. With what I'm suggesting, every client will check with the server before starting the file transfer to a peer.

[shoeper]

The client could send a list with the owncloud users in it's network (or just id's) with each download request. The server could then say ask client id x and use token y. The client requests the file from oc user x using token y via some encrypted channel. Before sending the file oc user x asks the server if the token is valid for the request file and if it was requested by the user wanting to have the file.

To get better speeds there could also be bulk mechanism where the client gets a token to receive 50 files and oc user x can verify that the requesting user has the permissions for those files with just one request.

And there could be a token valid for a day being used for folders being shared between two users.

[danimo]

The basic principle is clear, what's missing is an authorization protocol in the server. @LukasReschke and @DeepDiver1975 have a concept for that I think, but it's not there yet.

[sharadmittal]

It'll be great if @LukasReschke or @DeepDiver1975 could present/discuss their strategy here.

I've another simple way this could be done. Server encrypts the file once which then gets downloaded using a bittorrent like protocol. Clients have to get the encryption key from the server directly (using regular https authenticated route). This way the server gets to have the last word. A side effect of this approach vs the original approach i suggested is that the peers have to keep 2 copies - one encrypted for sharing and other decrypted for local consumption. This issue can be reduced by keeping a time limit on how long a client needs to "seed".

This approach is also mentioned in here .. http://blog.libtorrent.org/2012/01/bittorrent-over-ssl/

[jalcine]

Realistically, ownCloud's designed afaik to have the server be the authorative source of all files. LAN Sync breaks this since there's no querying of the server about the status of the files updated. Now, if it was possible to ask the server what files have been changed from the last provided timestamp and use that to determine between local clients what to sync, that'd be dope. But again, authorization would have to be in the hands of the server. I'm assuming that a time-based key, not necessarily OATH but something like this can be issued to each client and used to identity each client to one another, thus (kind of) solving the issue of determining who's who. For the paranoid, an option can be provided to have the client get a new local sync token after a period of inactivity of local sync.

It's not an easy problem :wink:

[lordloh]

Drop box tan sync does need the cloud. So, the cloud is still the authorative source.

[sharadmittal]

@jalcine - I have covered the server authority in both the proposed solutions. At no point will a peer provide file bits before checking with the server or the bits being clear text.. Hence the server is practically and technically the first and final authority.

Regarding the file change, the peer sync only works for a particular file version... The clients will find the current version number/identification from the server and will only look for pieces of that version .. The time based token mentioned by you is definitely one approach .. The other approach is having the peers verify the connecting peer's cert/token every time with the server to be bulletproof...

[step4wd]

So in short this feature will never be introduced in ownCloud?

On Fri, Mar 6, 2015 at 12:12 PM, sharadmittal notifications@github.com wrote:

@jalcine https://github.com/jalcine - I have covered the server authority in both the proposed solutions. At no point will a peer provide file bits before checking with the server or the bits being clear text.. Hence the server is practically and technically the first and final authority.

— Reply to this email directly or view it on GitHub https://github.com/owncloud/client/issues/230#issuecomment-77516741.

[shoeper]

Where did you read that?

[jalcine]

So in short this feature will never be introduced in ownCloud?

That’s a hell of a stretch, don’t you think? This issue is more for brainstorming an implementation strategy.

[dragotin]

Once we have checksums about file contents we can think on how to implement this. How is pretty straight forward: By doing the discovery against the server, the client gets to know which files it needs. With that information it can do a broadcast along the line "who can deliver me file with checksum xyz?". If it gets a positive answer, it might be able to download it from a different client rather than from the server.

On a general note, about the "will never be implemented..." comment: ownCloud is a open source software project, so you can never say that. Somebody might step up and simply do it.

[sharadmittal]

Wanted to mention that libtorrent could potentially be used to have clients use the torrent protocol (with extensions). It supports SSL, client-server certificates and other stuff.

[jalcine]

Re: @dragotin

Once we have checksums about file contents we can think on how to implement this. How is pretty > straight forward: By doing the discovery against the server, the client gets to know which files it needs. With that information it can do a broadcast along the line "who can deliver me file with checksum xyz?". If it gets a positive answer, it might be able to download it from a different client rather than from the server.

This sounds like a good idea, strategy wise to me, and actually does kind of sound like the act of torrenting, tbh. So this would require the server to accept a route that a client could send with the most recently changed checksums of files that the client has and determine which ones are out of sync then initiate that torrenting action.

I genuinely think something like this wouldn't be insane to prototype, and thanks to the current flexibility of ownCloud, one could make a separate app and client app to test if this is functionality that could be made main-line.

[wizdude]

As a side note, Windows 10 has implemented "WUDO" - Windows Update Delivery Optimisation. This uses a bit-torrent style peer to peer delivery of updates between machines.

More and more product delivery and sync style products are now utilising local peer to peer technology. It would be great for Owncloud to consider the same.

Perhaps one solution would be for the server to have an index of all files that exist for each sync client. Some form of hash could be created for these that also incorporates the node IP address. This would be provided to the sync client at the same point where a file download would be occurring from the OC server. it could then try and transfer the file locally (with WAN failover as well). This index could be populated by the clients, perhaps by the use of a modified bittorrent style tracker.

This method, while obviously not the most optimal, would give the OC server the final say in all of this.

I'm personally a big fan of the way bitsync works. the server runs a tracker so each node knows where other nodes are. when content is synced it comes from LAN, WAN or wherever to give the best possible transfer rate.

[danimo]

@wizdude See https://github.com/owncloud/client/issues/230#issuecomment-74153667

[phil-davis]

+1 from me for something along these lines. Adding my use case here as it will be different from the "1st world" use cases. In our use case we have very remote offices where we currently have a small server, pfSense router and OpenVPN back to main office. The remote office internet is really crud - comes and goes and when it is there it nearly always has 5-20% packet loss. Nepal Telecom is the only provider in those areas, so apart from crazy expensive satellite, some internet must be better than none, maybe. In some towns they have local hydro power. We have had the power station or dam or... broken for weeks at a time. The local Nepal Telecom does not have enough solar/battery to run the phone system plus internet services, so the internet is off for a few weeks. Enough background - you get the idea. It is proving very difficult to support any sort of "modern network" in these remote places - power troubles, server troubles, router troubles, cables pulled out by random staff... So we are thinking to simplify and give these offices just a basic internet device with WiFi (ADSL/WiFi/4-port LAN) combo device and no server... If we can give them ownCloud to share files and have LAN sync, then a file changed by 1 user in a remote office can be uploaded once but then also update all the other laptops in the office locally.

And for us it would be really nice if local LAN sync could somehow happen when the internet is down. Somehow 2 clients can know which folders they have shared in common, and hold some authentication information that lets them trust that the other client is an authenticated client of the same server... If that would not be possible to build into the base ownCloud Client in a secure way, then an extra app would also be fine.

I am happy to be involved in helping with anything along these lines.