owncloud / client

🖥️ Desktop Syncing Client for ownCloud
GNU General Public License v2.0
1.4k stars 668 forks source link

Adding ownCrypt : client-side encryption #4327

Closed orion1024 closed 2 years ago

orion1024 commented 8 years ago

Hello,

This issue is created so that we can discuss the details on how ownCrypt should be integrated in the current code, both for the initial integration and as work on it progresses.

Some discussion material

General documentation about ownCrypt

To start the discussion I created a more specific document.

Here is a quick brief of its content :

While section 1 is for the most part a verbose description of what the code does, and not very interesting for someone familiar with the existing code, section 2 and 3 are more "meta" and are an attempt at documenting things to get a better overall understanding of the code structure. NB : Section 1 to 3 are not completely finished but most of the remaining analysis is not critical for the initial integration.

I also created a "state graph" that describes how the discovery and reconcile phase decides what to do with each file. This is the graph (not included here because it too big).

While not very useful for someone who knows this by heart, it is a good reference when one needs (for instance) to know the possible paths for how an item could end up with the XXXX instruction.

Next steps

@ogoffart @danimo @dragotin @jturcotte

Starting point

The first thing I would like to do is reach an agreement with the client team on the proposals I made in the document, so that I can start coding. Further discussion will be needed on other topics after this one, but I believe this is the most structuring code-wise.

Integration tests

I plan to use Jenkins to run proper integration tests. Is there any templates I could use to speed up the initial set up of Jenkins (for instance, to configure the building test) ?

I'm also considering tools such as Sikuli to test more things than the current torture tests. I read about Smashbox but it seems more about testing server-side functionality. Comprehensive integration tests are pretty critical for ownCrypt since it will be implemented in the deepest layer of the client, so the risk of introducing regression is great.

--- Want to back this issue? **[Post a bounty on it!](https://www.bountysource.com/issues/29483452-adding-owncrypt-client-side-encryption?utm_campaign=plugin&utm_content=tracker%2F216457&utm_medium=issues&utm_source=github)** We accept bounties via [Bountysource](https://www.bountysource.com/?utm_campaign=plugin&utm_content=tracker%2F216457&utm_medium=issues&utm_source=github).
menelic commented 8 years ago

@orion1024 looks promising! Looking forward to using this asap - more grease to your elbow, mate for the coding and support from the core and client devs!

jospoortvliet commented 8 years ago

@LukasReschke @karlitschek fyi ;-)

karlitschek commented 8 years ago

Yes. Pretty cool. Looking forward to pull requests ;-)

gcollins commented 8 years ago

This is fantastic! Will it have support for all clients or just PC/Mac? I'd be interested in it for use on Android as well.

orion1024 commented 8 years ago

Ultimately I want to add the feature to the mobile clients too yes. But (and it's a big but) the priority is to get the desktop version right before spending time on the mobile clients. It is also a lot of additional work that needs a different skill set so as long as it's just me working on ownCrypt I really wouldn't expect mobile apps support before 2017.

danimo commented 8 years ago

Scoping this for 2.2 tentatively, so we can have the conversation once 2.1.1 is out. We haven't forgotten about it!

orion1024 commented 8 years ago

@danimo What is the general time frame for 2.1.1 and 2.2 ?

scheppi2610 commented 8 years ago

Hello orion and all others, (@danimo: 2.1.1 Beta is planned for tomorrow - see wiki - release dates)

Thank you for opening this request. Having read only your initial design document V. 0.7 i want to remark some of my thoughts:

Owncloud client.docx

orion1024 commented 8 years ago

Hi Scheppi,

thanks for your interest. I read your document, and first I need to clarify something because I'm not sure of what you want there : ownCrypt will do client-side encryption, but not "local disk encryption" (like EncFS/LUKS/TrueCrypt/BitLocker...). That is to say, the files will be stored unencrypted locally. Why ?

I agree with pretty much everything in your document. The distribution model I aim for is to have the feature baked into the standard client, to leverage ownCloud existing user base. Not only because i want to make something useful and easily available to more people, but also because the more people use it, the more feedback I get to enhance/bugfix it.

The only point where we seem to disagree is sharing. Sharing will be optional in ownCrypt. It's the user's choice whether to share part of its encrypted content to other people or not, or different parts with different people, so what are the reasons not to do it in your opinion ?

If you're interested in contributing, well first know that you're already doing it :) discussing and giving your opinion is a big help. Spreading the word around you about the project is also good.

If you're interested to contribute more (outside of coding), more opportunities will come. Designing an easy-to-use GUI and workflow, for instance, is a big part of ownCrypt, and one that needs as much feedback as possible from end-users. Beta testers will also be very appreciated further down the line.

What exactly do you mean by "hardware and software" ?

One last thing : I work on ownCrypt on my spare time, so the development speed is very much dependent on the amount of time left by my professional/personal life otherwise. I'm not sure when you need this feature, but ownCrypt won't be ready for production before several months. Just don't get your hopes too high if you need this feature quickly :)

scheppi2610 commented 8 years ago

Hi orion,

i agree with all what you say. Local encryption is not necessary at least for first version. Adding hard- or software now seems to me like a thumb idea as the whole development environment fits into a virtual box. Maybe i can contribute by asking a programmer to support you with code / GUI work. by the way, would it make sense not to update all the different files for design documentation but to create a wiki page for discussion and documentation?

scheppi2610 commented 8 years ago

Just to check that we have thought about it: What do you folks think about integrating a full-featured encryption product instead of programming all along newly? As i am not a real programmer i found this (https://veracrypt.codeplex.com/wikipage?title=Command%20Line%20Usage) for veracrypt. Unfortunately i did not find any hint to an API or something similar. But integration should be possible anyway, shouldn't it?

romainvieme commented 8 years ago

Hi orion ! I am very interested by this project. I wonder if it would be possible to let the user manage some of the keys : instead of having one key per file, you let the user maintain a set of custom keys (and insert a reference in the file metadata). This way you can encrypt a bunch of files with the same key. I see three advantages in letting the user do that:

Of course there are also some drawbacks e.g. :

Please let me know if you're interested, I can describe the more precisely ! I think it could be built on top of the app. Though I'm not a professional developer, not sure I could really be too helpful...

karlitschek commented 8 years ago

@romainvieme Pull request welcome.

orion1024 commented 8 years ago

@scheppi2610

No existing product, to my knowledge, can fit the requirements. We need something :

Overall that's too specific I think...

Also, I just saw your suggestion regarding a wiki page, or something to handle suggestions in a better way, but I will give it some thought.

@romainvieme

Thanks for your interest. I'm not so sure about your suggestion regarding key management though.

scheppi2610 commented 8 years ago

orion, you are right. syncing the container does not make sense. I could imagine that it makes sense if we integrate the syncing algorithm into the encryption module. But therefore you would need to re-implement the encryption process.

romainvieme commented 8 years ago

Thanks for your response. What I suggested is to let some users (the ones who choose it) manage their own keyring. But you're right in that it would probably break the ease of use for everyone. For the record, I'd like to explain what I thought (maybe you'll find it useful somehow).

As for the two other points, I totally aggree with you, all the more because I'm not the one writing the code ;) and it's maybe too early to talk about possible improvements !

orion1024 commented 8 years ago

Regarding entropy : yes, the issue you mentioned with first upload is something I identified, but didn't fully thought out yet. I need to get myself familiar with how entropy works and affect key strength. The (still quite vague) idea I have in my mind is to use the device entropy and some kind of user input (like : moving the mouse) to seed a CSPRNG. Hopefully the resulting entropy will be high enough for the CSPRNG to generate strong keys, but I need to think it through.

For hidden files : I misunderstood your 1st feedback actually. You are right, letting the user manage its keys would remove the need for hidden files for each device. But, it would kill usability IMHO. Think about someone wanting to share files with 20 people, it would be a nightmare.

Le 27 janv. 2016 à 13:31, romainvieme notifications@github.com a écrit :

Thanks for your response. What I suggested is to let some users (the ones who choose it) manage their own keyring. But you're right in that it would probably break the ease of use for everyone. For the record, I'd like to explain what I thought (maybe you'll find it useful somehow).

The issue with entropy presents itself mainly on the first upload of encrypted data. Probably some users at least will upload at once a lot of files : for instance, suppose I want to upload my photos on an owncloud server I don't trust. That could be ~ 10^4 files to be uploaded at once, i.e. 10^4 keys to generate. I do that on my laptop which has typically ~500-1000 bits of entropy available (browsing with HTTPS already eats some entropy). If lack of entropy is non-blocking, there could be some weak keys aming the whole. If not, the process could take longer. Anyway, that's something to think about I read the crypto design (maybe too quickly) and I was under the impression that there was 1 hidden file for each device / user, encrypted by MPuK (§ 4.1.8.2). Are those stored on the client ? As for the two other points, I totally aggree with you, all the more because I'm not the one writing the code ;) and it's maybe too early to talk about possible improvements !

— Reply to this email directly or view it on GitHub.

menelic commented 8 years ago

@orion1024 Hi, have you had a look at https://github.com/duplicati/duplicati ? tagline: "Store securely encrypted backups on cloud storage services". It does scheduled encrypted backups to owncloud, offers restore etc. Seems like the implementaion adressses most of the above?

orion1024 commented 8 years ago

Hi menelic,

Duplicati is interesting, but the purpose is not the same : duplicati is a backup solution, which protects data against loss or corruption, while ownCrypt aims to protect it against server-side spying, while still allowing the user to work, sync and share these protected files as usual .

Le 4 févr. 2016 à 00:19, menelic notifications@github.com a écrit :

@orion1024 Hi, have you had a look at https://github.com/duplicati/duplicati ? tagline: "Store securely encrypted backups on cloud storage services". It does scheduled encrypted backups to owncloud, offers restore etc. Seems like the implementaion adressses most of the above?

— Reply to this email directly or view it on GitHub.

ghost commented 8 years ago

Hi, I am not fully aware of what is going on but IIRC the Mega.nz sync client does exactly what is intended here. The point is that sync client source code is now available and may be useful for you:

https://github.com/meganz/MEGAsync

orion1024 commented 8 years ago

Thanks for the information. Coding has begun but it's definitely worth checking how Mega is doing it.

ghost commented 8 years ago

Every time I saw something about end-to-end encryption on ownCloud there are a very valid concern about the breakage of the web interface. I do not know if there is anything already implemented or planned to that issue but I took a look (I do not code) on what may be useful and I find out ProtonMail is using openpgpjs there:

https://github.com/openpgpjs/openpgpjs https://github.com/ProtonMail/WebClient

orion1024 commented 8 years ago

As of now, there is no plan to implement client-side encryption for the Web interface. This is intended : from what I gathered there is currently no secure way of running client-side code in a browser, which is a very insecure environment to run code by design. Now, I know that this is a fast evolving field and it will probably change in the future. And sooner than later, since there are a lot of incentives for it to happen.

Meanwhile, what does it mean for ownCrypt ? as you guessed it means that the encrypted files won't be usable from the Web interface. But ownCrypt is designed to be a feature that you enable selectively, for your most critical files, not something you enable for everything (you could do that, but you don't have to).

e-alfred commented 8 years ago

Will this make it still into 2.2? Until then, there is another open source software to encrypt files client-side in a cloud-friendly (service-agnostic) way:

https://github.com/cryptomator/cryptomator

danimo commented 8 years ago

No, 2.2 is closed.

orion1024 commented 8 years ago

Given the current pace, I wouldn't expect anything before several months. I'd like to at least have a beta version for this summer, but this is highly dependent on how much spare time "real-life" leaves me to work on this.

dumblob commented 8 years ago

@orion1024 thanks a lot for your initiative! I read quickly through your proposal and reactions to suggestions of others and from the position of an IT Project Manager responsible for a new enterprise secure data storage, I must agree with every single decision you made.

One think I'm curious about and I didn't see it anywhere is the question of deployment of ownCrypt over existing deployments of ownCloud client (e.g. to avoid refetching all the data). Will ownCrypt allow easy switching of the client-side encryption on and off (including an option to encrypt all current data)?

orion1024 commented 8 years ago

Thanks for your support !

You will be able to switch CSE on and off flexibly yes, that's the goal. Switching it on or off means at least once re-uploading the data to the server obviously. But I would also like to avoid any fetching from other clients after that. This adds extra complexity to the sync algorithm though, and I don't have a full grasp yet on how to do it properly, so I can't guarantee this feature will make it.

ghost commented 8 years ago

Hi, this is a very good initiative! Would you be able to spend more time on the development if you received financial support?

Best regards!

orion1024 commented 8 years ago

Hi @splashote ,

Thanks for your support and the proposal. I'm afraid it wouldn't help, I have a full-time job which I don't plan to quit, so... I hope to get some contributors interested once the project is more fleshed out though.

If you're willing to help there are other ways, even if you're not a contributor yourself : just spreading the word about the project around you, to people that you think might be interested is a big help, in and of itself.

dumblob commented 8 years ago

@orion1024 we could send a "Thank you" letter/certificate for you to your company, so that they are aware of your contributions and your value.

orion1024 commented 8 years ago

@dumblob Thank you that's nice, although since my company does not employ me as a developer, they would not care much I think. Not that they don't value my work, just that that it's not in the same field ; I'm closer to what you do yourself. In any case, as long as ownCrypt isn't delivered, usable and stable there is little to thank me for :)

@splashote actually I thought about some ideas where some money might help. Later on in the project it would be good to either have a bug bounty or some cryptographers auditing ownCrypt. It's definitely not the time yet for that, but I'll keep thinking about it.

ghost commented 8 years ago

fine!

i have a server running owncloud based on the principles of economic solidarity. every Euro not spent for the server is donated in order to support open source software. I'd love to support owncloud with this money.

On Sat, May 7, 2016 at 12:28 AM, Orion1024 notifications@github.com wrote:

@dumblob https://github.com/dumblob Thank you that's nice, although since my company does not employ me as a developer, they would not care much I think. Not that they don't value my work, just that that it's not in the same field ; I'm closer to what you do yourself. In any case, as long as ownCrypt isn't delivered, usable and stable there is little to thank me for :)

@splashote https://github.com/splashote actually I thought about some ideas where some money might help. Later on in the project it would be good to either have a bug bounty or some cryptographers auditing ownCrypt. It's definitely not the time yet for that, but I'll keep thinking about it.

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub https://github.com/owncloud/client/issues/4327#issuecomment-217578843

orion1024 commented 8 years ago

@splashote : I won't have any use of it for now, but if you're willing to help ownCloud I'm sure there are some ways you can spend it the regular ownCloud development. I'm not familiar with what is possible though, so you might want to ask someone more experienced.

orion1024 commented 8 years ago

This is a quick brief on the status of integrating ownCrypt in ownCloud. As usual, feedback is welcome.

1. Preliminary work

Goals

Status

This is done through several PR to the master branch, several of which are ongoing :

Status : PR #4724 submitted, ready to review

Status : PR #4816 submitted for feedback, 80% finished

Status : not started, requires the previous PR

2. Add code that will actually introduces changes (and thus, risks)

Goals

Status

Status : ongoing

Status : not started, requires preliminary work to be done

On @jturcotte advice, those changes would probably be pushed first to a feature branch, not directly to master.

ash663 commented 8 years ago

@orion1024 I am a CS undergrad. I just started working on ownCloud as part of my internship. I would love to help you and contribute to ownCrypt in anyway possible. I do not know where to begin, could you help me and point me in the right directions?

orion1024 commented 8 years ago

@ash663 thanks for your proposal ! Are you planning to contribute as a part of your internship ?

As a start you should read the documents linked in the top posts of this issue. I would start with "design, intents, etc.", then the crypto design graphs, then the crypto design doc (but this one is quite long, and not especially easy to read). This will give you a rough idea of what needs to be done, and help you in deciding how and where you can contribute. It can be code, UX design, ideas, remarks...

We can discuss this on IRC if you want.

mlmedia commented 8 years ago

OwnCrypt sounds like something I'd definitely be interested in as well and can lend a hand if possible.

Seafile does client-side encryption for their cloud file sharing application, so perhaps some of the design for this can be borrowed from that application? So we could also avoid the same issues they had - for example with the web interface (which would cause problems for the strict client-to-client transaction).

https://github.com/haiwen/seafile/issues/824

e-alfred commented 8 years ago

Wouldn't it be possible to do the decryption on the client side in a web browser as well? Mega does this for example: https://github.com/meganz/webclient

mlmedia commented 8 years ago

RE: Mega NZ web browser encryption -

Where is the decryption key stored? How is it passed from the client (desktop) application to the web browser?

If the (private) decryption key is passed from the server to the client (and then JS performs the necessary decryption), then you run into the same issues giving a server administrator (or hacker) the ability to decrypt files with server access to the decryption key.

My understanding of how the client-side end-to-end encryption would work would be similar to OpenSSL public private key encryption. The public keys would be managed by the server application (similar to the way you add your public keys to Github) and the private keys allowing the decryption would always remain on the client side and NEVER on the server.

I'm new to this discussion however so I don't want to speak out of turn. It may be that the intent is to use something like symmetric key crypto where the same encryption and decryption keys are passed (via phone / email / carrier pigeon) between users.

EDIT: Interesting - it appears that Seafile uses symmetric key crypto using a master key which is encrypted, passed between users, and decrypted in order to decrypt the encrypted file(s). From what I can tell the password used to encrypt and decrypt the master key is stored in the client only and never on the server.

See here: http://manual.seafile.com/security/security_features.html

orion1024 commented 8 years ago

@e-alfred

Wouldn't it be possible to do the decryption on the client side in a web browser as well? Mega does this for example: https://github.com/meganz/webclient

It would be possible technically, but it is not planned for ownCrypt for several reasons :

  1. you break the threat model : even though the decryption code runs in your web browser, not on the server, the code itself is provided by the server. So you implicitly trust the server with providing you non-malicious code. Trusting the server is exactly what we don't want to do when using ownCrypt.
  2. even if there was a reliable way to run trustworthy browser code to decrypt the file, web browsers are very insecure platforms to run critical code ; they were not designed to do that.
  3. finally, implementing access through web-browser wouldn't be such a big gain for ownCloud users. While you could certainly access your files, since the decryption is done on the client, any app that runs on the server wouldn't be able to leverage the decrypted file (since it runs on the server). That means you still can't use the gallery app, calendar app, etc. on your encrypted files.

Point 1 & 2 might be resolved at some point in the future ; there are definitely steps in the right directions ; but it's not quite there yet.

orion1024 commented 8 years ago

@mlmedia

ownCrypt will use a combination of symmetric and asymmetric encryption. This is described in detail in the documents posted in the OP, but the gist of it is :

When device A wants to share files with device B (they can belong to different users) :

  1. A & B exchanges their public keys first
  2. A encrypts the file keys with B public key and store it on the server
  3. B can then fetch this and decrypt it with its private key, giving him access to the files keys.

OwnCrypt sounds like something I'd definitely be interested in as well and can lend a hand if possible.

Sounds great, how do you think you could contribute ?

mlmedia commented 8 years ago

@orion1024 I'm LAMP/LEMP full stack + front end. 8+ years.

A cloud files app like this would be fairly new to me, but I'm experienced enough to figure out what's needed.

I use ownCloud daily and could definitely make use of a more secure application for more sensitive files that I would never trust to the current ownCloud app (or Dropbox, Google Drive, etc.). I currently use Syncthing for secure (TLS) syncing of files between 3 of my computers (Mac, PC, and Linux). Something like ownCrypt would be better, which is why I'd be up for helping it along. I have some bandwidth...

orion1024 commented 8 years ago

@mlmedia

The bulk of ownCrypt happens on the client, but ownCrypt usability can be enhanced with some helper features that need some development on the server-side :

For the web interface users convenience, and the overall reliability of ownCrypt :

To make key exchanges easier between users (these are still rough ideas) :

If you're interested on working on any of these, let me know. If you don't have enough bandwidth it's fine as well ; any feedback or clever idea you might have on the existing material, what you would like to see in ownCrypt, etc. is welcome.

ash663 commented 8 years ago

How would you recover files if user's disk gets wiped out due to some accident?

e-alfred commented 8 years ago

There is a new option available to encrypt files on the client side: https://www.cryfs.org/

orion1024 commented 8 years ago

How would you recover files if user's disk gets wiped out due to some accident?

If the user gave access to its files to other devices he owns, or to other users, he can use those to recover them. If only one device accessed the files, ownCrypt will provide an easy way for the user to backup its encryption keys (protected with a password or something similar), in order to be able to restore its files from the encrypted version on the server. If the keys backup get lost, however, the files are permanently lost.

There is a new option available to encrypt files on the client side: https://www.cryfs.org/

Those solutions can't be used if you want to share your encrypted files unfortunately.

Also, a short update : progress has been very slow in the last few months thanks to my dayjob being very busy. The high workload should abate after the summer though, and I'm still set on delivering the feature. The goal is to have a beta out at the end of the year.

scheppi2610 commented 8 years ago

Orion, is there anything particular I can give my programmer to solve ? Is there any code to download and verify?

orion1024 commented 8 years ago

@scheppi2610 nothing ready yet. I'm currently busy designing the code so that ownCrypt can be used as an API by different clients (ownCloud desktop, IOS and Android) ; as a bonus it will also be easier for several developers to work on different sections of the code. I expect the first draft of the API to be ready by mid-september.

vx28643 commented 8 years ago

I can't wait for this feature, thank you for working on it. I would just like to say that I hope you include the choice in the desktop client to exempt specific folders or accounts from the encryption, for cases where encryption is not desired - for sharing purposes and web access to certain files on Owncloud, while encrypting the other files.