Adding ownCrypt : code reuse across clients, and best way to do it

[orion1024]

I'm working on the structure of ownCrypt code, which aims to add client-side encryption.

I'm currently pondering how to best design the code so that it can be easily reused across all clients : desktop, Android and iOS.

I came up with some ideas, explained below, but I have hit a point where I think some insight from the teams would be very valuable. This time not only the desktop team, but also (and especially) from the mobile teams as well. @owncloud/android-developers @owncloud/ios-developers @owncloud/desktop-developers

This is how I approached the topic. Sorry if it's a bit lengthy, but i think a broader picture is needed to understand where I come from.

1. Need to reuse code => separate client-specific code from pure ownCrypt code
2. for pure ownCrypt code : build a library that can be reused across all clients. According to this and this, the best option seems to be a C++ library, which is convenient since the desktop code already uses that language. I would love to hear the mobile team opinions on these articles ; if they could confirm this is something feasible for the clients as they are today.
3. for client-specific code : it consists of 2 things as far as the sync logic is concerned (also, here is a graph that explains how it would play out for the desktop client)

A)

Classes that calls ownCrypt public interfaces. For instance if I want to get a file : ownCryptManager.getFILE("/foo/bar.pdf"); . This call is supposed to handle all the decryption transparently, and returns to the sync client the plain-text file. Internally, ownCrypt will of course need to do more than just a GET : fetching the encryption key at the very least. Below a very simplified version of what needs to happen :

GETFILE("/foo/bar.pdf");    // download encrypted file
GETFILE("/.encryption-keys");    // download encrypted keys
mykey = DECRYPT_KEY(".encryption-keys", "/foo/bar.pdf");     // get the encryption key for foo/bar
DECRYPT_FILE("/foo/bar.pdf", mykey)     // decrypt the file

This logic needs to be in the library : having to reproduce it across all 3 clients would be a nightmare. BUT the first 2 GETFILE calls are client-specific. So I would need B to maintain client-specific and ownCrypt code properly separated.

B)

Feeding the ownCrypt library with server primitives. In the example above, the GETFILE method should be provided by the sync client. The easiest way to do that is to declare in the library an AbstractServerInterface class, that will specify all the methods that ownCrypt needs to do its work. Such as : GETFILE, PUTFILE, LSDIR, MKDIR, DELETE, and so on. The library will use only the Abstract class. So GETFILE would actually be AbstractServerInterface::GETFILE in the example above.

And then each client can implement this interface in the way they see fit.

4- Last step, and this is where I'm stuck at the moment : whether to use a synchronous or asynchronous flow. This question is relevant for calls to ownCrypt library (ownCryptManager.getFILE("/foo/bar.pdf");) and for ownCrypt calls to the server interface (AbstractServerInterface::GETFILE)

Desktop sync client is asynchronous (except for csync code) IOS sync client seems asynchronous (not sure, I don't know enough about IOS platform) Android client seems synchronous (same, I'm not sure)

I'm currently leaning toward synchronous flow : it would make the library easier to code and maintain. I know I can make it fit in the desktop client asynchronous flow (this is already done with csync). But I don't know how easy it is, or even if it's possible for the mobile clients ?

Asynchronous flow seems harder to implement, especially if we need additional wrapper classes for mobile clients. Using Qt across all clients could make that easier, but how good is Qt support on mobile platforms, and is that a good idea to begin with ?

--- Want to back this issue? **[Post a bounty on it!](https://www.bountysource.com/issues/34256774-adding-owncrypt-code-reuse-across-clients-and-best-way-to-do-it?utm_campaign=plugin&utm_content=tracker%2F216457&utm_medium=issues&utm_source=github)** We accept bounties via [Bountysource](https://www.bountysource.com/?utm_campaign=plugin&utm_content=tracker%2F216457&utm_medium=issues&utm_source=github).

[davivel]

Having a look. Will tell you something soon about Android. Tricky issue here.

[rullzer]

So I know it will be more work. But I think having native platform libraries is actually a better approach. This forces clear separation between the specification and the implementation.

Also while in theory write once and run on all sounds nice. In practice this will be a world of pain and and you'll end up with a lot of platform specific stuff anyways.

[davivel]

I think having native platform libraries is actually a better approach.

In practice this will be a world of pain and and you'll end up with a lot of platform specific stuff anyways.

Man, you kill your own arguments, leave some fun for the discussion :P

[davivel]

Sorry, forget that, I totally misread your comment. Need coffee.

[orion1024]

In practice this will be a world of pain and and you'll end up with a lot of platform specific stuff anyways.

@rullzer So you think that it will be impossible to properly delegate all platform-specific stuff to code outside the library ? I'm fairly confident I can do it for the desktop client with the proposed design, but I don't know for the mobile clients.

[rullzer]

I'm not sure about the current state. but the last time I tried to crosscompile a 'gernric' c++ lib for andoid it was no fun...

[jturcotte]

If we ever need to share something with mobile clients, I wouldn't restrict this to the encryption part, everything needed for syncing and authentication could be available as a library, leaving only platform features (network access, notifications, etc.) and UI to be platform-specific.

I think it's possible to build a C++ library to be used on Android and iOS. It's not easy, but I think that it would still be easier than having 3 implementations of syncing and encryption. The real problem is however that this should have been done long ago, and at this point you'd need at least 1 man-year to refactor everything and ship it on desktop, Android and iOS. So practically it might not make sense for you to worry about this too much yet unless you have a spare man-year to spare for us :)

[jturcotte]

Using Qt on all mobile platforms is also a big part of the problem, it's complicated to build and it increases the download size considerably even if we just use QtCore. Ideally we'd make this a pure C++ library, but this prevents us from using signal/slots and would make this code look foreign in the rest.

If you are really able to isolate your encryption logic in a library and want to use it on mobile clients, I'd try to avoid relying on Qt.

However with any of those options you're adding constraints to an already very complicated problem, it's your project but personally I'd try to avoid tying myself to the floor until I've got a simple prototype working on one single platform.

[davivel]

@orion1024,

So you think that it will be impossible to properly delegate all platform-specific stuff to code outside the library ?

I'd wonder if the effort worths it better than with other alternatives, and if we have the proper people to do it.

I'm fairly confident I can do it for the desktop client with the proposed design, but I don't know for the mobile clients.

I'm a mobile developer, and I don't know either. Not a good indicator (or not a good mobile developer, of course)

@jturcotte,

It's not easy, but I think that it would still be easier than having 3 implementations of syncing and encryption.

What's your basis to come to that conclusion? Even if we would have started this way from the very beginning. I'm honestly curious :)

[jturcotte]

@davivel Qt's main value is code re-use and portability, we've seen the cost vs. benefit. Instead of having 3 implementations and 3 teams/communities we would only have one, using a sync library that we would have been forced to design and test properly, while right now the technical weight of each implementation accumulates on each side.

My sentence is probably too assertive, I can't really know if it would have been a better idea. But it's something I wish we had tried at least.

[davivel]

Thanks, I understand.

My main concern is, have we seen that cost vs benefit of QT in Android and iOS too? The three platforms where the desktop client work have a long history of C/C++ compilers, tool-chains and frameworks empowering developers. That's a great source of confidence for a first blind decision.

Could we expect that QT works so smoothly in mobile platforms that were conceived, from its very first inception, to constrain the options available for developers? I think there is an important level of uncertainty there.

But it's something I wish we had tried at least.

I totally respect that. Moreover, I couldn't understand that you wished in other way. From my own POV, sometimes I wonder how the things would be if we had tried from the very beginning to develop the desktop client in Java, that self-states it's main value in code re-use and portability too.

P.S: Don't worry, will not make a proposal in that sense :)

[jturcotte]

I have the same the same doubts regarding the use of Qt on mobile, I wouldn't write the UI of all platforms in Qt, and I would evaluate the work needed to avoid using Qt at all for common code if possible to make it easier to build and deploy. At least not for ownCloud, for other types of applications I think it does make sense.

I was just using Qt itself as an example of complex software that shows the value of code reuse across platforms. Even if the cost seems high at the beginning, as the amount of features to share between platforms increases, you're getting more and more gain for the pain.

[davivel]

I was just using Qt itself as an example (...)

Oh, I see. Sorry for the misunderstanding.

Even if the cost seems high at the beginning, as the amount of features to share between platforms increases, you're getting more and more gain for the pain.

You are right, and I don't want it looks like I happily discard the option without a second thought.

I'll try to book some time to study better the options to integrate C++ code in our app.

Besides, maybe someone else in @owncloud/android-developers has some experience with the Android NDK that wants to share here.

[orion1024]

Strange, I thought I submitted a comment 2 weeks ago, but it's not there so I probably messed up.

ok, so thanks @jturcotte @davivel @rullzer for the feedback. From what I gather ; isolating client-specific code is probably harder, but worth it long-term. How much harder, I do not know yet, but I will try and see how it goes. That implies avoiding as much dependencies as possible, Qt first among them.

I do need some libraries though ; and I was thinking of the following :

• SQLite for storing metadata
• Crypto++ for the crypto primitives

Does anyone here knows whether either one could mean trouble on the mobile clients ?

Finally : we discussed Android a lot, IOS not so much. Does anyone from @owncloud/ios-developers has an opinion on this topic ?

[davivel]

SQLite is included in the Android framework... in Java, of course. I don't know if it's accessible directly in native code (assuming that framework wraps it) or if there would be any problem in have a separate binary shipped in the app.