search

terl / lazysodium-java

A Java implementation of the Libsodium crypto library. For the lazy dev.
https://github.com/terl/lazysodium-java/wiki
Mozilla Public License 2.0
135 stars 48 forks source link

Raspberry Pi and Lazysodium #45

Closed gurpreet- closed 5 years ago

gurpreet- commented 5 years ago

@gurpreet- Thanks for the feedback. Very much appreciated.

I made an error in my original write-up. I was testing on an Ubuntu 16.04 box, not 18.04 as I said above. Sorry. And I think that explains why my apt-cache search libsodium is reporting libsodium18, which sounds like it's the issue. I assume if I test on a real 18.04 box, I'll not have a problem. Or if I test on a 16.04 box using the bundled libs, I also not have an issue. That's great and explains everything. Thanks!

Just to be clear about what I was doing... The only reason I was testing on Ubuntu with an OS-installed libsodium was because I was trying to test my logic that first tries a bundled libsodium, then falls back to an OS-installed libsodium. In my test code, I purposely caused the bundled libsodium to not be loaded. Under normal circumstances on Ubuntu, the bundled libsodium will be used and all will be good.

But, if I could shift gears for a second... I have a number of users who want to use my app on a Pi3. From what I can tell, the LazySodium jar doesn't include libs for that platform. Is that correct? Therefore, when they install libsodium using apt, they get libsodium18.

What's the solution in that use case? Should they compile libsodium for that platform? Or, would it be possible for you to include that lib in the LazySodium jar?

Originally posted by @mhilbush in https://github.com/terl/lazysodium-java/issues/34#issuecomment-491453864

mhilbush commented 5 years ago

Hi @gurpreet- I see your latest release, 4.0.0, includes the armv7 libsodium libs. Does this mean that you've sorted out the SHA256 and SHA512 hash/auth issues with the armv7 version of libsodium?

gurpreet- commented 5 years ago

Hi @mhilbush,

Yes I think I have solved it :) just need to test it now!

mhilbush commented 5 years ago

That's great! I did a build of my openHAB binding with the 4.0.0 version. And now that LazySodium is released with the armv7 lib, I submitted the PR to have my Doorbird binding included in the openHAB distribution. Thanks for all your help on this. It's very much appreciated.

https://github.com/openhab/openhab2-addons/pull/5857

gurpreet- commented 5 years ago

Did you test it all works before raising the PR? 😃

I only managed to test the main functions before I got busy fighting influenza 😷

mhilbush commented 5 years ago

Did you test it all works before raising the PR?

I tested it on my Linux box. :wink: And, I posted the new version for the current user base to install. No problems reported yet. Besides, it usually takes a several months for PRs to be merged (there's quite a big backlog of contributions), so there's plenty of time to sort anything out. I suppose the backlog is a good thing, as it means openHAB's popularity is quite good.

gurpreet- commented 5 years ago

Hi @mhilbush,

So I tried again to get Libsodium working with Raspberry Pi. As you are aware I recompiled the libsodium.so for the Raspberry Pi. It still fails on the SHA functions.

So as the shared library is correct and the loading of the library is correct, therefore the only thing I see being a problem is the code. I assume one or more of the parameters are incorrect.

As it stands it is difficult to get Libsodium working with the Raspberry Pi (1 to 3) because I am sure Libsodium was never created with the intention to run on such devices.

Thus, after a lot of coffee induced sleepless nights, I am pleased to unofficially announce Hydride that binds against Libhydrogen. Libhydrogen is from the same creator as Libsodium, who has deemed Libydrogen to be the successor of Libsodium. A key point regarding Libhydrogen is that it is aimed at universal compatibility, meaning it's aimed at lower powered devices such as the Raspberry Pi and even more obscure architectures. Libhydrogen is far simpler than Libhydrogen.

I have tested Hydride and it works flawlessly on RPi3, RPi2 and even RPi1. For the moment, it does not contain convenience functions like Lazysodium does. I plan to add those later on. Windows support is lacking too.

You can add Hydride straight into your project using Jitpack, see the readme file in the project.

I hope this is good news to you @mhilbush 😄

mhilbush commented 5 years ago

Absolutely good news. I'll check it out over the next couple days. Thanks!

Do you know when Windows support will be available?

gurpreet- commented 5 years ago

Just added support for Windows 32-bit and 64-bit now!

I hope someone can test it out, I am lacking a Windows machine to test. To test just need to clone the repo and then run gradle test. All tests should pass.

mhilbush commented 5 years ago

I have a Windows box but it doesn't have gradle installed. Can I run ./gradlew test?

mhilbush commented 5 years ago

On the first test, I get an exception during initialization. This is a 64-bit Win10 box.

java.lang.UnsupportedOperationException
    at java.nio.file.Files.setPosixFilePermissions(Files.java:2044)
    at co.libly.hydride.utils.LibraryLoader.setPermissions(LibraryLoader.java:309)
    at co.libly.hydride.utils.LibraryLoader.copyFromJarToTemp(LibraryLoader.java:259)
    at co.libly.hydride.utils.LibraryLoader.loadBundledLibrary(LibraryLoader.java:151)
    at co.libly.hydride.utils.LibraryLoader.loadLibrary(LibraryLoader.java:116)
    at co.libly.hydride.Hydrogen.<init>(Hydrogen.java:30)
    at co.libly.hydride.Hydrogen.<init>(Hydrogen.java:26)
    at co.libly.hydride.BaseTest.doBeforeEverything(BaseTest.java:23)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:675)
    at org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
        snip
mhilbush commented 5 years ago

Another question, @gurpreet-

I'm currently using the following two APIs:

I'm trying to find the equivalent APIs in Hydride.

I think hydro_pwhash_create might map to the first one, although some of the args are different.

But, I can't find the equivalent to cryptoAeadChaCha20Poly1305Decrypt.

gurpreet- commented 5 years ago

Updated to support Windows: https://github.com/libly/hydride-java/releases/tag/1.1.1

Regarding cryptoAeadChaCha20Poly1305Decrypt I can't see a similar method in Libhydrogen. How important is it?

mhilbush commented 5 years ago

Regarding cryptoAeadChaCha20Poly1305Decrypt I can't see a similar method in Libhydrogen.

That's what I was afraid of.

How important is it?

It's THE reason I need libsodium. Events (e.g. motion detected, doorbell pressed) from the Doorbird doorbell are encrypted, and require decryption using that API.

I get a password hash here. https://github.com/mhilbush/openhab2-addons/blob/doorbird-new/bundles/org.openhab.binding.doorbird/src/main/java/org/openhab/binding/doorbird/internal/listener/DoorbirdEvent.java#L203

And then decrypt the event here. https://github.com/mhilbush/openhab2-addons/blob/doorbird-new/bundles/org.openhab.binding.doorbird/src/main/java/org/openhab/binding/doorbird/internal/listener/DoorbirdEvent.java#L225

mhilbush commented 5 years ago

FYI from my x64 Win10 box...

$ ./gradlew test
Starting a Gradle Daemon (subsequent builds will be faster)

> Task :test

co.libly.hydride.HashTest > hashWithKey() PASSED

co.libly.hydride.HashTest > multiPartHash() PASSED

co.libly.hydride.HashTest > hashWithoutKey() PASSED

co.libly.hydride.HelpersTest > zeroArray() PASSED

co.libly.hydride.HelpersTest > hexadecimalTest() PASSED

co.libly.hydride.HelpersTest > arraysAreEqual() PASSED

co.libly.hydride.KdfTest > deriveKeys() PASSED

co.libly.hydride.KeyExchangeKKTest > keyExchange() PASSED

co.libly.hydride.KeyExchangeNTest > keyExchange() PASSED

co.libly.hydride.KeyExchangeXXTest > keyExchange() PASSED

co.libly.hydride.PwHashTest > passwordHashUsingMasterKey() PASSED

co.libly.hydride.PwHashTest > passwordHashUsingDerivedKey() PASSED

co.libly.hydride.PwHashTest > upgradeParameters() PASSED

co.libly.hydride.PwHashTest > passwordHashUsingDeterministicKey() PASSED

co.libly.hydride.PwHashTest > reEncryptUsingNewMasterKey() PASSED

co.libly.hydride.RandomTest > randomisedBufferDeterministic() PASSED

co.libly.hydride.RandomTest > getRandomUnsignedIntWithUpperBound() PASSED

co.libly.hydride.RandomTest > getRandomUnsignedInt() PASSED

co.libly.hydride.RandomTest > randomisedBuffer() PASSED

co.libly.hydride.SecretBoxTest > encrypt() PASSED

co.libly.hydride.SecretBoxTest > encryptWithProbe() PASSED

co.libly.hydride.SignTest > signDeterministic() PASSED

co.libly.hydride.SignTest > sign() PASSED

co.libly.hydride.SignTest > signMultiPart() PASSED

BUILD SUCCESSFUL in 7s
4 actionable tasks: 4 executed
gurpreet- commented 5 years ago

Okay cool so Hydride ran successfully on Windows 😁

Ah okay I understand. I have not dropped support for Lazysodium, so I will tinker further into the issue. Rest assured, I have been learning all about C compilation flags and C debugging so I am better equipped this time around to get it solved.

mhilbush commented 5 years ago

Ah okay I understand. I have not dropped support for Lazysodium, so I will tinker further into the issue.

Just to be clear... Those functions that don't work on the RPi... I don't need any of those. But, I understand your desire to make it fully functional.

Rest assured, I have been learning all about C compilation flags and C debugging so I am better equipped this time around to get it solved.

I'll bet. I wrote a lot of C code a long, long time ago (early 80's) -- back before most of the world knew what C was. LOL It's gotten way more complicated these days with the diversity of architectures.

gurpreet- commented 5 years ago

Just to be clear... Those functions that don't work on the RPi... I don't need any of those. But, I understand your desire to make it fully functional.

I understand this too. If worst comes to worst I can create a LazySodiumRPI object that has just the working functions in. The beauty of abstractions!

I'll bet. I wrote a lot of C code a long, long time ago (early 80's) -- back before most of the world knew what C was. LOL It's gotten way more complicated these days with the diversity of architectures.

Ah you could probably write a C encryption library then! 😁

gurpreet- commented 5 years ago

Good news @mhilbush!

I am 99.99% sure I have got all the functions working now. Not only that, I have compiled the libsodium.so against armv6 so theoretically it should be backwards-compatible all the way back to RPi1!

Try release 4.0.1 and hopefully it will work.

mhilbush commented 5 years ago

That's great!!

I'll give it a try. It may be a while before I can give feedback as I'll be on vacation for a few days.

mhilbush commented 5 years ago

I ran ./gradlew test on several platforms: Linux 18.04 on Intel, Raspbian on a RPi 3, Mac OSX Mojave, and Windows 10.

It was successful on all platforms except Windows 10. Output was this.

$ ./gradlew test
Downloading https://services.gradle.org/distributions/gradle-5.4.1-all.zip
..............................................................................................................................
Unzipping C:\Users\mark\.gradle\wrapper\dists\gradle-5.4.1-all\3221gyojl5jsh0helicew7rwx\gradle-5.4.1-all.zip to C:\Users\mark\.gradle\wrapper\dists\gradle-5.4.1-all\3221gyojl5jsh0helicew7rwx
Starting a Gradle Daemon (subsequent builds will be faster)

> Task :test

com.goterl.lazycode.lazysodium.utils.LibraryLoaderTest > testOneOffLoadingFromJar FAILED
    com.goterl.lazycode.lazysodium.utils.LibraryLoadingException at LibraryLoaderTest.java:53
        Caused by: java.io.FileNotFoundException at LibraryLoaderTest.java:53

com.goterl.lazycode.lazysodium.utils.LibraryLoaderTest > testLoadingSystemWhenAbsent FAILED
    com.goterl.lazycode.lazysodium.utils.LibraryLoadingException at LibraryLoaderTest.java:164
        Caused by: java.io.FileNotFoundException at LibraryLoaderTest.java:164
Failed tests:
[com.goterl.lazycode.lazysodium.utils.LibraryLoaderTest::testOneOffLoadingFromJar]
[com.goterl.lazycode.lazysodium.utils.LibraryLoaderTest::testLoadingSystemWhenAbsent]

76 tests completed, 2 failed, 1 skipped

> Task :test FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':test'.
> There were failing tests. See the report at: file:///C:/Users/mark/lazysodium-java/build/reports/tests/test/index.html

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 59s
4 actionable tasks: 4 executed

A zip of the full build/reports/test directory is here. https://drive.google.com/open?id=1OIBEpb50eC_MC94hTd1piL2vVnfqIzA5

gurpreet- commented 5 years ago

Woop, so it worked on every platform including the RPi3. This is outstanding news!

Digging into why it didn't work on Windows the actual error is: 

C:\Users\mark\AppData\Local\Temp\lazysodium\libsodium.dll
The process cannot access the file because it is being used by another process.

So yes, in actual reality, the libsodium.dll loaded fine and all the functions within it worked absolutely fine. It was just being used by another process. So, as long as two process don't use the same shared libsodium.dll file, it's is not an issue related to this RPi one.

Please see #53 to track the progress of this multiple processes using libsodium.dll issue.

gurpreet- commented 5 years ago

On a further note, it was relatively easy compiling libsodium for armv6 as long as you had an existing RPi (any version). The CFLAGS must include -shared -march=armv6 -mfpu=vfp -mfloat-abi=hard and then it should compile fine. The -march=armv6 is arguably the most important flag.

I think this issue can be closed now? 😁

mhilbush commented 5 years ago

I think this issue can be closed now? 😁

Absolutely! Hurray!

Thanks for making this happen!!

gurpreet- commented 5 years ago

No problem. It took quite a long time to resolve this issue, but I'm glad I went through this whole process. It's made me much better at debugging and I've learnt quite a bit about shared libraries and their compilation.

Have fun!