Raspberry Pi and Lazysodium

[gurpreet-]

@gurpreet- Thanks for the feedback. Very much appreciated.

I made an error in my original write-up. I was testing on an Ubuntu 16.04 box, not 18.04 as I said above. Sorry. And I think that explains why my apt-cache search libsodium is reporting libsodium18, which sounds like it's the issue. I assume if I test on a real 18.04 box, I'll not have a problem. Or if I test on a 16.04 box using the bundled libs, I also not have an issue. That's great and explains everything. Thanks!

Just to be clear about what I was doing... The only reason I was testing on Ubuntu with an OS-installed libsodium was because I was trying to test my logic that first tries a bundled libsodium, then falls back to an OS-installed libsodium. In my test code, I purposely caused the bundled libsodium to not be loaded. Under normal circumstances on Ubuntu, the bundled libsodium will be used and all will be good.

But, if I could shift gears for a second... I have a number of users who want to use my app on a Pi3. From what I can tell, the LazySodium jar doesn't include libs for that platform. Is that correct? Therefore, when they install libsodium using apt, they get libsodium18.

What's the solution in that use case? Should they compile libsodium for that platform? Or, would it be possible for you to include that lib in the LazySodium jar?

---

Originally posted by @mhilbush in https://github.com/terl/lazysodium-java/issues/34#issuecomment-491453864

[gurpreet-]

Hi @mhilbush,

Sure I can add the dynamic library for RaspberryPi into lazysodium.

I am not sure what users of RPi3 would get if they performed a apt install libsodium. Am I right in thinking that RPi3 users can upgrade to Ubuntu Mate 18.04? That would give us some leads into what version of libsodium they can run/obtain, we may be able to skip compilation of new shared libraries altogether if they can run libsodium23.

In the meantime, if you can get me a list of RPi versions you want lazysodium to run on, then I can compile their respective shared libraries and bundle them in.

[mhilbush]

Thanks @gurpreet-

I did a little research using my RPi3 Model B. I had one running Raspian Jessie, which I just upgraded to Stretch. Here's the info for each. Neither one reports libsodium23, unfortunately.

Jessie Linux pi 4.9.35-v7+ #1014 SMP Fri Jun 30 14:47:43 BST 2017 armv7l GNU/Linux

$ apt-cache search libsodium libsodium-dbg - Network communication, cryptography and signaturing library - debug symbols libsodium-dev - Network communication, cryptography and signaturing library - headers libsodium13 - Network communication, cryptography and signaturing library

Stretch Linux pi 4.19.42-v7+ #1219 SMP Tue May 14 21:20:58 BST 2019 armv7l GNU/Linux

mark@pi:~ $ apt-cache search libsodium libsodium-dbg - Network communication, cryptography and signaturing library - debug symbols libsodium-dev - Network communication, cryptography and signaturing library - headers libsodium18 - Network communication, cryptography and signaturing library

In the meantime, if you can get me a list of RPi versions you want lazysodium to run on, then I can compile their respective shared libraries and bundle them in.

Do you mean hardware versions, such as 2B and 3B & 3B+? If so, those versions running Raspian are the ones that I think are most widely used by the openHAB user base. Obviously, the 2 is more representative of the current installed base, as I expect most new installs are going on the 3.

[gurpreet-]

Okay so I quickly setup a VM and created a static library (libsodium.a) and could not compile a shared library (libsodium.so).

I'm not good with compiling stuff anyway and can never remember the right flags! If you can compile a libsodium.so for the armv6 architecture, then I'll add it. Otherwise it's going to take me some time to see how it's properly done.

[mhilbush]

I'm not good with compiling stuff anyway and can never remember the right flags!

Me either. 😄 But I I have a compiled version of libsodium-1.0.17 that I built on my RPi 3 using ./configure; make && make check. Does this look right?

$ file libsodium-1.0.17/src/libsodium/.libs/libsodium.so.23.2.0
libsodium-1.0.17/src/libsodium/.libs/libsodium.so.23.2.0: ELF 32-bit LSB shared object, ARM, EABI5 version 1 (SYSV), dynamically linked, BuildID[sha1]=79aa10ab50804aa0f4aa3e2c622ee71a6aa4d888, not stripped

$ ls -l libsodium-1.0.17/src/libsodium/.libs/libsodium.so.23.2.0
-rwxr-xr-x 1 mark mark 1213860 May 19 17:26 libsodium-1.0.17/src/libsodium/.libs/libsodium.so.23.2.0

Here's a link to the .so https://drive.google.com/open?id=1SSYbXSrUSr7rcdYfDyL8bOWOtJH1znOx

[mhilbush]

@gurpreet- BTW, do you know if that .so also works on an RPi 2B (i.e. are those two models binary compatible). I should know this, but I don't do much development at all on a RPi. 🙄

[gurpreet-]

Your .so file is correct, thank you.

Unfortunately, they are not compatible as they have different architectures. I think that the RPi 3 has an armv8 architecture and the RPi 2 has an armv7 architecture. If you compile for armv7 then it should be backwards-compatible with armv8.

[mhilbush]

Well, I'm not set up to cross compile for the RPi2, and my experience cross-compiling is weak, at best. 😦

I'll either try to pick up a RPi2 Model B, or find someone with an RPi2 who can compile libsodium for me.

[mhilbush]

find someone with an RPi2 who can compile libsodium for me

I found another openHAB developer who'll build a RPi2 version for me. Stay tuned.

[mhilbush]

@gurpreet- Here's a version built for the RPi2

https://drive.google.com/open?id=13VTr6gGeNaS9s_L9-Tn2-f1ynPqF2mJA

[gurpreet-]

Hi @mhilbush,

I'm pleased to say that I've added it to lazysodium. I just need to see if it works for Raspberry Pi 2+ devices.

Could you please check to see if it works? On a Raspberry Pi 2 or Raspberry Pi 3, I would git clone or download Lazysodium's master branch. Then in the top level folder, simply run gradle clean test.

Fingers crossed 🤞

[mhilbush]

That's awesome. Thanks!

I can test pretty quickly on a RPi 3. It may take some time for a RPi 2 because I need to find someone with a Doorbird doorbell running openHAB on a RPi 2 AND who wants to be a guinea pig.

[gurpreet-]

A Raspberry Pi 3 is fine 👍

[mhilbush]

What do you make of this? I've not used gradle before (everything openHAB is maven), so I just installed gradle on my Ubuntu 18.04 box.

FAILURE: Build failed with an exception.

* Where:
Build file '/home/mark/lazysodium-java/build.gradle' line: 55

* What went wrong:
A problem occurred evaluating root project 'lazysodium-java'.
> Could not find method useGpgCmd() for arguments [] on object of type org.gradle.plugins.signing.SigningExtension.

[mhilbush]

Ok, I commented out lines 55-56 in gradle.build, and it made it through the clean successfully. Figured I don't need to sign for now.

Now stuck on gradle test.

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':compileJava'.
> Could not find tools.jar. Please check that /usr/lib/jvm/java-8-openjdk-amd64 contains a valid JDK installation.

[gurpreet-]

Oh are you running it on a Ubuntu machine? In order to see if it actually works on a Raspberry Pi Lazysodium needs to be ran on it directly. So download Lazysodium on the Raspberry Pi device itself (you can use git clone https://github.com/terl/lazysodium-java.git or wget https://github.com/terl/lazysodium-java/archive/master.zip for this) and then run it using gradle clean test.

The Raspberry PI needs an installation of the Java JDK for Lazysodium to work. You might be able to get this using apt. I recommend Java 8. However, I think that RPi3 already includes a Java 8 installation? Not sure on this.

[mhilbush]

Yeah, I thought I would first try to build it on my Ubuntu box since that's where I do a bunch of other development and I figured it would be more straightforward getting it to build there first. If that was successful, I was going to switch over to the Pi.

Given the above issue, I think I'll skip Ubuntu and just try to build it on the Pi.

[gurpreet-]

Oh I see :)

Yes, try running the above directly on the RPi and then we can both be sure that it actually works on the RPi. I don't think that the build times would be too long as Java does everything for you through the gradle clean test commands.

Another reason why I don't think it would be wise to build on Ubuntu would be that Lazysodium detects the platform using Java's Platform classes, then finds the right libsodium.so file using your platform. If you run it on Ubuntu it would report the platform as being Linux and load linux/libsodium.so and so if you run it on a RPi it would report as being Arm and load arm/libsodium.so. If you then transfer the built files across to the RPi it might have cached linux/libsodium.so and not arm/libsodium.so and thus when you try to run it on the RPi it would confuse us both and cause us to pull out our hair in frustration 😂

[mhilbush]

Ugh. Installing gradle on the Pi gets me version 3.2.1, which I think is too old to build lazysodium? Need to see how to get something newer. What gradle version are you running?

[gurpreet-]

You can try running the ./gradlew file provided with Lazysodium. Hope that helps?

[gurpreet-]

So to run the test suite you can do ./gradlew clean test. Just use ./gradlew anywhere you see gradle.

[mhilbush]

Ah, thanks! Trying that now...

[mhilbush]

mark@pi:~/lazysodium-java $ ./gradlew clean test
> Task :clean UP-TO-DATE
> Task :compileJava
> Task :processResources
> Task :classes
> Task :compileTestJava
> Task :processTestResources NO-SOURCE
> Task :testClasses

> Task :test

SecretBoxAndroidTest > encrypt FAILED
    java.lang.UnsatisfiedLinkError

AuthTest > initializationError FAILED
    java.lang.NoClassDefFoundError

AEADAndroidTest > initializationError FAILED
    java.lang.NoClassDefFoundError

HashAndroidTest > initializationError FAILED
    java.lang.NoClassDefFoundError

AuthAndroidTest > initializationError FAILED
    java.lang.NoClassDefFoundError

SecretStreamAndroidTest > initializationError FAILED
    java.lang.NoClassDefFoundError

DiffieHellmanAndroidTest > initializationError FAILED
    java.lang.NoClassDefFoundError

SignAndroidTest > initializationError FAILED
    java.lang.NoClassDefFoundError

GenericHashAndroidTest > initializationError FAILED
    java.lang.NoClassDefFoundError

HelperAndroidTest > initializationError FAILED
    java.lang.NoClassDefFoundError

KeyDerivationAndroidTest > initializationError FAILED
    java.lang.NoClassDefFoundError

BoxAndroidTest > initializationError FAILED
    java.lang.NoClassDefFoundError

StreamJavaTest > initializationError FAILED
    java.lang.NoClassDefFoundError

PwHashAndroidTest > initializationError FAILED
    java.lang.NoClassDefFoundError

StreamAndroidTest > initializationError FAILED
    java.lang.NoClassDefFoundError

PwHashTest > initializationError FAILED
    java.lang.NoClassDefFoundError

KeyExchangeAndroidTest > initializationError FAILED
    java.lang.NoClassDefFoundError

ShortHashAndroidTest > initializationError FAILED
    java.lang.NoClassDefFoundError

SecureMemoryAndroidTest > initializationError FAILED
    java.lang.NoClassDefFoundError
Failed tests:
[SecretBoxAndroidTest::encrypt]
[AuthTest::initializationError]
[AEADAndroidTest::initializationError]
[HashAndroidTest::initializationError]
[AuthAndroidTest::initializationError]
[SecretStreamAndroidTest::initializationError]
[DiffieHellmanAndroidTest::initializationError]
[SignAndroidTest::initializationError]
[GenericHashAndroidTest::initializationError]
[HelperAndroidTest::initializationError]
[KeyDerivationAndroidTest::initializationError]
[BoxAndroidTest::initializationError]
[StreamJavaTest::initializationError]
[PwHashAndroidTest::initializationError]
[StreamAndroidTest::initializationError]
[PwHashTest::initializationError]
[KeyExchangeAndroidTest::initializationError]
[ShortHashAndroidTest::initializationError]
[SecureMemoryAndroidTest::initializationError]

19 tests completed, 19 failed

> Task :test FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':test'.
> There were failing tests. See the report at: file:///home/mark/lazysodium-java/build/reports/tests/test/index.html

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 20s
5 actionable tasks: 4 executed, 1 up-to-date

[gurpreet-]

Okay so initial thoughts are that it can't find the libsodium.so file due to the error java.lang.UnsatisfiedLinkError. Or it could also mean that some symbols were not identified in the libsodium.so file.

Either way, let's find out. Could you send the file: /home/mark/lazysodium-java/build/reports/tests/test/index.html to me. That would contain some useful hints as to what's going wrong.

[mhilbush]

Here you go. https://drive.google.com/open?id=1_3Rp9-akyJGpDS7gWNFGvonNjiiRMvb2

[gurpreet-]

Looks like I also need the file: /home/mark/lazysodium-java/build/reports/tests/test/classes/SecretBoxAndroidTest.html.

[mhilbush]

Saw that. I'm zipping up the whole reports directory... Hang on.

[mhilbush]

Zip file of entire reports directory. https://drive.google.com/open?id=1LuiW2e19oRJy8ocvrqSzEHUmx-IolnCd

[gurpreet-]

Thank you for the files. So the error in reality is:

/lib/arm-linux-gnueabihf/libc.so.6: version `GLIBC_2.25' not found (required by /tmp/libsodium/libsodium.so)

So we can try a few things:

1. Please could you verify that the file /tmp/libsodium/libsodium.so is actually there?

2. If it is, then update libc.

   sudo apt-get update
   sudo apt-get install libc6

[mhilbush]

verify that the file /tmp/libsodium/libsodium.so is actually there?

It is not there.

update libc This is the currently installed version.

ii  libc6:armhf                           2.24-11+deb9u4                            armhf        GNU C Library: Shared libraries

I'm on Raspbian Stretch, which is the most current distro. Unfortunately, it cannot be updated. After running sudo apt update.

$ sudo apt install libc6
Reading package lists... Done
Building dependency tree       
Reading state information... Done
libc6 is already the newest version.
0 upgraded, 0 newly installed, 0 to remove and 184 not upgraded.

[gurpreet-]

I have just updated Lazysodium with a different libsodium.so file which may work. Please re-download Lazysodium from the master branch and test again.

If this fails, I'll get a Raspberry Pi and compile it 😄

[mhilbush]

Ugh. Haven't seen one of these in a while...

mark@pi:~/lazysodium-java $ ./gradlew clean test
> Task :clean UP-TO-DATE
> Task :compileJava
> Task :processResources
> Task :classes
> Task :compileTestJava
> Task :processTestResources NO-SOURCE
> Task :testClasses

> Task :test
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x50b5c0d0, pid=21439, tid=1388311664
#
# JRE version: Java(TM) SE Runtime Environment (8.0_65-b17) (build 1.8.0_65-b17)
# Java VM: Java HotSpot(TM) Client VM (25.65-b01 mixed mode linux-arm )
# Problematic frame:
# C  [libsodium.so+0xb0d0]  crypto_auth_hmacsha512_init+0x70
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /home/mark/lazysodium-java/hs_err_pid21439.log
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#

> Task :test FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':test'.
> Process 'Gradle Test Executor 2' finished with non-zero exit value 134
  This problem might be caused by incorrect test process configuration.
  Please refer to the test execution section in the User Manual at https://docs.gradle.org/5.4.1/userguide/java_testing.html#sec:test_execution

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 9s
5 actionable tasks: 4 executed, 1 up-to-date

[mhilbush]

If this fails, I'll get a Raspberry Pi and compile it

If you go that route, I'll give you a few $$ towards the RPi. :wink:

[gurpreet-]

C [libsodium.so+0xb0d0] crypto_auth_hmacsha512_init+0x70

I can't believe I'm saying this about a segfault but this is good news! Lazysodium ran successfully! Lazysodium correctly identified the platform as Arm, the libsodium.so file was placed into the correct temporary directory and finally successfully loaded into memory.

So the fault you are seeing relates to the crypto_auth_hmacsha512_init function. So there's a problem with it on ARM devices. It might be due to Lazysodium or Libsodium. I'm 99% sure it's Lazysodium's parameters that are mismatched. I will debug into that function further.

Let's see if it works without SHA512. Could you delete all tests in the file HashAndroidTest.java and re-run gradle clean test?

[mhilbush]

I deleted all the tests in HashAndroidTest.java but I'm still getting the seg fault in crypto_auth_hmacsha512_init. Is there another test that requires this?

# Problematic frame:
# C  [libsodium.so+0xb0d0]  crypto_auth_hmacsha512_init+0x70

[gurpreet-]

I removed all SHA512 related hashes from Lazysodium and created a new branch called raspberry-pi to host all my changes that I'm doing.

Sorry for making you do this once again but could you please download from raspberry-pi and re-test.

[mhilbush]

Cloned the branch and ran the test again. Still seg faulting.

mark@pi:~/lazysodium-java $ git branch
* raspberry-pi
mark@pi:~/lazysodium-java $ git status
On branch raspberry-pi
Your branch is up-to-date with 'origin/raspberry-pi'.
nothing to commit, working tree clean
mark@pi:~/lazysodium-java $ ./gradlew clean test
> Task :clean
> Task :compileJava
> Task :processResources
> Task :classes
> Task :compileTestJava
> Task :processTestResources NO-SOURCE
> Task :testClasses

> Task :test
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x50aac0d0, pid=22721, tid=1389855856
#
# JRE version: Java(TM) SE Runtime Environment (8.0_65-b17) (build 1.8.0_65-b17)
# Java VM: Java HotSpot(TM) Client VM (25.65-b01 mixed mode linux-arm )
# Problematic frame:
# C  [libsodium.so+0xb0d0]  crypto_auth_hmacsha512_init+0x70

[gurpreet-]

I've got rid of more 512 related functions. At this point I just want to see if Lazysodium/Libsodium runs on it.

Please retry. 😅

[mhilbush]

Still no go.

Confirming I picked up both commits...

commit 204863abfc588b9d81d972a5452c96bf54978729
Author: Gurpreet Paul <gurpreet@terl.co>
Date:   Sat May 25 21:21:45 2019 +0100

    #45 Removed Auth512 related functions

commit b50d0be762a7dfd3599151d145335b3ffb225263
Author: Gurpreet Paul <gurpreet@terl.co>
Date:   Sat May 25 21:03:29 2019 +0100

    #45 Removed all 512 related hashes

Now seg faulting on crypto_auth_hmacsha256_init

# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x50b45d54, pid=23028, tid=1358947440
#
# JRE version: Java(TM) SE Runtime Environment (8.0_65-b17) (build 1.8.0_65-b17)
# Java VM: Java HotSpot(TM) Client VM (25.65-b01 mixed mode linux-arm )
# Problematic frame:
# C  [libsodium.so+0xad54]  crypto_auth_hmacsha256_init+0x70

[gurpreet-]

Okay we can't keep doing this forever 😄

I'll grab a Pi on my way from town tomorrow and I'll debug it and see what's happening. Seems to me that the hashing functions (SHA256 and SHA512) are not binding properly into Libsodium.

We are almost there, just need to work out some things on Lazysodium's side of things 🌞

[mhilbush]

Yes, I think that makes sense. Thanks for doing this. Can I send you a few $$$ towards the RPi?

[gurpreet-]

The offer is greatly appreciated but please don't worry about it. Save it for a rainy day or give it to those less fortunate than us 🙂

There are very few easy to use cryptographic open source libraries out there that have some support for Raspberry Pi. I believe it is long overdue.

I also admire what OpenHAB is doing within the smart home space. A totally vendor-neutral smart home solution is brilliant.

[mhilbush]

Thanks. Your help is much appreciated on this.

I also admire what OpenHAB is doing within the smart home space. A totally vendor-neutral smart home solution is brilliant.

Yeah, it's a great platform. A somewhat steep learning curve, but quite powerful. And growing every day in terms of supported technologies.

[gurpreet-]

I'm now the proud owner of a Raspberry Pi 3B 😄

I managed to find out that some functions are not playing well with the underlying Libsodium library.

There is a problem with SHA256 (auth), SHA512 (auth and hash) and signing (only the "lazy" functions, the native ones such as crypto_sign are still there).

So to summarise, the following native and their "lazy" related functions may not be available for use until I find out what's going on:

crypto_auth_hmacsha256*
crypto_auth_hmacsha512*
crypto_hash_sha512*

I have submitted all the changes to the raspberry-pi branch for now. Once you've got those latest changes you can build a JAR file with ./gradlew jar. Theoretically this should compile all of Lazysodium's dependencies into one neat JAR file which you can incorporate into your RPi projects and get coding straight away. You may need to read the Getting Started section of the Lazysodium docs for information on how to get started.

In the meantime I will try and find out what's causing the errors 🙂

[mhilbush]

I'm now the proud owner of a Raspberry Pi 3B

Congrats

I cloned the branch and built the jar. Built my app with the new jar and installed on the Pi. The libsodium library was loaded successfully, and the crypto APIs I'm using are working fine. Relevant debug log from my app...

2019-05-28 02:37:49.466 [DEBUG] [ding.doorbird.internal.DoorbirdEvent] - Received and successfully decrypted a Doorbird event!!
2019-05-28 02:37:49.469 [DEBUG] [ding.doorbird.internal.DoorbirdEvent] - Event is eventId='motion', intercomId='ghbyaa', timestamp=1559011067
2019-05-28 02:37:49.471 [DEBUG] [ng.doorbird.internal.DoorbirdHandler] - Handler: Update MOTION channels for thing doorbird:d101:doorbell

Thanks for making this happen!

[gurpreet-]

Really glad it worked! We got there in the end!

If there are any problems, please don't hesitate in opening a new issue.

Happy programming!

[mhilbush]

@gurpreet- I forgot to ask... Will this version also run on an RPi2?

[gurpreet-]

Yes so if it was compiled for the armv7 architecture, then it should work for all devices that run armv7 and above. So it should work for the Raspberry 2 (which is armv7) and the Raspberry Pi 3.

[mhilbush]

FYI https://community.openhab.org/t/doorbird-binding/73564/67?u=mhilbush

Thanks go to you really. :wink:

[gurpreet-]

Awesome! Really glad it works! 😄

I'll leave this issue open for a few weeks so that if there are any issues you can post them here.

[mhilbush]

Thanks. Very much appreciated.

Also at some point, after you do a formal release, I can update my app to automatically pull the LazySodium jar from the maven repo (which is the way it was set up before we started testing the RPi version).