Support configuring which library gets priority

[dmitry-timofeev]

Overview

Currently lazysodium can be instantiated either:

• with a bundled library from its JAR: new SodiumJava(),
• or with a system (installed) one: new SodiumJava("sodium"),
• or a custom one: new SodiumJava("/home/john.doe/app/libsodium.so").

For applications which are deployed in a known environment that is perfectly enough. However, if lazysodium is used in a library, or an application that is likely to be deployed in a wide range of environments (uncontrolled by the dev), it might be needed to get the following behaviour:

1. Prefer system library, fall-back to bundled. That allows the app/lib to benefit from updates to system sodium (e.g., security fixes) without the need to wait for lazysodium release, update and release their app/lib.
2. System-only mode. That might be useful if using the system sodium is a requirement. It is currently achievable with new SodiumJava("sodium").
3. Bundled-only mode. Might be useful if some incompatibilities are discovered with the system version and it is not possible to upgrade it. It is currently achievable with new SodiumJava().

Workaround

Option 1 (prefer system library) can currently be achieved with the following client code:

    SodiumJava sodium;
    try {
        // Try to use the system one, using its name
        sodium = new SodiumJava("sodium");
    } catch (UnsatisfiedLinkError e) {
        // Use the bundled library 
        sodium = new SodiumJava();
    }

Whilst it works, it is not intuitive to write, especially for new users.

Questions

1. Shall lazysodium support such functionality, or leave it to the users?
2. Shall it support all three options through a single interface for easy of use and discoverability (e.g., SodiumJava(LibraryLoadingMode))?
3. What shall be the default mode for SodiumJava() — bundled or prefer-system? First is guaranteed to work, but second is more likely to receive timely updates.

[gurpreet-]

Hello again @dmitry-timofeev,

These are very thought provoking questions.

Background

I wanted Lazysodium to be as frictionless as possible to get started with. I could have relied on the user to install libsodium.so or libsodium.dylib or libsodium.dll files on their system, but even that is a headache as users may have had to compile libsodium to get it installed. Bundling those shared libraries with Lazysodium had several positive effects such as being able to use Lazysodium as a dependency of a dependency. As you have rightly mentioned, Lazysodium is very flexible in the sense that it can load from a variety of different sources, such as the system or an absolute path or from within the bundled resources.

So in the spirit of making it as easy as possible to use for developers, I decided that Lazysodium should load its shared libraries from bundled resources first.

However, you have raised a very valid point that the developer may not control the environment (and therefore to a certain extent how Lazysodium loads its shared libraries). Thus, a developer may want control over that... And I wholeheartedly agree with that.

Answers

So to answer Q3, the default going forward should be:

1. We need to first try to load from the system installed libraries.
2. If that fails, load from the bundled shared libraries.

To answer Q1, the user does not always know what's best. We can make it easier for them by first loading from the system then falling back to the bundled shared libs, which are tried and tested to be working.

To answer Q2. This is a fantastic idea and provides the user with more explicit control over how Lazysodium loads its shared libsodium libraries. As mentioned, the default should be the procedure outlined above. But the user can override it by setting variables within LibraryLoadingMode.

Further questions

1. So now the question becomes what will LibraryLoadingMode look like? It needs to provide developers with a way of saying "Load from system first, then if that fails load from bundled resources, then if that fails load from absolute path, if that fails throw exception." I am not sure how best to achieve this.
2. Slightly related question, do you know how other libraries handle this problem? I imagine it is an infrequent (but not rare) problem, so I am curious how other libraries handle this? Do you know of any projects I could look at?

[dmitry-timofeev]

Hi @gurpreet- ,

Thank you for the background and the feedback!

Regarding your first question, I imagined the following interface for instantiation:

SodiumJava
----------
# The existing interface, uses the default loading policy
+ SodiumJava() 

# A new constructor that gives explicit control over the policy,
# if the default is not suitable
+ SodiumJava(mode: LibraryLoadingMode)

# The existing constructor accepting a 'library locator',
# which may be a filesystem path. 
#
# Mostly for advanced usage,
# e.g., when a user needs to load the sodium library
# that they manage themselves from a path, therefore,
# has its own constructor to not complicate the two previous
# that must be enough for most cases.
+ SodiumJava(libPath: String)

With such interface, LibraryLoadingMode may be a simple enum:

LibraryLoadingMode
------------------
+ PREFER_INSTALLED
+ PREFER_BUNDLED
+ INSTALLED_ONLY

It might also be reasonable to consider pushing handling of various loading modes into LibraryLoader for easier testing.

Do you think that would work?

I don't know of libraries with a similar mechanism, but maybe some Java bindings to popular libraries might have one.

If you don't mind, I could try to implement this feature (probably, later this week/next week). If other questions arise, we can ponder on them meanwhile.

[gurpreet-]

@dmitry-timofeev,

Yes I think that could work, although I was thinking LibraryLoadingMode could be something like the following (this should be treated like pseudocode as I'm writing by hand at the moment):

abstract class LibraryLoadingMode {

    // Returns load order. One of INSTALLED, BUNDLED, PATH
    abstract Mode[] loadOrder();

    // Will be an absolute path to the library, can be null.
    abstract String path();

    void loadLibrary() {
        // Using loadOrder(), load a shared library according to the user's wishes.
    }
}

And therefore can be used easily like:

new SodiumJava(new LibraryLoadingMode() {

    Mode[] loadOrder() {
        return [INSTALLED, BUNDLED, PATH];
    }

   String path() {
        return "/absolute/path/to/libsodium.so"
    }

});

[dmitry-timofeev]

:+1: that would be more flexible, and it would be possible to provide pre-defined instances (as they can be immutable) covering the common use-cases we discussed, so that the user does not need to create them themselves unless they need to.

Do you think though it is needed to attempt to load both system/bundled and using a certain path? I certainly can't imagine all the use-cases, but I think that one either ensures there is their library build supplied with the application and available by a certain path, or they rely on the system and/or the bundled.

I had a chance to hack on it yesterday, and currently went with the simpler implementation using three pre-defined modes: #52

[dmitry-timofeev]

Also, I've managed to enable integration testing of both modes by running the tests in corresponding classes in individual (forked) VMs: https://github.com/exonum/exonum-java-binding/pull/991 Each VM can then load its own version of sodium. The configuration in that PR is for Maven Surefire plugin + JUnit 5, but I am sure something similar exists for Gradle.

[gurpreet-]

Thank you @dmitry-timofeev,

I trialled my version using the abstract class and to be honest, it's not at all great. It's confusing to the end user as to what to override and if they're not using an absolute path shared library why should they override path() anyway.

Your implementation targets all 3 loading modes strategically and in a way that aligns with the mission of this library - which is to make it effortless to use cryptography in any project.

So I have merged your changes into the library. I am preparing a v4 of lazysodium which will include all the new changes to the core libsodium project.

Stay tuned 👍

[dmitry-timofeev]

Thank you! Looking forward to the release!

[gurpreet-]

I made some minor improvements to the loading functionality. Should be in release 4.0.1 now. Overall I am super happy with how it loads the shared libraries now!

Happy to close this 😀