[dargon2] Unable to load dynamic libraries with compiled native Dart project

[JagandeepBrar]

It appears that compiling to a native Dart binary will cause the library to fail to load.

When using dart run there is no issue with utilizing the library and all functionality can complete as expected, but when first using dart compile exe <file> and running the generated binary directly, the library is unable to be loaded.

Null check operator used on a null value
#0      DartLibLoader.getPath (package:dargon2/src/native/dart_lib_loader.dart:34)
#1      DartLibLoader.loadLib (package:dargon2/src/native/dart_lib_loader.dart:24)
#2      new LocalBinder._ (package:dargon2_core/src/native/local_binder.dart:155)
#3      LocalBinder.initialize (package:dargon2_core/src/native/local_binder.dart:130)
#4      new DArgon2Native (package:dargon2_core/src/native/dargon2_native.dart:17)
#5      argon2 (package:dargon2/src/argon2.dart:11)

This is occurring because the dynamic library loader is currently using the package path to find the required blobs/dynamic libraries, which is not always found on the host machine (we should be able to expect compiled Dart code to execute without the Dart SDK or packages).

I have a workaround by using dargon2_core directly and creating my own DynamicLibrary loader by shipping the compiled binary with the required blobs, but wanted to validate that there isn't another way to allow compiled Dart binaries to load the library directly from the dargon2 package.

Details:

• Host: MacOS 13.2.1 (ARM)
• Dart: 3.0.0-322.0.dev (dev) (Fri Mar 10 06:41:16 2023 -0800) on "macos_arm64"
• Package: 3.2.1

[tmthecoder]

@JagandeepBrar Hi, currently there isn't another way to do this. I'm thinking of adding an env variable lookup to get the path from there first then rely on the dart package, but I haven't decided on that route for sure

[hasimyerlikaya]

Hi, I have changed the getPath method to return the blob file path from Env, and I put the libargon2-linux.so file in my Docker container. Now I can load the library successfully. But when I want to use it, an error occurs, and the container crashes.

The error message is "libgcc_s.so.1 must be installed for pthread_cancel to work"

Do you have any ideas?

 @override
  String getPath() {
    var path = Platform.environment['DARGON2_LIB_PATH'] ?? '';
    print('File Path: $path');
    return path;
  }

[JagandeepBrar]

@hasimyerlikaya I have not yet tried running this within a Docker container, but presumably you need to install the libgcc package within the Docker container. You can try adding:

RUN apt-get update && apt-get -y install libgcc1-amd64-cross

[hasimyerlikaya]

@JagandeepBrar Thank you. I did it, but it didn't work. Here is my Docker file. It has been built successfully. But I got the same error.

I guess I can't use the Dargon2 package, and I need to find another solution.

FROM dart:stable AS build

# Resolve app dependencies.
WORKDIR /app
COPY pubspec.* ./
RUN dart pub get

# Copy app source code and AOT compile it.
COPY . .

RUN apt-get update -y
RUN apt-get install -y libgcc1-amd64-cross

# Ensure packages are still up-to-date if anything has changed
RUN dart pub get --offline
RUN dart compile exe bin/server.dart -o bin/server

# Build minimal serving image from AOT-compiled `/server` and required system
# libraries and configuration files stored in `/runtime/` from the build stage.
FROM scratch
COPY --from=build /runtime/ /
COPY --from=build /app/bin/server /app/bin/
COPY service-account-stage.json /app/bin/service-account-stage.json
COPY service-account-prod.json /app/bin/service-account-prod.json

COPY libargon2/libargon2-darwin.dylib /app/bin/libargon2/libargon2-darwin.dylib
COPY libargon2/libargon2-linux.so /app/bin/libargon2/libargon2-linux.so
COPY libargon2/libargon2-win.dll /app/bin/libargon2/libargon2-win.dll

# Start server.
EXPOSE 8080
CMD ["/app/bin/server"]

[GleammerRay]

@JagandeepBrar @hasimyerlikaya

I have the same exception, it only works when I launch the executable using the absolute path. Could be related to https://github.com/dart-lang/sdk/issues/32901.

I think it was working with relative paths for me before, but I'm not too sure. :thinking:

Edit 1: of course this is begging for a solution, but until then I am using the following workaround: have your script call itself via an absolute path with necessary arguments supplied for an argon2 call.

Edit 2: knowing that the dargon2 is searching for the libraries within itself this won't work, see https://github.com/tmthecoder/dargon2/issues/17#issuecomment-1678347791 and https://github.com/tmthecoder/dargon2/issues/17#issuecomment-1678430978 instead.

[hasimyerlikaya]

@GleammerRay Unfortunately, I will be looking for a different package. I don't have enough experience to solve this issue.

[JagandeepBrar]

@hasimyerlikaya @GleammerRay I was able to get this working without issue utilizing Docker, here is a minimum working version of my Dockerfile:

# <-- Libraries -->
FROM --platform=$TARGETPLATFORM ubuntu:latest as libraries
WORKDIR /app

RUN apt update && apt install curl git gcc make -y

RUN git clone https://github.com/P-H-C/phc-winner-argon2.git
RUN cd phc-winner-argon2 && git checkout 20190702 -b 20190702
RUN cd phc-winner-argon2 && make

# <-- Build -->
FROM --platform=$TARGETPLATFORM dart:stable as build
WORKDIR /app

COPY pubspec.yaml pubspec.lock ./
RUN dart pub get

COPY ./bin ./bin
COPY ./lib ./lib
RUN dart compile exe bin/executable.dart -o output_exe

# <-- Runtime -->
FROM --platform=$TARGETPLATFORM ubuntu:latest as runtime
WORKDIR /app

COPY --from=libraries /app/phc-winner-argon2/libargon2.so.1 /usr/local/lib/libargon2.so.1
COPY --from=build /app/output_exe /app/output_exe
RUN ldconfig

EXPOSE 9303
ENTRYPOINT [ "/app/output_exe" ]

Naturally the above uses stub names for the main Dart file and output executable as well as an arbitrary exposed port, but should give you a solid foundation. Now this splits the Docker build into 3 stages - libraries, build, and runtime.

1. libraries is where we build the argon2 library (I have additional libraries being built in this stage but are irrelevant for this scenario)
2. build is where the actual Dart executable is built
3. runtime is where the executable is run with the library installed

The reason for building the binary in the libraries stage instead of copying the existing binary for this repository is that unix shared libraries are architecture-dependant. Compiling the library ourselves is quick and allows for the Docker image to be built for ARM64 and AMD64 without running into library linking issues. Also note the --platform=$TARGETPLATFORM flag being set for each of the base images, again to ensure that we can have a correctly built library and Dart executable for the target platform since Dart compilation is architecture-specific.

For this scenario I build the argon2 shared library and copy it to the runtime image, which is just a standard Ubuntu image, into the unix-standard library location (/usr/local/lib). We also run ldconfig after which will ensure that the library is correctly dynamically linked.

As a side-note, splitting the image into stages allows the final publishable image to be (much) smaller, utilizing the dart base image I would see a final image of about 600MB vs. about 80MB utilizing staged building.

---

Now in the code, I installed the dargon2_core package and implement the LibLoader package which loads the dynamic library and can be passed to the DArgon2Native constructor to get a valid DArgon2 instance. Basic example:

import 'dart:ffi';
import 'dart:io';
import 'package:dargon2_core/dargon2_core.dart';

final dargon2 = DArgon2Native(_DArgon2Loader());

class _DArgon2Loader implements LibLoader {
  @override
  String getPath() {
    return '/usr/local/lib/libargon2.so.1';
  }

  @override
  DynamicLibrary loadLib() {
    final path = getPath();

    if (File(path).existsSync()) {
      return DynamicLibrary.open(path);
    }

    throw UnsupportedError('Argon2 dynamic library is not available.');
  }
}

...
/// Continue to utilize argon2 functions, such as `hashPasswordString` or `verifyHashString`, using the dargon2 instantiated variable. 

The above is just a quick example and does not account for other platforms or local-dev loading, but this can be accounted for in whichever way you would like. A simple solution would be to set an environment variable in the Dockerfile to tell your program to load the binary from the standard-location (for example, ENVIRONMENT=docker and then using Platform.environment('ENVIRONMENT') and checking and handling based on the value).

Hopefully this was useful in getting DArgon2 working for you guys!

[GleammerRay]

@JagandeepBrar thank you so much! I didn't know it wasn't compiling because it was failing to find the library.

I am not making any changes to my building environment, but I have forked the repository itself and am using my own fork. Since my Dart executable is shipped together with my Flutter app I was able to simply do the following: https://github.com/GlitterWare/dargon2/commit/011ba63fa78ea33d319acdcc5c576edde6daa908. Paths for MacOS/Windows may need adjustment, haven't tested them yet.

[hasimyerlikaya]

@JagandeepBrar Thank you very much. I have updated my docker and dart_lib_loader.dart files. Now It works perfectly. I could not do it without your help.

# <-- Libraries -->
FROM ubuntu:latest as libraries
WORKDIR /app

RUN apt update && apt install curl git gcc make -y

RUN git clone https://github.com/P-H-C/phc-winner-argon2.git
RUN cd phc-winner-argon2 && git checkout 20190702 -b 20190702
RUN cd phc-winner-argon2 && make

# <-- Build -->
FROM dart:stable AS build

# Resolve app dependencies.
WORKDIR /app
COPY pubspec.* ./
RUN dart pub get

# Copy app source code and AOT compile it.
COPY . .
# Ensure packages are still up-to-date if anything has changed
RUN dart pub get --offline
RUN dart compile exe bin/server.dart -o bin/server

# <-- Runtime -->
FROM ubuntu:latest as runtime
WORKDIR /app

COPY --from=build /app/bin/server /app/bin/
COPY --from=libraries /app/phc-winner-argon2/libargon2.so.1 /usr/local/lib/libargon2.so.1
RUN ldconfig

EXPOSE 8080
ENTRYPOINT ["/app/bin/server"]