Add Dockerfile

[BobyMCbobs]

Goal

This PR aims to add a Dockerfile for containerisation.

---

Fixes: #60

[teesloane]

Nice! Great to see your PR! Have you got it working yourself? I tried running:

docker build --tag firn:1.0 .

and got the following output:

 ---> Running in 00fb0b37ce1c
Error: Requirements for building native images are not fulfilled [cause: Unable to use jar-files from directory /usr/local/lib/jvmci]
com.oracle.svm.driver.NativeImage$NativeImageError: Requirements for building native images are not fulfilled [cause: Unable to use jar-files from directory /usr/local/lib/jvmci]
    at com.oracle.svm.driver.NativeImage.showError(NativeImage.java:1554)
    at com.oracle.svm.driver.NativeImage.build(NativeImage.java:1296)
    at com.oracle.svm.driver.NativeImage.performBuild(NativeImage.java:1269)
    at com.oracle.svm.driver.NativeImage.main(NativeImage.java:1228)
    at com.oracle.svm.driver.NativeImage$JDK9Plus.main(NativeImage.java:1740)
The command '/bin/sh -c native-image -jar target/firn-0.0.5-SNAPSHOT-standalone.jar   -H:Name=firn   -H:+ReportExceptionStackTraces   -J-Dclojure.spec.skip-macros=true   -J-Dclojure.compiler.direct-linking=true   --initialize-at-build-time   --report-unsupported-elements-at-runtime   -H:IncludeResources=libmylib.dylib   -H:IncludeResources=libmylib.so   -H:IncludeResources=firn/.*   -H:Log=registerResource:   -H:ReflectionConfigurationFiles=reflection.json   -H:+JNI   --verbose   --allow-incomplete-classpath   --no-server' returned a non-zero code: 1

I'm do not know Docker well so I'm not sure what needs to change yet. Just wanted to check if you had any success before I check it out more closely over the coming days.

[BobyMCbobs]

Nice! Great to see your PR! Have you got it working yourself? I tried running:

docker build --tag firn:1.0 .

and got the following output:

 ---> Running in 00fb0b37ce1c
Error: Requirements for building native images are not fulfilled [cause: Unable to use jar-files from directory /usr/local/lib/jvmci]
com.oracle.svm.driver.NativeImage$NativeImageError: Requirements for building native images are not fulfilled [cause: Unable to use jar-files from directory /usr/local/lib/jvmci]
  at com.oracle.svm.driver.NativeImage.showError(NativeImage.java:1554)
  at com.oracle.svm.driver.NativeImage.build(NativeImage.java:1296)
  at com.oracle.svm.driver.NativeImage.performBuild(NativeImage.java:1269)
  at com.oracle.svm.driver.NativeImage.main(NativeImage.java:1228)
  at com.oracle.svm.driver.NativeImage$JDK9Plus.main(NativeImage.java:1740)
The command '/bin/sh -c native-image -jar target/firn-0.0.5-SNAPSHOT-standalone.jar   -H:Name=firn   -H:+ReportExceptionStackTraces   -J-Dclojure.spec.skip-macros=true   -J-Dclojure.compiler.direct-linking=true   --initialize-at-build-time   --report-unsupported-elements-at-runtime   -H:IncludeResources=libmylib.dylib   -H:IncludeResources=libmylib.so   -H:IncludeResources=firn/.*   -H:Log=registerResource:   -H:ReflectionConfigurationFiles=reflection.json   -H:+JNI   --verbose   --allow-incomplete-classpath   --no-server' returned a non-zero code: 1

I'm do not know Docker well so I'm not sure what needs to change yet. Just wanted to check if you had any success before I check it out more closely over the coming days.

Hey @teesloane,

This is my current progress. I'm running into the same issue, but want to get the conversions happening with this PR. I haven't used Rust with a Clojure project before, there might something I'm missing for this error to occur

[teesloane]

Ok, thanks. I tried a few things but didn't have any luck. It seems that the JAVA_HOME directory is not set properly:

It should be ENV JAVA_HOME="/opt/graalvm-ce-java11-20.2.0/content/home" or something like that. To take a guess.. the Clojure docker image is not able to access the GraalVM folder?

[BobyMCbobs]

Ok, thanks. I tried a few things but didn't have any luck. It seems that the JAVA_HOME directory is not set properly:

It should be ENV JAVA_HOME="/opt/graalvm-ce-java11-20.2.0/content/home" or something like that. To take a guess.. the Clojure docker image is not able to access the GraalVM folder?

@teesloane, thank you for the suggestion! It seemed to be missing the target folder from the clojure build stage (which wasn't in there) - added in 8031964

It builds completely, but I'm getting this error:

Error: Could not find or load main class firn.core

Would you have an idea about that? I'm not the most familiar in the ways that Java works

[teesloane]

@BobyMCbobs what is the docker commands you are using to execute the Dockerfile? Is the idea that I have to run docker build ..., to create the internal images and docker run ... will create the binary?

[BobyMCbobs]

@BobyMCbobs what is the docker commands you are using to execute the Dockerfile? Is the idea that I have to run docker build ..., to create the internal images and docker run ... will create the binary?

Build

docker build -f firn .

Note: I'm only tagging it as firn until it's published

Run

docker run -it --rm firn

[teesloane]

I'm not the most familiar in the ways that Java works

Heh, same 😅. I did a little bit of research for a quick fix but couldn't find anything. Maybe the class path is not in the right place (I don't know if that makes sense). I'll try and poke around at in the coming days but I can't promise I'll be able to dedicate too much time to it.

[BobyMCbobs]

I'm not the most familiar in the ways that Java works

Heh, same . I did a little bit of research for a quick fix but couldn't find anything. Maybe the class path is not in the right place (I don't know if that makes sense). I'll try and poke around at in the coming days but I can't promise I'll be able to dedicate too much time to it.

Were you able to find anything out about this @teesloane?

[teesloane]

@BobyMCbobs Unfortunately not - I've been preoccupied with some other development. Still would like to see this go through. Can I leave this in your court for the time being? I might be able to jump on it over some coming days off, but I'm not sure.

In the meantime, I'm also keen to know what your original use case for the dockerfile would be? For me, I see it as an easier way to compile the binary locally for testing purposes (and not have to install GraalVM, Rust etc.) Are there other purposes I'm not thinking of -- for example, would the dockerfile actually compile the whole binary, and then run the firn commands as well?

[BobyMCbobs]

@BobyMCbobs Unfortunately not - I've been preoccupied with some other development. Still would like to see this go through. Can I leave this in your court for the time being? I might be able to jump on it over some coming days off, but I'm not sure.

Yeah of course, no stress.

I just paired with @zachmandeville on this briefly and we got it further to actually running the firn binary

$ docker run -it --rm firn
Firn - A static-site generator for org-mode.

Usage: firn [options] action

Options:
  -p, --port PORT  4000          Port number
  -h, --help
  -v, --version
  -r, --repl
  -d, --dir PATH   /app/clojure  Absolute path of directory to build/serve

Actions:
  build    Build a static site in a directory with org files.
  new      Scaffold files and folders needed to start a new site.
  serve    Runs a development server for processed org files.

But when any subcommand (e.g new, build, serve) is run, there's this error:

Exception in thread "main" java.lang.UnsatisfiedLinkError: no mylib in java.library.path
        at com.oracle.svm.core.jdk.NativeLibrarySupport.loadLibrary(NativeLibrarySupport.java:131)
        at java.lang.ClassLoader.loadLibrary(ClassLoader.java:228)
        at java.lang.Runtime.loadLibrary0(Runtime.java:830)
        at java.lang.Runtime.loadLibrary(Runtime.java:238)
        at java.lang.System.loadLibrary(System.java:352)
        at clojure.lang.RT.loadLibrary(RT.java:509)
        at firn.core$_main.invokeStatic(core.clj:98)
        at firn.core$_main.doInvoke(core.clj:89)
        at clojure.lang.RestFn.applyTo(RestFn.java:137)
        at firn.core.main(Unknown Source)

In the meantime, I'm also keen to know what your original use case for the dockerfile would be? For me, I see it as an easier way to compile the binary locally for testing purposes (and not have to install GraalVM, Rust etc.) Are there other purposes I'm not thinking of -- for example, would the dockerfile actually compile the whole binary, and then run the firn commands as well?

I'm wanting to use firn in my CI for https://gitlab.com/flattrack/flattrack / https://github.com/FlatTrackio/FlatTrack as well as https://github.com/sharingio/pair. There are many benefits to having it containerised, the ones that you name of course included. It would also make development of docs more rapid. Portability of course too.

[teesloane]

But when any subcommand (e.g new, build, serve) is run, there's this error:

Exception in thread "main" java.lang.UnsatisfiedLinkError: no mylib in java.library.path
        at com.oracle.svm.core.jdk.NativeLibrarySupport.loadLibrary(NativeLibrarySupport.java:131)
        at java.lang.ClassLoader.loadLibrary(ClassLoader.java:228)
        at java.lang.Runtime.loadLibrary0(Runtime.java:830)
        at java.lang.Runtime.loadLibrary(Runtime.java:238)
        at java.lang.System.loadLibrary(System.java:352)
        at clojure.lang.RT.loadLibrary(RT.java:509)
        at firn.core$_main.invokeStatic(core.clj:98)
        at firn.core$_main.doInvoke(core.clj:89)
        at clojure.lang.RestFn.applyTo(RestFn.java:137)
        at firn.core.main(Unknown Source)

Ah, I had an idea but I'm not sure if it's the answer. The clojure<>rust interop is not contained within a single binary (unfortunately). Because of this, when Firn is installed, it moves the dylib into ~/.firn. So, I'm guessing the docker container can't find that library. Aside: I should really rename mylib to firn-parser or something :|.

I'm wanting to use firn in my CI for https://gitlab.com/flattrack/flattrack / https://github.com/FlatTrackio/FlatTrack as well as https://github.com/sharingio/pair. There are many benefits to having it containerised, the ones that you name of course included. It would also make development of docs more rapid. Portability of course too.

Ah gotcha. I've been using a hacky script that pulls firn down into a container in my CI process for my personal wiki. Here it is if it's any help in the meantime:

#!/usr/bin/env bash
set -euo pipefail

latest_release="$(curl -sL https://raw.githubusercontent.com/theiceshelf/firn/master/clojure/resources/FIRN_VERSION)"

case "$(uname -s)" in
    Linux*)     platform=linux;;
    Darwin*)    platform=mac;;
esac

download_url="https://github.com/theiceshelf/firn/releases/download/v$latest_release/firn-$platform.zip"

echo -e "Downloading Firn from: $download_url."
curl -o "firn-$latest_release-$platform.zip" -sL $download_url
unzip -qqo "firn-$latest_release-$platform.zip"
chmod +x firn
rm "firn-$latest_release-$platform.zip"
./firn build

Goodluck! Flattrack sounds pretty cool by the way!

Edit: unrelated - I just found and cleared the image for Firn from prototyping with docker... it was 17gb 🙀 .

[BobyMCbobs]

But when any subcommand (e.g new, build, serve) is run, there's this error:

Exception in thread "main" java.lang.UnsatisfiedLinkError: no mylib in java.library.path
        at com.oracle.svm.core.jdk.NativeLibrarySupport.loadLibrary(NativeLibrarySupport.java:131)
        at java.lang.ClassLoader.loadLibrary(ClassLoader.java:228)
        at java.lang.Runtime.loadLibrary0(Runtime.java:830)
        at java.lang.Runtime.loadLibrary(Runtime.java:238)
        at java.lang.System.loadLibrary(System.java:352)
        at clojure.lang.RT.loadLibrary(RT.java:509)
        at firn.core$_main.invokeStatic(core.clj:98)
        at firn.core$_main.doInvoke(core.clj:89)
        at clojure.lang.RestFn.applyTo(RestFn.java:137)
        at firn.core.main(Unknown Source)

Ah, I had an idea but I'm not sure if it's the answer. The clojure<>rust interop is not contained within a single binary (unfortunately). Because of this, when Firn is installed, it moves the dylib into ~/.firn. So, I'm guessing the docker container can't find that library. Aside: I should really rename mylib to firn-parser or something :|.

So is firn meant to extract libmylib to ~/.firn on start time? If so it kinda seems like a chicken an egg kinda thing. Perhaps I could try it been placed in that location for the container image too?

I'm wanting to use firn in my CI for https://gitlab.com/flattrack/flattrack / https://github.com/FlatTrackio/FlatTrack as well as https://github.com/sharingio/pair. There are many benefits to having it containerised, the ones that you name of course included. It would also make development of docs more rapid. Portability of course too.

Ah gotcha. I've been using a hacky script that pulls firn down into a container in my CI process for my personal wiki. Here it is if it's any help in the meantime:

#!/usr/bin/env bash
set -euo pipefail

latest_release="$(curl -sL https://raw.githubusercontent.com/theiceshelf/firn/master/clojure/resources/FIRN_VERSION)"

case "$(uname -s)" in
    Linux*)     platform=linux;;
    Darwin*)    platform=mac;;
esac

download_url="https://github.com/theiceshelf/firn/releases/download/v$latest_release/firn-$platform.zip"

echo -e "Downloading Firn from: $download_url."
curl -o "firn-$latest_release-$platform.zip" -sL $download_url
unzip -qqo "firn-$latest_release-$platform.zip"
chmod +x firn
rm "firn-$latest_release-$platform.zip"
./firn build

This is good.

Goodluck! Flattrack sounds pretty cool by the way! Thank you!

Edit: unrelated - I just found and cleared the image for Firn from prototyping with docker... it was 17gb . Now that's a load of cache!

[teesloane]

So is firn meant to extract libmylib to ~/.firn on start time? If so it kinda seems like a chicken an egg kinda thing. Perhaps I could try it been placed in that location for the container image too?

Sort of. The init function handles checking if the dylib is in the user's home directory; if it's not, it moves it into (System/getProperty "user.home"). So, if one can simulate that using docker, it might work.

[BobyMCbobs]

So is firn meant to extract libmylib to ~/.firn on start time? If so it kinda seems like a chicken an egg kinda thing. Perhaps I could try it been placed in that location for the container image too?

Sort of. The init function handles checking if the dylib is in the user's home directory; if it's not, it moves it into (System/getProperty "user.home"). So, if one can simulate that using docker, it might work.

In 510a96d, I added a non-root user. It appears that by copying it libmylib.so into the .firn folder of the user, firn now works!

[teesloane]

Wonderful! Thank you for your hard work @BobyMCbobs . One last thing - do you mind adding a note or two to the readme.org in the root of this repo? It sounds like we could make a note in the "Usage" and/or the "Development" section. I think you are probably understand the problem better than I. If you don't have time that's no problem - I will just need to tinker around with docker in Firn's CI process before I can speak to it more clearly.

[BobyMCbobs]

@teesloane, We're ready for a review!

Testing

Preparation

git clone https://github.com/theiceshelf/firn
cd firn

docker build -t firn/firn .

Serving

docker run -it --rm -p 4000:4000 -v "$PWD":/home/user/firn --workdir /home/user/firn/docs firn/firn serve

Building

docker run -it --rm -p 4000:4000 -v "$PWD":/home/user/firn --workdir /home/user/firn/docs firn/firn build

(or s/docker/podman/g)

Publishing

The most well known registry for container images is DockerHub, I would recommend using that as one of the places to publish it. Can you create the firn organisation and firn repo inside of it (will show up at https://hub.docker.com/r/firn/firn)

Build CI

It appears that you're using GitHub actions, I don't have any experience with it. You may wish to use a configuration like https://github.com/docker/build-push-action

Remaining questions

• 1. since there's the native image stage, should it actually require Java/OpenJDK to run? (I like to make containers as slim as possible)
• 2. currently we're using Java 11, should/can it be any newer release?

[BobyMCbobs]

Wonderful! Thank you for your hard work @BobyMCbobs.

@teesloane, My pleasure!

One last thing - do you mind adding a note or two to the readme.org in the root of this repo? It sounds like we could make a note in the "Usage" and/or the "Development" section.

Yeah sure, I can do that.

I think you are probably understand the problem better than I. If you don't have time that's no problem - I will just need to tinker around with docker in Firn's CI process before I can speak to it more clearly.

Documentation is a wonderful thing, I'd be more than happy to help! Perhaps the container image can help with firn's own docs in the CI! :smile:

[BobyMCbobs]

@teesloane, I've just updated README.org

[teesloane]

Excellent - merging!

[teesloane]

I forgot to address some points

1. since there's the native image stage, should it actually require Java/OpenJDK to run? (I like to make containers as slim as possible)

Not sure if I understand exactly - do you mean, since it's compiling a native image/executable, does it really need java to run that image? If so... it shouldn't, if I'm understanding graal's native-image thing (I think there's a fallback flag, but I don't know much about it). Do you mean that we could remove FROM openjdk:11 as final... etc?

• 1. currently we're using Java 11, should/can it be any newer release?

I don't see why not so long as it compiles and everything works.

[teesloane]

Last question, hopefully @BobyMCbobs - now that there is this docker image, I can now technically build the firn binary using the command: docker build -t theiceshelf/firn . in my CI job, yes? If so, that would clean up the need for the compile script (and maybe even work on the mac executor?) If this is the case, do you know how I would pull that the compiled binary out of docker container and make it available to be uploaded as an artifact? I suppose the pushing of the docker image will have to happen first, then the linux/mac artifacts could be created by pulling from that image.

[BobyMCbobs]

Excellent - merging!

Thank you so much! Woohoo!

[BobyMCbobs]

Not sure if I understand exactly - do you mean, since it's compiling a native image/executable, does it really need java to run that image? If so... it shouldn't, if I'm understanding graal's native-image thing (I think there's a fallback flag, but I don't know much about it). Do you mean that we could remove FROM openjdk:11 as final... etc?

To make the container image smaller, it would be nice to have the final base as either alpine (minimal) or scratch (nothing). For scratch to be supported, the binary would need to be statically compiled. I think it's dynamically linked

[BobyMCbobs]

Last question, hopefully @BobyMCbobs - now that there is this docker image, I can now technically build the firn binary using the command: docker build -t theiceshelf/firn . in my CI job, yes?

Yeah, you could probably docker build -t theiceshelf/firn . follow by docker run -it --rm -v "$PWD":"/tmp/folder" --entrypoint /bin/bash theiceshelf/firn cp /app/bin/firn /tmp/folder/firn to copy it. However this may lead to a disparity between platform build processes.

If so, that would clean up the need for the compile script (and maybe even work on the mac executor?) If this is the case, do you know how I would pull that the compiled binary out of docker container and make it available to be uploaded as an artifact? I suppose the pushing of the docker image will have to happen first, then the linux/mac artifacts could be created by pulling from that image.

You won't be able to do the suggested process above for macOS, as Docker runs only native Linux and Windows binaries under the hood

[BobyMCbobs]

Extra thing about the CI for the container build: when you tag a release, is it set up to use that tag in the container image?

[teesloane]

Extra thing about the CI for the container build: when you tag a release, is it set up to use that tag in the container image?

I don't think so. Not sure how to do that (do you?). I assumed that it could just push the docker image as "latest" every time — but now that I think of it, it should have a tagged version too.

Yeah, you could probably docker build -t theiceshelf/firn . follow by docker run -it --rm -v "$PWD":"/tmp/folder" --entrypoint /bin/bash theiceshelf/firn cp /app/bin/firn /tmp/folder/firn to copy it. However this may lead to a disparity between platform build processes.

Hmm, I'll think about it. Thanks for posting the command to use, though. Too bad Graalvm can't cross compile.

To make the container image smaller, it would be nice to have the final base as either alpine (minimal) or scratch (nothing). For scratch to be supported, the binary would need to be statically compiled. I think it's dynamically linked

I can tinker around with this and see what works. As for this and the tagging however, I might not be able to get to them super soon— any bandwidth you can spare with guidance is definitely appreciated it as I'm a bit heads down on a completely separate project. Thanks again for following up and getting this merged!

[BobyMCbobs]

Extra thing about the CI for the container build: when you tag a release, is it set up to use that tag in the container image?

I don't think so. Not sure how to do that (do you?). I assumed that it could just push the docker image as "latest" every time — but now that I think of it, it should have a tagged version too.

I would suggest something like this https://github.com/docker/build-push-action#handle-tags-and-labels

To make the container image smaller, it would be nice to have the final base as either alpine (minimal) or scratch (nothing). For scratch to be supported, the binary would need to be statically compiled. I think it's dynamically linked

I can tinker around with this and see what works. As for this and the tagging however, I might not be able to get to them super soon— any bandwidth you can spare with guidance is definitely appreciated it as I'm a bit heads down on a completely separate project. Thanks again for following up and getting this merged!

That would be wonderful, ideally we should be able to ship an alpine version for those who need a shell (useful in CI) and a scratch version for those who don't (useful for container build stage. example: https://gitlab.com/flattrack/flattrack/-/blob/21f94fe59e46ec4e6fee57a565a1dbba953cd2a2/build/docs.Dockerfile#L1).