Fail to build native function with -Dnative-image.docker-build

[Elegie]

Describe the bug Running mvn package -Pnative -Dnative-image.docker-build=true on the code generated by quarkus-amazon-lambda-archetype fails with:

docker: Error response from daemon: invalid volume specification: 'C:\Users\Elegie\springyme_quarkus\lambdatest\target\lambdatest-1.0-SNAPSHOT-native-image-source-jar:/project:z'.

Expected behavior The packaging works and a function.zip is available in the target.

Actual behavior The packaging fails with the following:

[INFO] [io.quarkus.deployment.QuarkusAugmentor] Beginning quarkus augmentation
[INFO] [io.quarkus.deployment.pkg.steps.JarResultBuildStep] Building native image source jar: C:\Users\Elegie\springyme_quarkus\lambdatest\target\lambdatest-1.0-SNAPSHOT-native-image-source-jar\lambdatest-1.0-SNAPSHOT-runner.jar
[INFO] [io.quarkus.deployment.pkg.steps.NativeImageBuildStep] Building native image from C:\Users\Elegie\springyme_quarkus\lambdatest\target\lambdatest-1.0-SNAPSHOT-native-image-source-jar\lambdatest-1.0-SNAPSHOT-runner.jar
[INFO] [io.quarkus.deployment.pkg.steps.NativeImageBuildStep] Running Quarkus native-image plugin on OpenJDK 64-Bit Server VM
[INFO] [io.quarkus.deployment.pkg.steps.NativeImageBuildStep] docker run -v C:\Users\Elegie\springyme_quarkus\lambdatest\target\lambdatest-1.0-SNAPSHOT-native-image-source-jar:/project:z --rm quay.io/quarkus/ubi-quarkus-native-image:19.2.1 -J-Djava.util.logging.manager=org.jboss.logmanager.LogManager --initialize-at-build-time= -H:InitialCollectionPolicy=com.oracle.svm.core.genscavenge.CollectionPolicy$BySpaceAndTime -jar lambdatest-1.0-SNAPSHOT-runner.jar -J-Djava.util.concurrent.ForkJoinPool.common.parallelism=1 -H:FallbackThreshold=0 -H:+ReportExceptionStackTraces -H:-AddAllCharsets -H:EnableURLProtocols=http -H:-JNI -H:-UseServiceLoaderFeature -H:+StackTrace lambdatest-1.0-SNAPSHOT-runner
docker: Error response from daemon: invalid volume specification: 'C:\Users\Elegie\springyme_quarkus\lambdatest\target\lambdatest-1.0-SNAPSHOT-native-image-source-jar:/project:z'.

To Reproduce Steps to reproduce the behavior:

1. mvn archetype:generate -DarchetypeGroupId=io.quarkus -DarchetypeArtifactId=quarkus-amazon-lambda-archetype -DarchetypeVersion=1.0.0.CR1
2. Go the app folder, then mvn package -Pnative -Dnative-image.docker-build=true.

Environment (please complete the following information):

• Output of uname -a or ver: Microsoft Windows [version 10.0.18362.418]
• Output of java -version: openjdk version "11.0.1" 2018-10-16 OpenJDK Runtime Environment 18.9 (build 11.0.1+13) OpenJDK 64-Bit Server VM 18.9 (build 11.0.1+13, mixed mode)

Additional context I have Windows Family, which requires me to use the Docker Toolbox. The problem seems to come from the way volumes are defined in the command line, with the Windows format (C:\...\) being wrong, but unfortunately I have no idea on how to fix it (I'm beginner-level at Docker).

[machi1990]

Hi @Elegie thanks for reporting this. I do not have a windows PC with me, so I'll like your help on this figuring out what's going on.

Can you make sure that the C drive is shared on your Docker Toolbox and try building again or simply running this command if you still have the build artefacts

docker run -v C:/Users/Elegie/springyme_quarkus/lambdatest/target/lambdatest-1.0-SNAPSHOT-native-image-source-jar:/project:z --rm quay.io/quarkus/ubi-quarkus-native-image:19.2.1 -J-Djava.util.logging.manager=org.jboss.logmanager.LogManager --initialize-at-build-time= -H:InitialCollectionPolicy=com.oracle.svm.core.genscavenge.CollectionPolicy$BySpaceAndTime -jar lambdatest-1.0-SNAPSHOT-runner.jar -J-Djava.util.concurrent.ForkJoinPool.common.parallelism=1 -H:FallbackThreshold=0 -H:+ReportExceptionStackTraces -H:-AddAllCharsets -H:EnableURLProtocols=http -H:-JNI -H:-UseServiceLoaderFeature -H:+StackTrace lambdatest-1.0-SNAPSHOT-runner

from your project dir?

[Elegie]

Hi @machi1990, thank you for your answer. Unfortunately it yields the same result.

C:\Users\Elegie\springyme_quarkus\lambdatest>docker run -v C:/Users/Elegie/springyme_quarkus/lambdatest/target/lambdatest-1.0-SNAPSHOT-native-image-source-jar:/project:z --rm quay.io/quarkus/ubi-quarkus-native-image:19.2.1 -J-Djava.util.logging.manager=org.jboss.logmanager.LogManager --initialize-at-build-time= -H:InitialCollectionPolicy=com.oracle.svm.core.genscavenge.CollectionPolicy$BySpaceAndTime -jar lambdatest-1.0-SNAPSHOT-runner.jar -J-Djava.util.concurrent.ForkJoinPool.common.parallelism=1 -H:FallbackThreshold=0 -H:+ReportExceptionStackTraces -H:-AddAllCharsets -H:EnableURLProtocols=http -H:-JNI -H:-UseServiceLoaderFeature -H:+StackTrace lambdatest-1.0-SNAPSHOT-runner
docker: Error response from daemon: invalid volume specification: 'C:/Users/Elegie/springyme_quarkus/lambdatest/target/lambdatest-1.0-SNAPSHOT-native-image-source-jar:/project:z'.
See 'docker run --help'.

Just so that you can be sure, I have attached a screenshot of the virtualbox configuration (https://imgur.com/a/yQa5Qda). By default, the directory was already shared (it is located in my user directory for that very reason), but I have added the sharing on the C drive, re-started the VM and re-done the test. I also tested it in another shared directory (D:/docker). All to no avail - same result.

I've tried and changed the path, as follows:

C:\Users\Elegie\springyme_quarkus\lambdatest>docker run -v /C/Users/Elegie/springyme_quarkus/lambdatest/target/lambdatest-1.0-SNAPSHOT-native-image-source-jar:/project:z --rm quay.io/quarkus/ubi-quarkus-native-image:19.2.1 -J-Djava.util.logging.manager=org.jboss.logmanager.LogManager --initialize-at-build-time= -H:InitialCollectionPolicy=com.oracle.svm.core.genscavenge.CollectionPolicy$BySpaceAndTime -jar lambdatest-1.0-SNAPSHOT-runner.jar -J-Djava.util.concurrent.ForkJoinPool.common.parallelism=1 -H:FallbackThreshold=0 -H:+ReportExceptionStackTraces -H:-AddAllCharsets -H:EnableURLProtocols=http -H:-JNI -H:-UseServiceLoaderFeature -H:+StackTrace lambdatest-1.0-SNAPSHOT-runner
Error: Invalid Path entry lambdatest-1.0-SNAPSHOT-runner.jar
Caused by: java.nio.file.NoSuchFileException: /project/lambdatest-1.0-SNAPSHOT-runner.jar

The result is different (the same output as if I remove the -v option completely). I don't know if this is relevant to you. I'm sorry my Docker level is so low.

[machi1990]

Hi @Elegie thanks for reporting back. I honestly do not have further ideas as I suspect it a docker related problem. Just out of curiosity, you mentioned restarting a VM, what kind of setup do you have ? are you using the Docker Desktop for windows ?

[Elegie]

Hi @machi1990, thanks for your comment! Docker Desktop for Windows only works for Windows Professional or Enterprise editions, not the other personal Windows editions, like the one I own. I have to use the legacy Docker Toolbox, which installs and uses Oracle Virtualbox.

I've created the VM using with the following command: docker-machine create -d virtualbox --virtualbox-cpu-count=2 --virtualbox-memory=4096 --virtualbox-disk-size=25000 default.

I don't know whether so-called "guest additions" are up, but when I log on using docker-machine ssh, I can see the mounted directories and create files into them (which then appear Windows Explorer). I think the issue lies with the way Windows paths are written in the plugin (apparently the /c/... and //c/ syntaxes should work)?

I suppose I could in fine set up a dev station myself in a VM (e.g. Java, Docker and Maven) and try building the native image there, but I'd really prefer using the -Dnative-image.docker-build=true option provided by the Quarkus plugin, which looks pretty neat.

[machi1990]

Interesting...

I don't know whether so-called "guest additions" are up, but when I log on using docker-machine ssh, I can see the mounted directories and create files into them (which then appear Windows Explorer). I think the issue lies with the way Windows paths are written in the plugin (apparently the /c/... and //c/ syntaxes should work)?

Yes, I initially thought it is the paths issue.

So we have already tried the command below without success,

docker run -v C:/Users/Elegie/springyme_quarkus/lambdatest/target/lambdatest-1.0-SNAPSHOT-native-image-source-jar:/project:z --rm quay.io/quarkus/ubi-quarkus-native-image:19.2.1 -J-Djava.util.logging.manager=org.jboss.logmanager.LogManager --initialize-at-build-time= -H:InitialCollectionPolicy=com.oracle.svm.core.genscavenge.CollectionPolicy$BySpaceAndTime -jar lambdatest-1.0-SNAPSHOT-runner.jar -J-Djava.util.concurrent.ForkJoinPool.common.parallelism=1 -H:FallbackThreshold=0 -H:+ReportExceptionStackTraces -H:-AddAllCharsets -H:EnableURLProtocols=http -H:-JNI -H:-UseServiceLoaderFeature -H:+StackTrace lambdatest-1.0-SNAPSHOT-runner

Does changing the above command so that the mounted path corresponds to use one of the suggested ways from the article you shared solves anything? If so, maybe it is something that can be fixed in the plugin or a behaviour that we can document in the guides.

Thanks for your help in this @Elegie.

[Elegie]

Hi @machi1990, thank you for your answer. The new syntax works:

C:\Users\Elegie\springyme_quarkus\lambdatest>docker run -v /c/Users/Elegie/springyme_quarkus/lambdatest/target/lambdatest-1.0-SNAPSHOT-native-image-source-jar:/project:z --rm quay.io/quarkus/ubi-quarkus-native-image:19.2.1 -J-Djava.util.logging.manager=org.jboss.logmanager.LogManager --initialize-at-build-time= -H:InitialCollectionPolicy=com.oracle.svm.core.genscavenge.CollectionPolicy$BySpaceAndTime -jar lambdatest-1.0-SNAPSHOT-runner.jar -J-Djava.util.concurrent.ForkJoinPool.common.parallelism=1 -H:FallbackThreshold=0 -H:+ReportExceptionStackTraces -H:-AddAllCharsets -H:EnableURLProtocols=http -H:-JNI -H:-UseServiceLoaderFeature -H:+StackTrace lambdatest-1.0-SNAPSHOT-runner
Build on Server(pid: 24, port: 45323)*
[lambdatest-1.0-SNAPSHOT-runner:24]    classlist:   5,804.38 ms
[lambdatest-1.0-SNAPSHOT-runner:24]        (cap):   1,864.88 ms
[lambdatest-1.0-SNAPSHOT-runner:24]        setup:   4,491.36 ms
06:44:00,190 INFO  [org.jbo.threads] JBoss Threads version 3.0.0.Final
[lambdatest-1.0-SNAPSHOT-runner:24]   (typeflow):  26,943.66 ms
[lambdatest-1.0-SNAPSHOT-runner:24]    (objects):  13,198.37 ms
[lambdatest-1.0-SNAPSHOT-runner:24]   (features):     397.16 ms
[lambdatest-1.0-SNAPSHOT-runner:24]     analysis:  41,396.84 ms
[lambdatest-1.0-SNAPSHOT-runner:24]     (clinit):     674.37 ms
[lambdatest-1.0-SNAPSHOT-runner:24]     universe:   1,788.87 ms
[lambdatest-1.0-SNAPSHOT-runner:24]      (parse):   6,989.14 ms
[lambdatest-1.0-SNAPSHOT-runner:24]     (inline):   8,799.90 ms
[lambdatest-1.0-SNAPSHOT-runner:24]    (compile):  46,433.42 ms
[lambdatest-1.0-SNAPSHOT-runner:24]      compile:  64,016.10 ms
[lambdatest-1.0-SNAPSHOT-runner:24]        image:   4,138.87 ms
[lambdatest-1.0-SNAPSHOT-runner:24]        write:     752.49 ms
[lambdatest-1.0-SNAPSHOT-runner:24]      [total]: 122,644.68 ms

There is a catch though. The mounted path must be /c/ in lowercase, and not uppercase. I had tried the uppercase version yesterday (in my first comment), but it failed with a NoSuchFileException and I didn't ponder why at the time. Here are the two commands I ran in the VM itself this morning (the first one fails, the second one succeeds):

docker@default:/c/Users/Elegie/springyme_quarkus/lambdatest/target$ ls /C/Users/Elegie/springyme_quarkus/lambdatest/target/lambdatest-1.0-SNAPSHOT-native-image-source-jar
ls: /C/Users/Elegie/springyme_quarkus/lambdatest/target/lambdatest-1.0-SNAPSHOT-native-image-source-jar: No such file or directory

docker@default:/c/Users/Elegie/springyme_quarkus/lambdatest/target$ ls /c/Users/Elegie/springyme_quarkus/lambdatest/target/lambdatest-1.0-SNAPSHOT-native-image-source-jar
lambdatest-1.0-SNAPSHOT-runner.jar  lib

I hope this helps.

[machi1990]

Hi @machi1990, thank you for your answer. The new syntax works:

Good to hear.

There is a catch though. The mounted path must be /c/ in lowercase, and not uppercase. I had tried the uppercase version yesterday (in my first comment), but it failed with a NoSuchFileException and I didn't ponder why at the time. Here are the two commands I ran in the VM itself this morning (the first one fails, the second one succeeds):

docker@default:/c/Users/Elegie/springyme_quarkus/lambdatest/target$ ls /C/Users/Elegie/springyme_quarkus/lambdatest/target/lambdatest-1.0-SNAPSHOT-native-image-source-jar
ls: /C/Users/Elegie/springyme_quarkus/lambdatest/target/lambdatest-1.0-SNAPSHOT-native-image-source-jar: No such file or directory

docker@default:/c/Users/Elegie/springyme_quarkus/lambdatest/target$ ls /c/Users/Elegie/springyme_quarkus/lambdatest/target/lambdatest-1.0-SNAPSHOT-native-image-source-jar
lambdatest-1.0-SNAPSHOT-runner.jar  lib

I hope this helps.

@Elegie Good to hear and thanks for being of much help along the way. When fixing this in our side, we have to make sure that the proposed command also works when using Docker Desktop for Windows. Having said that, would you like to contribute the fix? It should be here https://github.com/quarkusio/quarkus/blob/d3bedb554ee7a7def9f0c342052219d0bfa4dfcd/core/deployment/src/main/java/io/quarkus/deployment/pkg/steps/NativeImageBuildStep.java#L84-L85

[Elegie]

Hi @machi1990, thanks for your kind comment. Unfortunately, contributing a working fix for the issue is probably beyond my capabilities.

Translating the absolute path is fine I suppose (see a proposal below). However, I don't know if Docker for Windows would be able to interpret such a path, as I only own a Windows with Docker Toolbox. If it does, then the simple path translation should indeed suffice. However, if it does not, things might become ugly.

The original problem seems to lie in which VM technology is being used. Docker suggests that one should install either Docker Toolbox or Docker For Windows depending on what Windows Edition is available - but nothing actually prevents a user from using the Docker Toolbox on a Windows professional Edition.

In such cases, I have no idea on how to detect which VM technology is being used, thus which scheme should be used. I can only think of inferior approaches, such as letting the user specify which version to use as a plugin parameter (e.g. toolbox=true), or try/catching the mount, falling back on a translated path if Docker chokes on the original absolute path.

In any case, here's a proposal for a path-translating function. It works with the test path, and also defends against paths like "c", "c:", "c:/".

    private String toUnixStyle(String windowsStyle) {
        String translated = windowsStyle.replace('\\', '/');
        Pattern p = Pattern.compile("^(\\w)(?:$|:(/)?(.*))");
        Matcher m = p.matcher(translated);
        if (m.matches()) {
            String slash = Optional.ofNullable(m.group(2)).orElse("/");
            String path = Optional.ofNullable(m.group(3)).orElse("");
            return "//" + m.group(1).toLowerCase() + slash + path;
        }
        return translated;
    }

I use the "//" prefix rather than the "/" prefix, it works equally well, plus it's supposed to be more generic.

Hope this helps.

[Elegie]

Hi again @machi1990. To give you more feedback, I have pulled the sources, made the change, recompiled, ran the tests, installed and tried the updated plugin, and this time it worked and generated a function.zip file. I cannot deploy the function to verify whether it works, though (I haven't properly set up my AWS account yet).

Towards the end, the console still prints an error, I don't know if it is relevant.

[INFO] --- maven-assembly-plugin:3.1.0:single (zip-assembly) @ quarkus-lambda-native ---
[INFO] Reading assembly descriptor: src/assembly/zip.xml
[ERROR] OS=Windows and the assembly descriptor contains a *nix-specific root-relative-reference (starting with slash) /
[INFO] Building zip: C:\Users\Elegie\springyme_quarkus\quarkus-lambda-native\target\function.zip
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------

[machi1990]

Hi again @machi1990. To give you more feedback, I have pulled the sources, made the change, recompiled, ran the tests, installed and tried the updated plugin, and this time it worked and generated a function.zip file. I cannot deploy the function to verify whether it works, though (I haven't properly set up my AWS account yet).

Towards the end, the console still prints an error, I don't know if it is relevant.

[INFO] --- maven-assembly-plugin:3.1.0:single (zip-assembly) @ quarkus-lambda-native ---
[INFO] Reading assembly descriptor: src/assembly/zip.xml
[ERROR] OS=Windows and the assembly descriptor contains a *nix-specific root-relative-reference (starting with slash) /
[INFO] Building zip: C:\Users\Elegie\springyme_quarkus\quarkus-lambda-native\target\function.zip
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------

Hi @Elegie thanks for the efforts and really good progress. At this point I'd say the hardest part is done, you can feel free to open a PR (or even a draft one) with your working solution. It will be easier for the team to have a look at it and providing valuable feedbacks - including running the tests for Docker Desktop for Windows (I would have done so if I had a Windows machine).

Again thanks a lot for investigating this one and on coming up with a working solution for Docker Toolbox. Looking forward to your PR ;-)

[Elegie]

Hi @machi1990, thanks for your comment. I'll be opening a PR by the end of the week then, if that's alright. Cheers.

[machi1990]

You are welcome. Yeah sure, that's totally fine, thanks.

[oztimpower]

@Elegie the Amazon Lambda docs have been updated in #5377 to show you how to test local with the SAM CLI which runs in Docker (given your comment about not having an AWS Account)

I use Gradle so have to hand construct the function.zip, so I am curious if this will work for you, and yes also on Mac :)

[Elegie]

Hi @oztimpower, thank you for updating the doc! The local invocation worked right out of the box (Windows 10, Home Edition, legacy Docker Toolbox).

C:\Users\Elegie\springyme_quarkus\lambdatest>sam local invoke --template sam.native.yml --event payload.json
Invoking not.used.in.provided.runtime (provided)
2019-11-13 22:43:10 Found credentials in shared credentials file: ~/.aws/credentials
Decompressing C:\Users\Elegie\springyme_quarkus\lambdatest\target\function.zip

Fetching lambci/lambda:provided Docker container image......
Mounting C:\Users\Elegie\AppData\Local\Temp\tmp4pkc1cmb as /var/task:ro,delegated inside runtime container
2019-11-13 21:43:12,095 INFO  [io.quarkus] (main) lambdatest 1.0-SNAPSHOT (running on Quarkus 999-SNAPSHOT) started in 0.114s.
2019-11-13 21:43:12,111 INFO  [io.quarkus] (main) Profile prod activated.
2019-11-13 21:43:12,111 INFO  [io.quarkus] (main) Installed features: [amazon-lambda, cdi]
START RequestId: a0225029-cae1-1b62-0c1c-bcde852b6e77 Version: $LATEST
END RequestId: a0225029-cae1-1b62-0c1c-bcde852b6e77
REPORT RequestId: a0225029-cae1-1b62-0c1c-bcde852b6e77  Init Duration: 185.10 ms        Duration: 16.55 ms      Billed Duration: 100 ms                                                                                                     Memory Size: 128 MB      Max Memory Used: 14 MB
{"result":"hello Bill","requestId":"a0225029-cae1-1b62-0c1c-bcde852b6e77"}

[gsmet]

@Elegie so can we consider this issue resolved for you?

[Elegie]

@gsmet @machi1990 Hi, I've provided a pull request for the fix.

Unfortunately I could only test on Windows 10 Home Edition with Docker Toolbox, as I don't have any other environments available. Additional regression testing might therefore be necessary, for other target environments (Windows 10 with Docker For Windows, Mac/Linux).

Running mvn clean package -Dnative=true -Dnative-image.docker-build=true and verifying a function/zip is properly generated should be enough I think.

Hope this helps.