search

bmuschko / gradle-docker-plugin

Gradle plugin for managing Docker images and containers.
https://bmuschko.github.io/gradle-docker-plugin/current/user-guide/
Apache License 2.0
1.23k stars 361 forks source link

Docker file not found when using custom `dockerFile` #772

Closed sschuberth closed 5 years ago

sschuberth commented 5 years ago

Expected Behavior

When setting a custom dockerFile for a DockerBuildImage task like here I'd expect the Docker file to be found.

Current Behavior

The Docker file is not found and I get

Error during callback
com.github.dockerjava.api.exception.InternalServerErrorException: {"message":"Cannot locate specified Dockerfile: /home/seschube/Development/HERE/oss/tools/review-toolkit/cli/Dockerfile-to-run-ORT"}

        at com.github.dockerjava.jaxrs.filter.ResponseStatusExceptionFilter.filter(ResponseStatusExceptionFilter.java:52)
        at com.bmuschko.gradle.docker.shaded.org.glassfish.jersey.client.ClientFilteringStages$ResponseFilterStage.apply(ClientFilteringStages.java:133)
        at com.bmuschko.gradle.docker.shaded.org.glassfish.jersey.client.ClientFilteringStages$ResponseFilterStage.apply(ClientFilteringStages.java:121)
        at com.bmuschko.gradle.docker.shaded.org.glassfish.jersey.process.internal.Stages.process(Stages.java:171)
        at com.bmuschko.gradle.docker.shaded.org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:283)
        at com.bmuschko.gradle.docker.shaded.org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$1(JerseyInvocation.java:767)
        at com.bmuschko.gradle.docker.shaded.org.glassfish.jersey.internal.Errors.process(Errors.java:316)
        at com.bmuschko.gradle.docker.shaded.org.glassfish.jersey.internal.Errors.process(Errors.java:298)
        at com.bmuschko.gradle.docker.shaded.org.glassfish.jersey.internal.Errors.process(Errors.java:229)
        at com.bmuschko.gradle.docker.shaded.org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:414)
        at com.bmuschko.gradle.docker.shaded.org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:765)
        at com.bmuschko.gradle.docker.shaded.org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:456)
        at com.bmuschko.gradle.docker.shaded.org.glassfish.jersey.client.JerseyInvocation$Builder.post(JerseyInvocation.java:357)
        at com.github.dockerjava.jaxrs.async.POSTCallbackNotifier.response(POSTCallbackNotifier.java:29)
        at com.github.dockerjava.jaxrs.async.AbstractCallbackNotifier.call(AbstractCallbackNotifier.java:50)
        at com.github.dockerjava.jaxrs.async.AbstractCallbackNotifier.call(AbstractCallbackNotifier.java:24)

Although:

$ ls  /home/seschube/Development/HERE/oss/tools/review-toolkit/cli/Dockerfile-to-run-ORT
/home/seschube/Development/HERE/oss/tools/review-toolkit/cli/Dockerfile-to-run-ORT

Context

I'm trying to use a custom Docker file which has not the default name of "Dockerfile".

Steps to Reproduce (for bugs)

$ git clone -b docker-debug https://github.com/heremaps/oss-review-toolkit.git $ ./gradlew cli:dockerBuildImage

Your Environment

Ubuntu 18.04, 64-bit

bmuschko commented 5 years ago

The issue is probably to be found somewhere here: https://github.com/bmuschko/gradle-docker-plugin/blob/master/src/main/groovy/com/bmuschko/gradle/docker/tasks/image/DockerBuildImage.groovy#L131-L135. My guess is that you don't need to set a base directory if you provide a Dockerfile already.

Would you be interested in providing a fix?

sschuberth commented 5 years ago

I'll give it a try. What's an easy way to test the plugin from source code?

bmuschko commented 5 years ago

Thanks! You will want to write a functional test. I'd say add a test case for this to DockerBuildImageFunctionalTest.groovy.

sschuberth commented 5 years ago

At least from IntelliJ I cannot run the tests in DockerBuildImageFunctionalTest as I'm getting:

org.gradle.testkit.runner.InvalidPluginMetadataException: Test runtime classpath does not contain plugin metadata file 'plugin-under-test-metadata.properties'

Is some special setup required? I just imported the build.gradle.kts file to create a "project from existing sources".

Edit: Nevermind, found an answer here.

bmuschko commented 5 years ago

You will be able to find all the relevant information on testing Gradle plugins in this guide (including IDE integration).

sharavs commented 5 years ago

I'm having some problems using the DockerBuildImage plugin except my docker file is simply named 'Dockerfile' and I've tried not specifying a Dockerfile and also specifying it but neither way seems to work. It's a little strange and I can't figure out why. I get this:

{"message":"Cannot locate specified Dockerfile: /root/sharav/INT/Dockerfile"}

I've tried to test in a global build context and a local build function only context and seems to fail in both. The Dockerfile does indeed exist in the directory. Maybe I'm overlooking something stupid but any suggestions?

sschuberth commented 5 years ago

My (brief) investigation so far indicates that the real issue is in fact in the underlying docker-java code (where this message comes from). Something is weird about the behavior when (not) setting baseDirectory, probably somewhere around https://github.com/docker-java/docker-java/blob/de0c1d4b352b81c2f58aa4758eb9c431e4b05346/src/main/java/com/github/dockerjava/core/command/BuildImageCmdImpl.java#L333. But I didn't yet have time to look deeper.

I'm also slight confused by the naming. My understanding is that in docker-java's code the "baseDirectory" is what the Docker CLI's build command calls "context". If a Dockerfile does not use an COPY or related command, specifying the context is optional, and so should be specifying "baseDirectory". However, it seems the code never allows "baseDirectory" to be unset, and will default to the parent directory of the Dockerfile, and maybe the problem is related to that.

sharavs commented 5 years ago

Ok, yes that underlying docker java code definitely provides some context. Well the Dockerfile doesn't and that is why at first I thought not specifying should work, and when it didn't I specified and still didn't work which boggled me. After some troubleshooting still nothing seemed to work but let me try to see if there's a workaround with this "baseDirectory" issue. Thanks for your help and if you happen to dig more, updates would be appreciated :smile:

llaforest commented 5 years ago

Hello I'm having the same problem here. Took me time to find this thread. I'm not sure to follow the baseDirectory logic. I'm not setting any. I do not mind adding something to my dockerfile if needs be.

So is there a workaround to specify a docker file that works or not?

bmuschko commented 5 years ago

Looking at the code again I think we should probably get rid of the property inputDir. The base directory can always be derived of the parent directory of the Dockerfile aka the value of the property dockerFile. I am not sure quite sure why there even is a differentiation between base directory and Dockerfile in Docker Java.

llaforest commented 5 years ago

Yes I agree. My test shows that when a Dockerfile sits in the inputDir, it will be taken. Also I can specify a dockerFile but the path specified is not absolute, it must be relative to the input dir specified. So lets say I set my inputDir to $projectDir and my dockerFile to "docker/Dockerfile" it works.

One other problem I found is that since it uses any Dockerfile that sits in the inputDir, if the Dockerfile changes, the up-to-date is not invalidated because the @input dockerFile was never specified therefore not part of the cache key.

cdancy commented 5 years ago

@bmuschko might be something to do for the 5.0 release?

bmuschko commented 5 years ago

@cdancy Probably best to make this a 5.0.0 change. We'd be happy to accept a PR for this change if anyone want to contribute.

llaforest commented 5 years ago

@bmuschko if we specify only dockerFile, we must make sure the inputDir is still set even if it is private so that the cache key is invalidated if any file of the inputDir changed. If we just set the baseDirectory of buildImageCmd by using the dockerFile parent dir, the folder won't be scanned for changes...

sschuberth commented 5 years ago

I am not sure quite sure why there even is a differentiation between base directory and Dockerfile in Docker Java.

Isn't the "base directory" what Docker CLI calls the "context" when building an image, i.e. the directory COPY and similar commands operate relative to? If so, it makes sense to set the default base directory to the parent directory of the Dockerfile, but you certainly also want to be able to set the base directory / context to some other directory.

Edit: https://github.com/bmuschko/gradle-docker-plugin/blob/96acbb9b5646c3f9df61f7b8d881ddfab3bd0120/src/main/groovy/com/bmuschko/gradle/docker/tasks/image/DockerBuildImage.groovy#L45 https://github.com/bmuschko/gradle-docker-plugin/blob/96acbb9b5646c3f9df61f7b8d881ddfab3bd0120/src/main/groovy/com/bmuschko/gradle/docker/tasks/image/DockerBuildImage.groovy#L171 seems to confirm that what Docker CLI calls "context" is called "base directory" in Docker Java and "input Directory" in this Docker Plugin.

llaforest commented 5 years ago

A good way to overcome all of this without requiring change is to:

sschuberth commented 5 years ago

I agree that's a work-around, but I wouldn't call it a "good way" as there's too many things to remember. A good API avoids accidental misuse.

llaforest commented 5 years ago

Yeah it's even simpler than what I said.

The only misuse here would be that your Dockerfile is outside inputDir tree. This is not allowed.

bmuschko commented 5 years ago

Any chance someone wants contribute the test cases for all the use cases that have been laid out? We can come up with a solution from there.

sschuberth commented 5 years ago

I stopped using the plugin, so I'm out...

bmuschko commented 5 years ago

Closing this issue for now. If someone wants to step up, please go for it.