Closed michalrames closed 4 years ago
Thanks for the report.
The problem appears to be that BitBucket considers the POST
request that is creating a container to have an invalid Content-Length
:
time="2020-08-09T11:30:12.607341026Z" level=error msg="AuthZRequest for POST /v1.24/containers/create returned error: authorization denied by plugin pipelines: Invalid content length provided"
As far as I can tell, BitBucket's problem is that the POST
request is sent using Transfer-Encoding: chunked
as the content length isn't known. There is, therefore, no Content-Length
header in the request. This is what I assume causes BitBucket to consider the Content-Length
to be invalid but we can't be certain as, AFAIK, BitBucket Pipelines isn't open source.
In this specific case, we could supply a Content-Length
with the container creation POST request. However, it would then fail for the subsequent PUT
that uploads the container's content.
I think this will have to be raised with Atlassian. Assuming my analysis above is correct, ideally they would support chunked transfer encoding as Docker usually does. Failing that, if my analysis is incorrect or they're unable to support chunked transfer encoding and they can provide detailed information about the Docker-related restrictions, we can re-open this and take another look.
Thanks for providing feedback about this issue.
For anyone else coming here, I'd like to add some more info. There is a related public bug issue that can be found here: https://jira.atlassian.com/browse/BCLOUD-15844
For now, I added jib gradle plugin and replaced bootBuildImage in my pipeline with jibDockerBuild which can create the desired image. Note that this image is roughly 20% smaller compared to the image generated by bootBuildImage and I haven't figured out yet why, but the tests are passing, so it seems to be an alternative..
Today the bitbucket team made an update explaining why they can't support Transfer-Encoding Chunked see here
In short, this is not an issue with bitbucket in particular, but with how dockers authorization plugins work and they can't do much about it currently, although they would appreciate an update by docker, since other projects are experience similar problems. Other environments with dockers authZ plugin configured are affected as well.
In this specific case, we could supply a Content-Length with the container creation POST request. However, it would then fail for the subsequent PUT that uploads the container's content.
The plugin only applies to content-type application/json. The subsequent PUT should be content-type application/x-tar, so the plugin wouldn't intercept and reject a chunked transfer there.
Could you reopen this issue and update the POST request, since a change to dockers authZ plugin seems unlikely currently?
From the linked duplicate
It's not clear to me if this is an RFC-style should where it's something that's recommended but optional or if it's a requirement a must in RFC terms). It would appear that the local Docker daemon treats it as optional (but perhaps recommended) whereas BitBucket's pipelines treat it as a requirement. Either way, it's probably worth us complying with the API's recommendations if we can.
I guess the answer is, it is a requirement that applies as soon as docker plugin infrastructure is involved 😄
As far as I got, for those who is searching for a solution -> it is fixed in Spring Boot 2.3.5, so the solution is upgrading to this version
As far as I got, for those who is searching for a solution -> it is fixed in Spring Boot 2.3.5, so the solution is upgrading to this version
Have tried 2.3.5 and 2.4.0 - still get the error. Hmm...
@andrey-bushik Is the error you are getting exactly the same as shown in the original issue? Can you show the output from the bitbucket docker logs and the Spring Boot plugin?
Same error using Gradle 6.6.1 and spring-boot 2.4.0. ./gradlew bootBuildImage
works fine in local but fails in bitbucket-pipelines.
Caused by: org.springframework.boot.buildpack.platform.docker.transport.DockerEngineException: Docker API call to 'localhost:2375/v1.24/containers/create' failed with status code 403 "Forbidden"
at org.springframework.boot.buildpack.platform.docker.transport.HttpClientTransport.execute(HttpClientTransport.java:156)
at org.springframework.boot.buildpack.platform.docker.transport.HttpClientTransport.execute(HttpClientTransport.java:136)
at org.springframework.boot.buildpack.platform.docker.transport.HttpClientTransport.post(HttpClientTransport.java:108)
at org.springframework.boot.buildpack.platform.docker.DockerApi$ContainerApi.createContainer(DockerApi.java:302)
at org.springframework.boot.buildpack.platform.docker.DockerApi$ContainerApi.create(DockerApi.java:293)
at org.springframework.boot.buildpack.platform.build.Lifecycle.createContainer(Lifecycle.java:180)
at org.springframework.boot.buildpack.platform.build.Lifecycle.run(Lifecycle.java:160)
at org.springframework.boot.buildpack.platform.build.Lifecycle.execute(Lifecycle.java:119)
at org.springframework.boot.buildpack.platform.build.Builder.executeLifecycle(Builder.java:184)
at org.springframework.boot.buildpack.platform.build.Builder.build(Builder.java:103)
[...]
@agtraveso That may not be the same error as it's impossible to tell without also seeing the BitBucket Docker logs. Can you please share those logs as well?
@wilkinsona You are right, it seems to fail for other reason:
time="2020-12-10T08:31:34Z" level=info msg="Pipelines plugin request authorization." allowed=false method=POST plugin=pipelines uri=/v1.24/containers/create
time="2020-12-10T08:31:34.130350057Z" level=error msg="AuthZRequest for POST /v1.24/containers/create returned error: authorization denied by plugin pipelines: -v only supports $BITBUCKET_CLONE_DIR and its subdirectories"
I recently started a brand new spring-boot 2.4.0 project in bitbucket cloud and still see the same error
[ERROR] Failed to execute goal org.springframework.boot:spring-boot-maven-plugin:2.4.0:build-image (default-cli) on project esail-server: Execution default-cli of goal org.springframework.boot:spring-boot-maven-plugin:2.4.0:build-image failed: Docker API call to 'localhost:2375/v1.24/containers/create' failed with status code 403 "Forbidden" -> [Help 1]
After upgrading to Boot 2.4.1 same problem:
[ERROR] Failed to execute goal org.springframework.boot:spring-boot-maven-plugin:2.4.1:build-image (default-cli) on project esail-server: Execution default-cli of goal org.springframework.boot:spring-boot-maven-plugin:2.4.1:build-image failed: Docker API call to 'localhost:2375/v1.24/containers/create' failed with status code 403 "Forbidden" -> [Help 1]
Docker output says:
mkdir: can't create directory '/certs/ca': Read-only file system
genrsa: Can't open "/certs/ca/key.pem" for writing, No such file or directory
Can't open /certs/ca/key.pem for reading, No such file or directory
140701949046088:error:02001002:system library:fopen:No such file or directory:crypto/bio/bss_file.c:69:fopen('/certs/ca/key.pem','r')
140701949046088:error:2006D080:BIO routines:BIO_new_file:no such file:crypto/bio/bss_file.c:76:
unable to load Private Key
and
time="2020-12-14T08:42:47.501603085Z" level=error msg="AuthZRequest for HEAD /_ping returned error: authorization denied by plugin pipelines: "
@lehmannk Given the output from Docker, you aren't experiencing the same problem as the one that has been addressed by this issue. It also doesn't look to me like a problem caused by Spring Boot as it isn't responsible for creating the file system or ensuring that it's writable. I suspect this is a problem with your pipeline configuration.
I came across this: https://github.com/spotify/docker-client/pull/1021 over here https://community.atlassian.com/t5/Bitbucket-questions/create-container-authorization-denied-by-plugin-pipelines/qaq-p/731364 it may provide a way to resolve this issue. Which docker client is used by bootBuildImage?
@corneil The commit that closed this issue has addressed the same problem as spotify/docker-client#1021. It sets the Content-Length
header which prevents the use of chunked transfer encoding.
@wilkinsona I created a simple project using 2.3.9 and the 403 still happens.
https://bitbucket.org/tsctech/demo-container
What does jibDockerBuild
do differently?
> Running creator
> Task :bootBuildImage FAILED
:bootBuildImage (Thread[Execution worker for ':' Thread 2,5,main]) completed. Took 12.207 secs.
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':bootBuildImage'.
> Docker API call to 'localhost:2375/v1.24/containers/create' failed with status code 403 "Forbidden"
* Try:
Run with --debug option to get more log output. Run with --scan to get full insights.
* Exception is:
org.gradle.api.tasks.TaskExecutionException: Execution failed for task ':bootBuildImage'.
at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.lambda$executeIfValid$3(ExecuteActionsTaskExecuter.java:186)
at org.gradle.internal.Try$Failure.ifSuccessfulOrElse(Try.java:268)
at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeIfValid(ExecuteActionsTaskExecuter.java:184)
at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.execute(ExecuteActionsTaskExecuter.java:173)
at org.gradle.api.internal.tasks.execution.CleanupStaleOutputsExecuter.execute(CleanupStaleOutputsExecuter.java:109)
at org.gradle.api.internal.tasks.execution.FinalizePropertiesTaskExecuter.execute(FinalizePropertiesTaskExecuter.java:46)
at org.gradle.api.internal.tasks.execution.ResolveTaskExecutionModeExecuter.execute(ResolveTaskExecutionModeExecuter.java:62)
at org.gradle.api.internal.tasks.execution.SkipTaskWithNoActionsExecuter.execute(SkipTaskWithNoActionsExecuter.java:57)
at org.gradle.api.internal.tasks.execution.SkipOnlyIfTaskExecuter.execute(SkipOnlyIfTaskExecuter.java:56)
at org.gradle.api.internal.tasks.execution.CatchExceptionTaskExecuter.execute(CatchExceptionTaskExecuter.java:36)
at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.executeTask(EventFiringTaskExecuter.java:77)
at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.call(EventFiringTaskExecuter.java:55)
at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.call(EventFiringTaskExecuter.java:52)
at org.gradle.internal.operations.DefaultBuildOperationRunner$CallableBuildOperationWorker.execute(DefaultBuildOperationRunner.java:200)
at org.gradle.internal.operations.DefaultBuildOperationRunner$CallableBuildOperationWorker.execute(DefaultBuildOperationRunner.java:195)
at org.gradle.internal.operations.DefaultBuildOperationRunner$3.execute(DefaultBuildOperationRunner.java:75)
at org.gradle.internal.operations.DefaultBuildOperationRunner$3.execute(DefaultBuildOperationRunner.java:68)
at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:153)
at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:68)
at org.gradle.internal.operations.DefaultBuildOperationRunner.call(DefaultBuildOperationRunner.java:62)
at org.gradle.internal.operations.DefaultBuildOperationExecutor.lambda$call$2(DefaultBuildOperationExecutor.java:76)
at org.gradle.internal.operations.UnmanagedBuildOperationWrapper.callWithUnmanagedSupport(UnmanagedBuildOperationWrapper.java:54)
at org.gradle.internal.operations.DefaultBuildOperationExecutor.call(DefaultBuildOperationExecutor.java:76)
at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter.execute(EventFiringTaskExecuter.java:52)
at org.gradle.execution.plan.LocalTaskNodeExecutor.execute(LocalTaskNodeExecutor.java:41)
at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$InvokeNodeExecutorsAction.execute(DefaultTaskExecutionGraph.java:411)
at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$InvokeNodeExecutorsAction.execute(DefaultTaskExecutionGraph.java:398)
at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareExecutionAction.execute(DefaultTaskExecutionGraph.java:391)
at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareExecutionAction.execute(DefaultTaskExecutionGraph.java:377)
at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.lambda$run$0(DefaultPlanExecutor.java:127)
at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.execute(DefaultPlanExecutor.java:191)
at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.executeNextNode(DefaultPlanExecutor.java:182)
at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.run(DefaultPlanExecutor.java:124)
at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:48)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:56)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: org.springframework.boot.buildpack.platform.docker.transport.DockerEngineException: Docker API call to 'localhost:2375/v1.24/containers/create' failed with status code 403 "Forbidden"
at org.springframework.boot.buildpack.platform.docker.transport.HttpClientTransport.execute(HttpClientTransport.java:135)
at org.springframework.boot.buildpack.platform.docker.transport.HttpClientTransport.execute(HttpClientTransport.java:122)
at org.springframework.boot.buildpack.platform.docker.transport.HttpClientTransport.post(HttpClientTransport.java:94)
at org.springframework.boot.buildpack.platform.docker.DockerApi$ContainerApi.createContainer(DockerApi.java:241)
at org.springframework.boot.buildpack.platform.docker.DockerApi$ContainerApi.create(DockerApi.java:232)
at org.springframework.boot.buildpack.platform.build.Lifecycle.createContainer(Lifecycle.java:166)
at org.springframework.boot.buildpack.platform.build.Lifecycle.run(Lifecycle.java:146)
at org.springframework.boot.buildpack.platform.build.Lifecycle.execute(Lifecycle.java:113)
at org.springframework.boot.buildpack.platform.build.Builder.executeLifecycle(Builder.java:122)
at org.springframework.boot.buildpack.platform.build.Builder.build(Builder.java:71)
at org.springframework.boot.gradle.tasks.bundling.BootBuildImage.buildImage(BootBuildImage.java:236)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.gradle.internal.reflect.JavaMethod.invoke(JavaMethod.java:104)
at org.gradle.api.internal.project.taskfactory.StandardTaskAction.doExecute(StandardTaskAction.java:58)
at org.gradle.api.internal.project.taskfactory.StandardTaskAction.execute(StandardTaskAction.java:51)
at org.gradle.api.internal.project.taskfactory.StandardTaskAction.execute(StandardTaskAction.java:29)
at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter$2.run(ExecuteActionsTaskExecuter.java:494)
at org.gradle.internal.operations.DefaultBuildOperationRunner$1.execute(DefaultBuildOperationRunner.java:29)
at org.gradle.internal.operations.DefaultBuildOperationRunner$1.execute(DefaultBuildOperationRunner.java:26)
at org.gradle.internal.operations.DefaultBuildOperationRunner$3.execute(DefaultBuildOperationRunner.java:75)
at org.gradle.internal.operations.DefaultBuildOperationRunner$3.execute(DefaultBuildOperationRunner.java:68)
at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:153)
at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:68)
at org.gradle.internal.operations.DefaultBuildOperationRunner.run(DefaultBuildOperationRunner.java:56)
at org.gradle.internal.operations.DefaultBuildOperationExecutor.lambda$run$1(DefaultBuildOperationExecutor.java:71)
at org.gradle.internal.operations.UnmanagedBuildOperationWrapper.runWithUnmanagedSupport(UnmanagedBuildOperationWrapper.java:45)
at org.gradle.internal.operations.DefaultBuildOperationExecutor.run(DefaultBuildOperationExecutor.java:71)
at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeAction(ExecuteActionsTaskExecuter.java:479)
at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeActions(ExecuteActionsTaskExecuter.java:462)
at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.access$400(ExecuteActionsTaskExecuter.java:105)
at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter$TaskExecution.executeWithPreviousOutputFiles(ExecuteActionsTaskExecuter.java:273)
at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter$TaskExecution.execute(ExecuteActionsTaskExecuter.java:251)
at org.gradle.internal.execution.steps.ExecuteStep.lambda$executeOperation$1(ExecuteStep.java:66)
at java.base/java.util.Optional.orElseGet(Optional.java:369)
at org.gradle.internal.execution.steps.ExecuteStep.executeOperation(ExecuteStep.java:66)
at org.gradle.internal.execution.steps.ExecuteStep.access$000(ExecuteStep.java:34)
at org.gradle.internal.execution.steps.ExecuteStep$1.call(ExecuteStep.java:47)
at org.gradle.internal.execution.steps.ExecuteStep$1.call(ExecuteStep.java:44)
at org.gradle.internal.operations.DefaultBuildOperationRunner$CallableBuildOperationWorker.execute(DefaultBuildOperationRunner.java:200)
at org.gradle.internal.operations.DefaultBuildOperationRunner$CallableBuildOperationWorker.execute(DefaultBuildOperationRunner.java:195)
at org.gradle.internal.operations.DefaultBuildOperationRunner$3.execute(DefaultBuildOperationRunner.java:75)
at org.gradle.internal.operations.DefaultBuildOperationRunner$3.execute(DefaultBuildOperationRunner.java:68)
at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:153)
at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:68)
at org.gradle.internal.operations.DefaultBuildOperationRunner.call(DefaultBuildOperationRunner.java:62)
at org.gradle.internal.operations.DefaultBuildOperationExecutor.lambda$call$2(DefaultBuildOperationExecutor.java:76)
at org.gradle.internal.operations.UnmanagedBuildOperationWrapper.callWithUnmanagedSupport(UnmanagedBuildOperationWrapper.java:54)
at org.gradle.internal.operations.DefaultBuildOperationExecutor.call(DefaultBuildOperationExecutor.java:76)
at org.gradle.internal.execution.steps.ExecuteStep.execute(ExecuteStep.java:44)
at org.gradle.internal.execution.steps.ExecuteStep.execute(ExecuteStep.java:34)
at org.gradle.internal.execution.steps.RemovePreviousOutputsStep.execute(RemovePreviousOutputsStep.java:72)
at org.gradle.internal.execution.steps.RemovePreviousOutputsStep.execute(RemovePreviousOutputsStep.java:42)
at org.gradle.internal.execution.steps.ResolveInputChangesStep.execute(ResolveInputChangesStep.java:53)
at org.gradle.internal.execution.steps.ResolveInputChangesStep.execute(ResolveInputChangesStep.java:39)
at org.gradle.internal.execution.steps.CancelExecutionStep.execute(CancelExecutionStep.java:44)
at org.gradle.internal.execution.steps.TimeoutStep.executeWithoutTimeout(TimeoutStep.java:77)
at org.gradle.internal.execution.steps.TimeoutStep.execute(TimeoutStep.java:58)
at org.gradle.internal.execution.steps.CreateOutputsStep.execute(CreateOutputsStep.java:54)
at org.gradle.internal.execution.steps.CreateOutputsStep.execute(CreateOutputsStep.java:32)
at org.gradle.internal.execution.steps.CaptureStateAfterExecutionStep.execute(CaptureStateAfterExecutionStep.java:57)
at org.gradle.internal.execution.steps.CaptureStateAfterExecutionStep.execute(CaptureStateAfterExecutionStep.java:38)
at org.gradle.internal.execution.steps.BroadcastChangingOutputsStep.execute(BroadcastChangingOutputsStep.java:63)
at org.gradle.internal.execution.steps.BroadcastChangingOutputsStep.execute(BroadcastChangingOutputsStep.java:30)
at org.gradle.internal.execution.steps.BuildCacheStep.executeWithoutCache(BuildCacheStep.java:176)
at org.gradle.internal.execution.steps.BuildCacheStep.execute(BuildCacheStep.java:76)
at org.gradle.internal.execution.steps.BuildCacheStep.execute(BuildCacheStep.java:47)
at org.gradle.internal.execution.steps.StoreExecutionStateStep.execute(StoreExecutionStateStep.java:43)
at org.gradle.internal.execution.steps.StoreExecutionStateStep.execute(StoreExecutionStateStep.java:32)
at org.gradle.internal.execution.steps.RecordOutputsStep.execute(RecordOutputsStep.java:39)
at org.gradle.internal.execution.steps.RecordOutputsStep.execute(RecordOutputsStep.java:25)
at org.gradle.internal.execution.steps.SkipUpToDateStep.executeBecause(SkipUpToDateStep.java:102)
at org.gradle.internal.execution.steps.SkipUpToDateStep.lambda$execute$0(SkipUpToDateStep.java:95)
at java.base/java.util.Optional.map(Optional.java:265)
at org.gradle.internal.execution.steps.SkipUpToDateStep.execute(SkipUpToDateStep.java:55)
at org.gradle.internal.execution.steps.SkipUpToDateStep.execute(SkipUpToDateStep.java:39)
at org.gradle.internal.execution.steps.ResolveChangesStep.execute(ResolveChangesStep.java:83)
at org.gradle.internal.execution.steps.ResolveChangesStep.execute(ResolveChangesStep.java:44)
at org.gradle.internal.execution.steps.legacy.MarkSnapshottingInputsFinishedStep.execute(MarkSnapshottingInputsFinishedStep.java:37)
at org.gradle.internal.execution.steps.legacy.MarkSnapshottingInputsFinishedStep.execute(MarkSnapshottingInputsFinishedStep.java:27)
at org.gradle.internal.execution.steps.ResolveCachingStateStep.execute(ResolveCachingStateStep.java:96)
at org.gradle.internal.execution.steps.ResolveCachingStateStep.execute(ResolveCachingStateStep.java:52)
at org.gradle.internal.execution.steps.CaptureStateBeforeExecutionStep.execute(CaptureStateBeforeExecutionStep.java:83)
at org.gradle.internal.execution.steps.CaptureStateBeforeExecutionStep.execute(CaptureStateBeforeExecutionStep.java:54)
at org.gradle.internal.execution.steps.ValidateStep.execute(ValidateStep.java:74)
at org.gradle.internal.execution.steps.SkipEmptyWorkStep.lambda$execute$2(SkipEmptyWorkStep.java:88)
at java.base/java.util.Optional.orElseGet(Optional.java:369)
at org.gradle.internal.execution.steps.SkipEmptyWorkStep.execute(SkipEmptyWorkStep.java:88)
at org.gradle.internal.execution.steps.SkipEmptyWorkStep.execute(SkipEmptyWorkStep.java:34)
at org.gradle.internal.execution.steps.legacy.MarkSnapshottingInputsStartedStep.execute(MarkSnapshottingInputsStartedStep.java:38)
at org.gradle.internal.execution.steps.LoadExecutionStateStep.execute(LoadExecutionStateStep.java:46)
at org.gradle.internal.execution.steps.LoadExecutionStateStep.execute(LoadExecutionStateStep.java:34)
at org.gradle.internal.execution.steps.AssignWorkspaceStep.lambda$execute$0(AssignWorkspaceStep.java:43)
at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter$TaskExecution$3.withWorkspace(ExecuteActionsTaskExecuter.java:286)
at org.gradle.internal.execution.steps.AssignWorkspaceStep.execute(AssignWorkspaceStep.java:43)
at org.gradle.internal.execution.steps.AssignWorkspaceStep.execute(AssignWorkspaceStep.java:33)
at org.gradle.internal.execution.steps.IdentityCacheStep.execute(IdentityCacheStep.java:40)
at org.gradle.internal.execution.steps.IdentityCacheStep.execute(IdentityCacheStep.java:30)
at org.gradle.internal.execution.steps.IdentifyStep.execute(IdentifyStep.java:54)
at org.gradle.internal.execution.steps.IdentifyStep.execute(IdentifyStep.java:40)
at org.gradle.internal.execution.impl.DefaultExecutionEngine.rebuild(DefaultExecutionEngine.java:46)
at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.lambda$executeIfValid$0(ExecuteActionsTaskExecuter.java:182)
at java.base/java.util.Optional.map(Optional.java:265)
at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeIfValid(ExecuteActionsTaskExecuter.java:182)
at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.execute(ExecuteActionsTaskExecuter.java:173)
at org.gradle.api.internal.tasks.execution.CleanupStaleOutputsExecuter.execute(CleanupStaleOutputsExecuter.java:109)
at org.gradle.api.internal.tasks.execution.FinalizePropertiesTaskExecuter.execute(FinalizePropertiesTaskExecuter.java:46)
at org.gradle.api.internal.tasks.execution.ResolveTaskExecutionModeExecuter.execute(ResolveTaskExecutionModeExecuter.java:62)
at org.gradle.api.internal.tasks.execution.SkipTaskWithNoActionsExecuter.execute(SkipTaskWithNoActionsExecuter.java:57)
6 actionable tasks: 6 executed
at org.gradle.api.internal.tasks.execution.SkipOnlyIfTaskExecuter.execute(SkipOnlyIfTaskExecuter.java:56)
at org.gradle.api.internal.tasks.execution.CatchExceptionTaskExecuter.execute(CatchExceptionTaskExecuter.java:36)
at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.executeTask(EventFiringTaskExecuter.java:77)
at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.call(EventFiringTaskExecuter.java:55)
at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.call(EventFiringTaskExecuter.java:52)
at org.gradle.internal.operations.DefaultBuildOperationRunner$CallableBuildOperationWorker.execute(DefaultBuildOperationRunner.java:200)
at org.gradle.internal.operations.DefaultBuildOperationRunner$CallableBuildOperationWorker.execute(DefaultBuildOperationRunner.java:195)
at org.gradle.internal.operations.DefaultBuildOperationRunner$3.execute(DefaultBuildOperationRunner.java:75)
at org.gradle.internal.operations.DefaultBuildOperationRunner$3.execute(DefaultBuildOperationRunner.java:68)
at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:153)
at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:68)
at org.gradle.internal.operations.DefaultBuildOperationRunner.call(DefaultBuildOperationRunner.java:62)
at org.gradle.internal.operations.DefaultBuildOperationExecutor.lambda$call$2(DefaultBuildOperationExecutor.java:76)
at org.gradle.internal.operations.UnmanagedBuildOperationWrapper.callWithUnmanagedSupport(UnmanagedBuildOperationWrapper.java:54)
at org.gradle.internal.operations.DefaultBuildOperationExecutor.call(DefaultBuildOperationExecutor.java:76)
at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter.execute(EventFiringTaskExecuter.java:52)
at org.gradle.execution.plan.LocalTaskNodeExecutor.execute(LocalTaskNodeExecutor.java:41)
at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$InvokeNodeExecutorsAction.execute(DefaultTaskExecutionGraph.java:411)
at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$InvokeNodeExecutorsAction.execute(DefaultTaskExecutionGraph.java:398)
at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareExecutionAction.execute(DefaultTaskExecutionGraph.java:391)
at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareExecutionAction.execute(DefaultTaskExecutionGraph.java:377)
at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.lambda$run$0(DefaultPlanExecutor.java:127)
at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.execute(DefaultPlanExecutor.java:191)
at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.executeNextNode(DefaultPlanExecutor.java:182)
at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.run(DefaultPlanExecutor.java:124)
at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:48)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:56)
at java.base/java.lang.Thread.run(Thread.java:834)
time="2021-03-16T21:00:15Z" level=info msg="Pipelines plugin request authorization." allowed=true method=POST plugin=pipelines uri="/v1.24/images/create?fromImage=gcr.io%2Fpaketo-buildpacks%2Fbuilder%3Abase-platform-api-0.3"
time="2021-03-16T21:00:26Z" level=info msg="Pipelines plugin request authorization." allowed=true method=GET plugin=pipelines uri="/v1.24/images/gcr.io/paketo-buildpacks/builder@sha256:872e925fe54df9f4392483ef548eb4eda19851139efd71e8d5925db92cfd394d/json"
time="2021-03-16T21:00:26Z" level=info msg="Pipelines plugin request authorization." allowed=true method=POST plugin=pipelines uri="/v1.24/images/create?fromImage=docker.io%2Fpaketobuildpacks%2Frun%3Abase-cnb"
time="2021-03-16T21:00:27Z" level=info msg="Pipelines plugin request authorization." allowed=true method=GET plugin=pipelines uri="/v1.24/images/docker.io/paketobuildpacks/run@sha256:367a43536f60c21190cea5c06d040d01d29f4102840d6b3e1dcd72ed2eb71721/json"
time="2021-03-16T21:00:27Z" level=info msg="Pipelines plugin request authorization." allowed=true method=POST plugin=pipelines uri=/v1.24/images/load
time="2021-03-16T21:00:27Z" level=info msg="Container create request." ArgsEscaped=false AttachStderr=false AttachStdin=false AttachStdout=false ExposedPorts="map[]" Healthcheck="<nil>" Labels="map[author:spring-boot]" MacAddress= NetworkDisabled=false OnBuild="[]" OpenStdin=false StdinOnce=false StopSignal= StopTimeout="<nil>" Tty=false plugin=pipelines
time="2021-03-16T21:00:27Z" level=info msg="Container create request." AutoRemove=false BlkioDeviceReadBps="[]" BlkioDeviceReadIOps="[]" BlkioDeviceWriteBps="[]" BlkioDeviceWriteIOps="[]" BlkioWeight=0 BlkioWeightDevice="[]" CPUCount=0 CPUPercent=0 CPUPeriod=0 CPUQuota=0 CPURealtimePeriod=0 CPURealtimeRuntime=0 CPUShares=0 CapAdd="[]" CapDrop="[]" Capabilities="[]" Cgroup= CgroupParent= ConsoleSize="[0 0]" ContainerIDFile= CpusetCpus= CpusetMems= DNS="[]" DNSOptions="[]" DNSSearch="[]" DeviceCgroupRules="[]" Devices="[]" ExtraHosts="[]" GroupAdd="[]" IOMaximumBandwidth=0 IOMaximumIOps=0 Init="<nil>" IpcMode= Isolations= KernelMemory=0 Links="[]" LogConfig="{ map[]}" MaskedPaths="[]" Memory=0 MemoryReservation=0 MemorySwap=0 MemorySwappiness="<nil>" Mounts="[]" NanoCPUs=0 NetworkMode=default OomKillDisable="<nil>" OomScoreAdj=0 PidMode= PidsLimit="<nil>" PortBindings="map[]" Privileged=false PublishAllPorts=false ReadOnlyPaths="[]" RestartPolicy="{ 0}" Runtime= SecurityOpt="[]" ShmSize=0 StorageOpt="map[]" Sysctls="map[]" Ulimits="[]" UsernsMode= VolumeDriver= VolumesFrom="[]" plugin=pipelines
time="2021-03-16T21:00:27Z" level=info msg="Pipelines plugin request authorization." allowed=false method=POST plugin=pipelines uri=/v1.24/containers/create
time="2021-03-16T21:00:27.763743186Z" level=error msg="AuthZRequest for POST /v1.24/containers/create returned error: authorization denied by plugin pipelines: -v only supports $BITBUCKET_CLONE_DIR and its subdirectories"
time="2021-03-16T21:00:27Z" level=info msg="Pipelines plugin request authorization." allowed=true method=DELETE plugin=pipelines uri="/v1.24/volumes/pack-layers-emjtaniasm?force=1"
time="2021-03-16T21:00:27Z" level=info msg="Pipelines plugin request authorization." allowed=true method=DELETE plugin=pipelines uri="/v1.24/volumes/pack-app-ggwrhmgsws?force=1"
time="2021-03-16T21:00:27Z" level=info msg="Pipelines plugin request authorization." allowed=true method=DELETE plugin=pipelines uri="/v1.24/images/pack.local/builder/bsjzzxpubv:latest?force=1"
@corneil This is the same error that @wilkinsona commented on above:
time="2021-03-16T21:00:27.763743186Z" level=error msg="AuthZRequest for POST /v1.24/containers/create returned error: authorization denied by plugin pipelines: -v only supports $BITBUCKET_CLONE_DIR and its subdirectories"
This appears to be related to configuration in BitBucket, and not something Spring Boot has control over.
What does
jibDockerBuild
do differently?
As I understand it, Jib creates an image on the local file system without relying on a Docker daemon. The Spring Boot Maven plugin spring-boot:build-image
integrates with Cloud Native Buildpacks, which requires a running Docker Daemon.
Does packeto create tmp
or other folders outside of the build tree?
containers/create returned error: authorization denied by plugin pipelines: -v only supports $BITBUCKET_CLONE_DIR and its subdirectories
Also found https://jira.atlassian.com/browse/BCLOUD-17592 that may be related.
@corneil This discussion has strayed from the original problem report, which is related to an issue in Spring Boot which has been fixed. The Spring Boot team are not experts on the inner workings of Cloud Native Buildpacks and Paketo. It would be better to move the discussion of the -v only supports $BITBUCKET_CLONE_DIR and its subdirectories
error to StackOverflow, the CNB Slack, or the Paketo Slack where people who are closer to those implementations can be involved.
I found another post of someone using pack
also failing with the same error.
https://community.atlassian.com/t5/Bitbucket-discussions/Using-CNCF-Buildpacks-on-Bitbucket-Pipelines/td-p/1516690
I have reported to BitBucket support in the hope that they will take it serious if CNCF tool doesn't work in the pipelines.
BitBucket/Atlassian have not solved this issue: jira.atlassian.com/browse/BCLOUD-17592s
I have got the same error when trying to build a Spring Boot 3.1.5 container in BitBucket using maven 3.9.5 and Java 21:
mvn spring-boot:build-image
[INFO] --- spring-boot:3.1.5:build-image (default-cli) @ example ---
[INFO] Building image '[docker.io/library/example:0.0.1-SNAPSHOT](http://docker.io/library/example:0.0.1-SNAPSHOT)'
[INFO]
[INFO] > Pulling builder image '[docker.io/paketobuildpacks/builder-jammy-base:latest](http://docker.io/paketobuildpacks/builder-jammy-base:latest)' 2%
[INFO] > Pulling builder image '[docker.io/paketobuildpacks/builder-jammy-base:latest](http://docker.io/paketobuildpacks/builder-jammy-base:latest)' 5%
[INFO] > Pulling builder image '[docker.io/paketobuildpacks/builder-jammy-base:latest](http://docker.io/paketobuildpacks/builder-jammy-base:latest)' 9%
[INFO] > Pulling builder image '[docker.io/paketobuildpacks/builder-jammy-base:latest](http://docker.io/paketobuildpacks/builder-jammy-base:latest)' 15%
[INFO] > Pulling builder image '[docker.io/paketobuildpacks/builder-jammy-base:latest](http://docker.io/paketobuildpacks/builder-jammy-base:latest)' 26%
[INFO] > Pulling builder image '[docker.io/paketobuildpacks/builder-jammy-base:latest](http://docker.io/paketobuildpacks/builder-jammy-base:latest)' 37%
[INFO] > Pulling builder image '[docker.io/paketobuildpacks/builder-jammy-base:latest](http://docker.io/paketobuildpacks/builder-jammy-base:latest)' 49%
[INFO] > Pulling builder image '[docker.io/paketobuildpacks/builder-jammy-base:latest](http://docker.io/paketobuildpacks/builder-jammy-base:latest)' 59%
[INFO] > Pulling builder image '[docker.io/paketobuildpacks/builder-jammy-base:latest](http://docker.io/paketobuildpacks/builder-jammy-base:latest)' 74%
[INFO] > Pulling builder image '[docker.io/paketobuildpacks/builder-jammy-base:latest](http://docker.io/paketobuildpacks/builder-jammy-base:latest)' 100%
[INFO] > Pulled builder image 'paketobuildpacks/builder-jammy-base@sha256:5a2945e4c741b029b6abc0ab91d02a4e7d60edb3d3b2502f74d1b639a2076e6b'
[INFO] > Pulling run image '[docker.io/paketobuildpacks/run-jammy-base:latest](http://docker.io/paketobuildpacks/run-jammy-base:latest)' 100%
[INFO] > Pulled run image 'paketobuildpacks/run-jammy-base@sha256:8745d5878b439a7ffaf9ad755f7b61b89a9e5d33c125600180f0ec7ce95d5f20'
[INFO] > Executing lifecycle version v0.18.1
[INFO] > Using build cache volume 'pack-cache-8847f704ad41.build'
[INFO]
[INFO] > Running creator
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 45.481 s
[INFO] Finished at: 2023-11-03T16:23:40Z
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.springframework.boot:spring-boot-maven-plugin:3.1.5:build-image (default-cli) on project example: Execution default-cli of goal org.springframework.boot:spring-boot-maven-plugin:3.1.5:build-image failed: Docker API call to 'localhost:2375/v1.24/containers/create' failed with status code 403 "Forbidden" -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginExecutionException
Could we reopen this issue please?
@dbaltor As you can see in the discussions above, there are many potential root causes of the error you are seeing that are not related to the original cause. We'd need to see the Docker logs from BitBucket to know if your problem is related to the original issue.
Hi @scottfrederick , thanks for the swift response. Yeah, you're right. I run it in debug mode and it's the same error reported by @corneil above. It seems related to how the buildpack runs not being compatible with the BitBucket env.
[DEBUG] http-outgoing-0 >> POST /v1.24/containers/create HTTP/1.1
[DEBUG] http-outgoing-0 >> Accept-Encoding: gzip, x-gzip, deflate
[DEBUG] http-outgoing-0 >> Content-Length: 742
[DEBUG] http-outgoing-0 >> Content-Type: application/json
[DEBUG] http-outgoing-0 >> Content-Encoding: UTF-8
[DEBUG] http-outgoing-0 >> Host: localhost:2375
[DEBUG] http-outgoing-0 >> Connection: keep-alive
[DEBUG] http-outgoing-0 >> User-Agent: Apache-HttpClient/5.2.1 (Java/21.0.1)
[DEBUG] http-outgoing-0 >> "POST /v1.24/containers/create HTTP/1.1[\r][\n]"
[DEBUG] http-outgoing-0 >> "Accept-Encoding: gzip, x-gzip, deflate[\r][\n]"
[DEBUG] http-outgoing-0 >> "Content-Length: 742[\r][\n]"
[DEBUG] http-outgoing-0 >> "Content-Type: application/json[\r][\n]"
[DEBUG] http-outgoing-0 >> "Content-Encoding: UTF-8[\r][\n]"
[DEBUG] http-outgoing-0 >> "Host: localhost:2375[\r][\n]"
[DEBUG] http-outgoing-0 >> "Connection: keep-alive[\r][\n]"
[DEBUG] http-outgoing-0 >> "User-Agent: Apache-HttpClient/5.2.1 (Java/21.0.1)[\r][\n]"
[DEBUG] http-outgoing-0 >> "[\r][\n]"
[DEBUG] http-outgoing-0 >> "{[\n]"
[DEBUG] http-outgoing-0 >> " "User" : "root",[\n]"
[DEBUG] http-outgoing-0 >> " "Image" : "pack.local/builder/wrficodrgr:latest",[\n]"
[DEBUG] http-outgoing-0 >> " "Cmd" : [ "/cnb/lifecycle/creator", "-app", "/workspace", "-platform", "/platform", "-run-image", "docker.io/paketobuildpacks/run-jammy-base:latest", "-layers", "/layers", "-cache-dir", "/cache", "-launch-cache", "/launch-cache", "-daemon", "docker.io/library/example:0.0.1-SNAPSHOT" ],[\n]"
[DEBUG] http-outgoing-0 >> " "Env" : [ "CNB_PLATFORM_API=0.11" ],[\n]"
[DEBUG] http-outgoing-0 >> " "Labels" : {[\n]"
[DEBUG] http-outgoing-0 >> " "author" : "spring-boot"[\n]"
[DEBUG] http-outgoing-0 >> " },[\n]"
[DEBUG] http-outgoing-0 >> " "HostConfig" : {[\n]"
[DEBUG] http-outgoing-0 >> " "Binds" : [ "/var/run/docker.sock:/var/run/docker.sock", "pack-layers-tjjgagqsgk:/layers", "pack-app-stzbsvwuxn:/workspace", "pack-cache-8847f704ad41.build:/cache", "pack-cache-8847f704ad41.launch:/launch-cache" ],[\n]"
[DEBUG] http-outgoing-0 >> " "SecurityOpt" : [ "label=disable" ][\n]"
[DEBUG] http-outgoing-0 >> " }[\n]"
[DEBUG] http-outgoing-0 >> "}"
[DEBUG] http-outgoing-0 << "HTTP/1.1 403 Forbidden[\r][\n]"
[DEBUG] http-outgoing-0 << "Content-Type: application/json[\r][\n]"
[DEBUG] http-outgoing-0 << "Date: Fri, 03 Nov 2023 17:28:44 GMT[\r][\n]"
[DEBUG] http-outgoing-0 << "Content-Length: 117[\r][\n]"
[DEBUG] http-outgoing-0 << "[\r][\n]"
[DEBUG] http-outgoing-0 << "{"message":"authorization denied by plugin pipelines: -v only supports $BITBUCKET_CLONE_DIR and its subdirectories"}[\n]"
[DEBUG] http-outgoing-0 << HTTP/1.1 403 Forbidden
[DEBUG] http-outgoing-0 << Content-Type: application/json
[DEBUG] http-outgoing-0 << Date: Fri, 03 Nov 2023 17:28:44 GMT
[DEBUG] http-outgoing-0 << Content-Length: 117
I'll follow your suggestion and look for help with the buildpack folks. Have a great weekend :slightly_smiling_face:
@dbaltor, I am having this same issue have you had any success with the buildpak folks?
Running
mvn spring-boot:build-image
in bitbucket pipeline (maven:3.6.3-openjdk-11 image with default docker service) fails in the "Running creator" step.From the docker log it appears that "authorization (was) denied by plugin pipelines: Invalid content length provided". This could be a red herring as bitbucket documentation indicates the docker commands are restricted https://support.atlassian.com/bitbucket-cloud/docs/run-docker-commands-in-bitbucket-pipelines/
Unfortunately, there isn't enough debug logging to troubleshoot the
createContainer
method further.Can you please indicate whether this is a known limitation or if there is some configuration that can be included to work around the bitbucket restrictions?
Bitbucket docker log:
Maven stack trace: