search

jenkinsci / pipeline-aws-plugin

Jenkins Pipeline Step Plugin for AWS
https://plugins.jenkins.io/pipeline-aws/
Apache License 2.0
431 stars 201 forks source link

withAWS step always use master's instance profile security token even running on slave. #64

Closed chrisLeeTW closed 6 years ago

chrisLeeTW commented 6 years ago

jenkins version : 2.109 plugin version : 1.24

I found that withAWS step always use master's instance profile security token when build job is running on slave.

here is my pipeline code

node('slave-type2') {
    stage('Post Build.') {
        sh('aws sts get-caller-identity')
        def identity = awsIdentity()
        withAWS(role:'test', roleAccount:'012345679091') {
            files = s3FindFiles(bucket:'my-bucket', glob:'test.html')
            for (file in files) {
                println file.name
            }
        }
    }
}

here is console log

[Pipeline] node
Running on ubuntu 16.04 slave-type2 (sir-xxxx) in /mnt/jenkins/workspace/media-test/_test_be_jenkins_func
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Post Build.)
[Pipeline] sh
[_test_be_jenkins_func] Running shell script
+ aws sts get-caller-identity
{
    "Account": "012345679091",
    "UserId": "XXXXXXXXXXXXXXXX:i-xxxxxxxxxxxx", 
    "Arn": "arn:aws:sts::012345679091:assumed-role/mm-jenkins-ci-slave/i-xxxxxxxxxxxx"
}
[Pipeline] awsIdentity
Current AWS identity: 012345679091 - XXXXXXXXXXXXXXXX:i-xxxxxxxxxxxx - arn:aws:sts::012345679091:assumed-role/mm-jenkins-ci-master/i-xxxxxxxxxxxx
[Pipeline] withAWS
[Pipeline] // withAWS
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
[BFA] Scanning build for known causes...
[BFA] No failure causes found
[BFA] Done. 0s
com.amazonaws.services.securitytoken.model.AWSSecurityTokenServiceException: User: arn:aws:sts::012345679091:assumed-role/mm-jenkins-ci-master/i-xxxxxxxxxxxxxxx is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::012345679091:role/test (Service: AWSSecurityTokenService; Status Code: 403; Error Code: AccessDenied; Request ID: xxxxxxxxxxxxxxxxxx)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1639)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1304)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1056)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:743)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)
    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)
    at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.doInvoke(AWSSecurityTokenServiceClient.java:1271)
    at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.invoke(AWSSecurityTokenServiceClient.java:1247)
    at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.executeAssumeRole(AWSSecurityTokenServiceClient.java:454)
    at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.assumeRole(AWSSecurityTokenServiceClient.java:431)
    at de.taimos.pipeline.aws.WithAWSStep$Execution.withRole(WithAWSStep.java:310)
    at de.taimos.pipeline.aws.WithAWSStep$Execution.start(WithAWSStep.java:235)
    at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:229)
    at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:153)
    at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:108)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
    at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1213)
    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022)
    at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:42)
    at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
    at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:157)
    at org.kohsuke.groovy.sandbox.GroovyInterceptor.onMethodCall(GroovyInterceptor.java:23)
    at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:133)
    at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:155)
    at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:159)
    at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:129)
    at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:129)
    at com.cloudbees.groovy.cps.sandbox.SandboxInvoker.methodCall(SandboxInvoker.java:17)
    at WorkflowScript.run(WorkflowScript:4)
    at ___cps.transform___(Native Method)
    at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:57)
    at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:109)
    at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:82)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
    at com.cloudbees.groovy.cps.impl.ClosureBlock.eval(ClosureBlock.java:46)
    at com.cloudbees.groovy.cps.Next.step(Next.java:83)
    at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:174)
    at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:163)
    at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:122)
    at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:261)
    at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:163)
    at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:19)
    at org.jenkinsci.plugins.workflow.cps.SandboxContinuable$1.call(SandboxContinuable.java:35)
    at org.jenkinsci.plugins.workflow.cps.SandboxContinuable$1.call(SandboxContinuable.java:32)
    at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.GroovySandbox.runInSandbox(GroovySandbox.java:108)
    at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:32)
    at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:174)
    at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:331)
    at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$200(CpsThreadGroup.java:82)
    at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:243)
    at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:231)
    at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:64)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:131)
    at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Finished: FAILURE
hoegertn commented 6 years ago

This is the intended behavior of the plugin. See https://issues.jenkins-ci.org/browse/JENKINS-44000

bdruth commented 6 years ago

So, I understand that this is the original intended behavior, but is there any reason we can't add a parameter to allow the plugin to evaluate this on the slave? This approach is a non-starter for organizations that have an on-prem Jenkins master w/ slaves in AWS (in addition to on-prem slaves).

hoegertn commented 6 years ago

Main thing is, there is no "easy" way to execute code on the agent. Only file operations are executed on agents. The plugin development documentation is very scarce on agent execution.

ebickle commented 6 years ago

Whether or not this is the "intended behavior", you should absolutely consider changing it - right now, the plugin won't work at all for certain use cases.

In our case, we have an ECS cluster for Jenkins and different roles for Master and Agent tasks. We assign the roles to the agents as a security best practice to reduce blast radius in the event of a worm compromising the Jenkins master - having master able to have VERY high permissions to our AWS accounts (for Continuous Deployment/etc) isn't needed - only the agents should have that level of access.

I'm already running aws commands, even shell commands to the AWS CLI, on the agent and with the appropriate role in the same account, and those agents have a different role than the master. Whether or not it's hard to do, I don't feel as though this task should be closed. I can already 'sh script: aws assume-role' on the agents.

Maybe someone else in the Jenkins team is able to help with the best practices/documentation on how to do this?

automatictester commented 5 years ago

@hoegertn I might be getting something wrong, but I guess all it would take to capture instance profile of the agent rather than master, would be to use AWSClientFactory not from within class extending StepExecution (which runs on master) as it is now, but from a subclass of MasterToSlaveFileCallable (which runs on slave).

albertsj1 commented 4 years ago

@hoegertn or anyone else on the dev team that can respond, Can we please re-open this issue, since it definitely used the master profile rather than the slave the job is running on? Or is this a more appropriate way to request this be fixed?

hoegertn commented 4 years ago

I'm open to merging a pull request implementing another behavior, but there is no planned implementation on my side.

davegallant commented 3 years ago

@hoegertn I might be getting something wrong, but I guess all it would take to capture instance profile of the agent rather than master, would be to use AWSClientFactory not from within class extending StepExecution (which runs on master) as it is now, but from a subclass of MasterToSlaveFileCallable (which runs on slave).

I have experimented with extending MasterToSlaveFileCallable to pass back a Serialized version of AWSCredentialsProvider. Interested in any thoughts around this approach. I plan on continuing to poke around with this.

hoegertn commented 3 years ago

If it works I am happy to accept a PR. Please make sure it is an optional implementation to not break the existing behavior