search

civitaspo / digdag-operator-ecs_task

digdag plugin for AWS ECS Task
Apache License 2.0
29 stars 9 forks source link

[Request]Support for credential_source in profile file #74

Open senshokazu opened 4 years ago

senshokazu commented 4 years ago

Problem

I'd expect the same support for credential_source as the CLI. But, profiles with only iam_role and credential_source properties are ignored.

Error Log

2020-05-28 11:33:58 +0900 [ERROR] (0018@[0:default]+test+ecs_task_run^sub+register): Task failed with unexpected error: Unable to load credentials from profile [roleA]: Source profile name is not specified
com.amazonaws.SdkClientException: Unable to load credentials from profile [roleA]: Source profile name is not specified
        at com.amazonaws.auth.profile.internal.ProfileAssumeRoleCredentialsProvider.fromAssumeRole(ProfileAssumeRoleCredentialsProvider.java:60)
        at com.amazonaws.auth.profile.internal.ProfileAssumeRoleCredentialsProvider.<init>(ProfileAssumeRoleCredentialsProvider.java:46)
        at com.amazonaws.auth.profile.ProfilesConfigFile.fromProfile(ProfilesConfigFile.java:209)
        at com.amazonaws.auth.profile.ProfilesConfigFile.getCredentials(ProfilesConfigFile.java:160)
        at com.amazonaws.auth.profile.ProfileCredentialsProvider.getCredentials(ProfileCredentialsProvider.java:161)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.getCredentialsFromContext(AmazonHttpClient.java:1251)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.runBeforeRequestHandlers(AmazonHttpClient.java:827)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:777)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:764)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:738)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:698)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:680)
        at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:544)
        at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:524)
        at com.amazonaws.services.ecs.AmazonECSClient.doInvoke(AmazonECSClient.java:4238)
        at com.amazonaws.services.ecs.AmazonECSClient.invoke(AmazonECSClient.java:4205)
        at com.amazonaws.services.ecs.AmazonECSClient.invoke(AmazonECSClient.java:4194)
        at com.amazonaws.services.ecs.AmazonECSClient.executeRegisterTaskDefinition(AmazonECSClient.java:2923)
        at com.amazonaws.services.ecs.AmazonECSClient.registerTaskDefinition(AmazonECSClient.java:2893)
        at pro.civitaspo.digdag.plugin.ecs_task.register.EcsTaskRegisterOperator.$anonfun$runTask$1(EcsTaskRegisterOperator.scala:414)
        at pro.civitaspo.digdag.plugin.ecs_task.aws.Aws.withEcs(Aws.scala:47)
        at pro.civitaspo.digdag.plugin.ecs_task.register.EcsTaskRegisterOperator.runTask(EcsTaskRegisterOperator.scala:414)
        at io.digdag.util.BaseOperator.run(BaseOperator.java:35)
        at io.digdag.core.agent.OperatorManager.callExecutor(OperatorManager.java:315)
        at io.digdag.cli.Run$OperatorManagerWithSkip.callExecutor(Run.java:705)
        at io.digdag.core.agent.OperatorManager.runWithWorkspace(OperatorManager.java:257)
        at io.digdag.core.agent.OperatorManager.lambda$runWithHeartbeat$2(OperatorManager.java:137)
        at io.digdag.core.agent.LocalWorkspaceManager.withExtractedArchive(LocalWorkspaceManager.java:25)
        at io.digdag.core.agent.OperatorManager.runWithHeartbeat(OperatorManager.java:135)
        at io.digdag.core.agent.OperatorManager.run(OperatorManager.java:119)
        at io.digdag.cli.Run$OperatorManagerWithSkip.run(Run.java:687)
        at io.digdag.core.agent.MultiThreadAgent.lambda$null$0(MultiThreadAgent.java:127)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

~/.aws.credentials

[roleA]
region=ap-northeast-1
role_arn=xxxxxxxxxxxxx
credential_source=Ec2InstanceMetadata

Possible Solution

Currently using aws-java-sdk version is 1.11.751, but supported versions of Support for credential_source is 2.5.30 or higher.

build.gradle: https://github.com/civitaspo/digdag-operator-ecs_task/blob/master/build.gradle#L32

Support for credential_source in profile file: https://github.com/aws/aws-sdk-java-v2/issues/1169

Would you please consider upgrading aws-adk version?
I'm very sorry, but I have never used scala, so it's difficult to submit a pull request.

civitaspo commented 4 years ago

Changing aws-sdk-java v1 -> v2 is not easy, so please wait for resolving the issues.