Receiving 401 API access error to wrong region when using Chaos Monkey with instance roles and sts service.

[pwhitham]

I am currently running Chaos Monkey on an ubuntu 16 node in the us-west-2 region which has an instance role with inline policies attached to this instance. When I execute ssh'd based chaos monkey types they fail because they are accessing the us-east-1 region and not the expected us-west-2. Shutdown instance type is not affected by this issue.

The following line is in my client.properties file simianarmy.client.aws.region = us-west-2

However, when I modify the client.properties to contain an explicit AWS access key/secret this issue does not occur and it accesses the appropriate region.

2016-09-12 14:40:20.339 - WARN ChaosInstance - [ChaosInstance.java:105] Error making SSH connection to instance org.jclouds.rest.AuthorizationException: POST **https://ec2.us-east-1.amazonaws.com/ HTTP/1.1** -> HTTP/1.1 401 Unauthorized at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.refineException(ParseAWSErrorFromXmlContent.java:122) at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:89) at org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:65) at org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:132) at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:101) at org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:90) at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:73) at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:44) at org.jclouds.reflect.FunctionalReflection$FunctionalInvocationHandler.handleInvocation(FunctionalReflection.java:117) at com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:87) at com.sun.proxy.$Proxy169.describeRegions(Unknown Source) at org.jclouds.ec2.suppliers.DescribeRegionsForRegionURIs.get(DescribeRegionsForRegionURIs.java:50) at org.jclouds.ec2.suppliers.DescribeRegionsForRegionURIs.get(DescribeRegionsForRegionURIs.java:38) at org.jclouds.rest.suppliers.MemoizedRetryOnTimeOutButNotOnAuthorizationExceptionSupplier$SetAndThrowAuthorizationExceptionSupplierBackedLoader.load(MemoizedRetryOnTimeOutButNotOnAuthorizationExceptionSupplier.java:73) at org.jclouds.rest.suppliers.MemoizedRetryOnTimeOutButNotOnAuthorizationExceptionSupplier$SetAndThrowAuthorizationExceptionSupplierBackedLoader.load(MemoizedRetryOnTimeOutButNotOnAuthorizationExceptionSupplier.java:57) at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527) at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319) at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282) at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197) at com.google.common.cache.LocalCache.get(LocalCache.java:3937) at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941) at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824) at org.jclouds.rest.suppliers.MemoizedRetryOnTimeOutButNotOnAuthorizationExceptionSupplier.get(MemoizedRetryOnTimeOutButNotOnAuthorizationExceptionSupplier.java:119) at org.jclouds.location.suppliers.derived.RegionIdsFromRegionIdToURIKeySet.get(RegionIdsFromRegionIdToURIKeySet.java:45) at org.jclouds.location.suppliers.derived.RegionIdsFromRegionIdToURIKeySet.get(RegionIdsFromRegionIdToURIKeySet.java:33) at com.google.common.base.Suppliers$SupplierComposition.get(Suppliers.java:67) at org.jclouds.rest.suppliers.MemoizedRetryOnTimeOutButNotOnAuthorizationExceptionSupplier$SetAndThrowAuthorizationExceptionSupplierBackedLoader.load(MemoizedRetryOnTimeOutButNotOnAuthorizationExceptionSupplier.java:73) at org.jclouds.rest.suppliers.MemoizedRetryOnTimeOutButNotOnAuthorizationExceptionSupplier$SetAndThrowAuthorizationExceptionSupplierBackedLoader.load(MemoizedRetryOnTimeOutButNotOnAuthorizationExceptionSupplier.java:57) at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527) at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319) at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282) at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197) at com.google.common.cache.LocalCache.get(LocalCache.java:3937) at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941) at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824) at org.jclouds.rest.suppliers.MemoizedRetryOnTimeOutButNotOnAuthorizationExceptionSupplier.get(MemoizedRetryOnTimeOutButNotOnAuthorizationExceptionSupplier.java:119) at org.jclouds.aws.ec2.compute.strategy.AWSEC2ListNodesStrategy.pollRunningInstances(AWSEC2ListNodesStrategy.java:65) at org.jclouds.ec2.compute.strategy.EC2ListNodesStrategy.listDetailsOnNodesMatching(EC2ListNodesStrategy.java:107) at org.jclouds.ec2.compute.strategy.EC2ListNodesStrategy.listNodes(EC2ListNodesStrategy.java:86) at org.jclouds.ec2.compute.strategy.EC2ListNodesStrategy.listNodes(EC2ListNodesStrategy.java:58) at org.jclouds.compute.internal.BaseComputeService.listNodes(BaseComputeService.java:335) at com.netflix.simianarmy.client.aws.AWSClient.getJcloudsNode(AWSClient.java:857) at com.netflix.simianarmy.client.aws.AWSClient.connectSsh(AWSClient.java:837) at com.netflix.simianarmy.chaos.ChaosInstance.connectSsh(ChaosInstance.java:123) at com.netflix.simianarmy.chaos.ChaosInstance.canConnectSsh(ChaosInstance.java:101) at com.netflix.simianarmy.chaos.ScriptChaosType.canApply(ScriptChaosType.java:55) at com.netflix.simianarmy.chaos.FillDiskChaosType.canApply(FillDiskChaosType.java:52) at com.netflix.simianarmy.basic.chaos.BasicChaosMonkey.pickChaosType(BasicChaosMonkey.java:143) at com.netflix.simianarmy.basic.chaos.BasicChaosMonkey.doMonkeyBusiness(BasicChaosMonkey.java:123) at com.netflix.simianarmy.Monkey.run(Monkey.java:134) at com.netflix.simianarmy.Monkey$1.run(Monkey.java:155) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.jclouds.http.HttpResponseException: request: POST https://ec2.us-east-1.amazonaws.com/ HTTP/1.1 [Action=DescribeRegions] failed with response: HTTP/1.1 401 Unauthorized at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:63) ... 56 more

[jsuh555]

Were you able to resolve this?

[VinnieGogniti]

Is there no way around this issue? Looks like there is no definitive solution for this yet.

[archthegit]

Hi has anyone figured out a solution to this? I am facing the same issue