search

nathanmarz / storm-deploy

One click deploy for Storm clusters on AWS
Other
516 stars 148 forks source link

Errors authorizing security groups from supervisor/zookeeper to nimbus group (InvalidPlacementGroup.Unknown) #24

Closed guarly closed 10 years ago

guarly commented 11 years ago

The API Request to AWS requesting to add the supervisor/zookeeper groups to the nimbus security group fails consistently with an "InvalidPlacementGroup.Unknown" error, due to the fact that the API AuthorizeSecurityGroupIngress doesn't like telling a SourceSecurityGroupOwnerId when it's yourself:

2013-02-08 19:22:16,814 DEBUG [org.jclouds.http.internal.JavaUrlHttpCommandExecutorService] (i/o thread 7) Sending request -1262905079: POST https://ec2.us-east-1.amazonaws.com/ HTTP/1.1
2013-02-08 19:22:16,814 DEBUG [jclouds.wire] (i/o thread 7) >> "Action=AuthorizeSecurityGroupIngress&GroupName=jclouds%23nimbus-amplcluster%23us-east-1&Signature=8S5KsndBbwFYY8Asnd7x%2FLv1nmfBH%2B7YF63lQydRM4M%3D&SignatureMethod
=HmacSHA256&SignatureVersion=2&SourceSecurityGroupName=jclouds%23supervisor-amplcluster%23us-east-1&SourceSecurityGroupOwnerId=e26cd44119560f8e1e272d9bbc0928224c2414fde521bd81e4621789f9283ea6&Timestamp=2013-02-08T18%3A22%3A16.81
0Z&Version=2011-05-15&AWSAccessKeyId=AKIANNNNNNNNNNNNNNNNN"
2013-02-08 19:22:16,814 DEBUG [jclouds.headers] (i/o thread 7) >> POST https://ec2.us-east-1.amazonaws.com/ HTTP/1.1
2013-02-08 19:22:16,814 DEBUG [jclouds.headers] (i/o thread 7) >> Host: ec2.us-east-1.amazonaws.com
2013-02-08 19:22:16,814 DEBUG [jclouds.headers] (i/o thread 7) >> Content-Type: application/x-www-form-urlencoded
2013-02-08 19:22:16,815 DEBUG [jclouds.headers] (i/o thread 7) >> Content-Length: 449
2013-02-08 19:22:17,157 DEBUG [org.jclouds.http.internal.JavaUrlHttpCommandExecutorService] (i/o thread 7) Receiving response -1262905079: HTTP/1.1 400 Bad Request
2013-02-08 19:22:17,157 DEBUG [jclouds.headers] (i/o thread 7) << HTTP/1.1 400 Bad Request
2013-02-08 19:22:17,158 DEBUG [jclouds.headers] (i/o thread 7) << Transfer-Encoding: chunked
2013-02-08 19:22:17,158 DEBUG [jclouds.headers] (i/o thread 7) << Date: Fri, 08 Feb 2013 18:21:45 GMT
2013-02-08 19:22:17,158 DEBUG [jclouds.headers] (i/o thread 7) << Cneonction: close
2013-02-08 19:22:17,158 DEBUG [jclouds.headers] (i/o thread 7) << Server: AmazonEC2
2013-02-08 19:22:17,158 DEBUG [jclouds.headers] (i/o thread 7) << Content-Type: application/unknown
2013-02-08 19:22:17,158 DEBUG [jclouds.wire] (i/o thread 7) << "<?xml version="1.0" encoding="UTF-8"?>[\n]"
2013-02-08 19:22:17,158 DEBUG [jclouds.wire] (i/o thread 7) << "<Response><Errors><Error><Code>InvalidGroup.NotFound</Code><Message>Unable to find group 'jclouds#supervisor-amplcluster#us-east-1'</Message></Error></Errors><Reque
stID>31001939-f631-402a-a040-7ce66a25f41a</RequestID></Response>"
2013-02-08 19:22:17,177 ERROR [java.logging] (main) Exception in thread "main"
2013-02-08 19:22:17,177 ERROR [java.logging] (main) org.jclouds.rest.ResourceNotFoundException: Unable to find group 'jclouds#supervisor-amplcluster#us-east-1' (NO_SOURCE_FILE:1)
2013-02-08 19:22:17,177 ERROR [java.logging] (main) at clojure.lang.Compiler.eval(Compiler.java:5440)
2013-02-08 19:22:17,177 ERROR [java.logging] (main) at clojure.lang.Compiler.eval(Compiler.java:5415)
2013-02-08 19:22:17,177 ERROR [java.logging] (main) at clojure.lang.Compiler.eval(Compiler.java:5391)
2013-02-08 19:22:17,177 ERROR [java.logging] (main) at clojure.core$eval.invoke(core.clj:2382)
2013-02-08 19:22:17,178 ERROR [java.logging] (main) at clojure.main$eval_opt.invoke(main.clj:235)
2013-02-08 19:22:17,178 ERROR [java.logging] (main) at clojure.main$initialize.invoke(main.clj:254)
2013-02-08 19:22:17,178 ERROR [java.logging] (main) at clojure.main$null_opt.invoke(main.clj:279)
2013-02-08 19:22:17,178 ERROR [java.logging] (main) at clojure.main$main.doInvoke(main.clj:354)
2013-02-08 19:22:17,178 ERROR [java.logging] (main) at clojure.lang.RestFn.invoke(RestFn.java:422)
2013-02-08 19:22:17,178 ERROR [java.logging] (main) at clojure.lang.Var.invoke(Var.java:369)
2013-02-08 19:22:17,178 ERROR [java.logging] (main) at clojure.lang.AFn.applyToHelper(AFn.java:165)
2013-02-08 19:22:17,178 ERROR [java.logging] (main) at clojure.lang.Var.applyTo(Var.java:482)
2013-02-08 19:22:17,179 ERROR [java.logging] (main) at clojure.main.main(main.java:37)
2013-02-08 19:22:17,179 ERROR [java.logging] (main) Caused by: org.jclouds.rest.ResourceNotFoundException: Unable to find group 'jclouds#supervisor-amplcluster#us-east-1'
2013-02-08 19:22:17,179 ERROR [java.logging] (main) at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.refineException(ParseAWSErrorFromXmlContent.java:109)
2013-02-08 19:22:17,179 ERROR [java.logging] (main) at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:92)
2013-02-08 19:22:17,179 ERROR [java.logging] (main) at org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:69)
2013-02-08 19:22:17,179 ERROR [java.logging] (main) at org.jclouds.http.internal.BaseHttpCommandExecutorService$HttpResponseCallable.shouldContinue(BaseHttpCommandExecutorService.java:191)
2013-02-08 19:22:17,179 ERROR [java.logging] (main) at org.jclouds.http.internal.BaseHttpCommandExecutorService$HttpResponseCallable.call(BaseHttpCommandExecutorService.java:161)
2013-02-08 19:22:17,180 ERROR [java.logging] (main) at org.jclouds.http.internal.BaseHttpCommandExecutorService$HttpResponseCallable.call(BaseHttpCommandExecutorService.java:130)
2013-02-08 19:22:17,180 ERROR [java.logging] (main) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
2013-02-08 19:22:17,180 ERROR [java.logging] (main) at java.util.concurrent.FutureTask.run(FutureTask.java:166)
2013-02-08 19:22:17,180 ERROR [java.logging] (main) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
2013-02-08 19:22:17,180 ERROR [java.logging] (main) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
2013-02-08 19:22:17,180 ERROR [java.logging] (main) at java.lang.Thread.run(Thread.java:679)
2013-02-08 19:22:17,180 ERROR [java.logging] (main) at org.jclouds.concurrent.config.ExecutorServiceModule$DescribingExecutorService.submit(ExecutorServiceModule.java:184)
2013-02-08 19:22:17,180 ERROR [java.logging] (main) at org.jclouds.http.internal.BaseHttpCommandExecutorService.submit(BaseHttpCommandExecutorService.java:127)
2013-02-08 19:22:17,181 ERROR [java.logging] (main) at org.jclouds.http.TransformingHttpCommandExecutorServiceImpl.submit(TransformingHttpCommandExecutorServiceImpl.java:54)
2013-02-08 19:22:17,181 ERROR [java.logging] (main) at org.jclouds.http.TransformingHttpCommandImpl.execute(TransformingHttpCommandImpl.java:73)
2013-02-08 19:22:17,181 ERROR [java.logging] (main) at org.jclouds.rest.internal.AsyncRestClientProxy.createListenableFutureForHttpRequestMappedToMethodAndArgs(AsyncRestClientProxy.java:198)
2013-02-08 19:22:17,181 ERROR [java.logging] (main) at org.jclouds.rest.internal.AsyncRestClientProxy.invoke(AsyncRestClientProxy.java:134)
2013-02-08 19:22:17,181 ERROR [java.logging] (main) at $Proxy74.authorizeSecurityGroupIngressInRegion(Unknown Source)
2013-02-08 19:22:17,181 ERROR [java.logging] (main) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2013-02-08 19:22:17,181 ERROR [java.logging] (main) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
2013-02-08 19:22:17,182 ERROR [java.logging] (main) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2013-02-08 19:22:17,182 ERROR [java.logging] (main) at java.lang.reflect.Method.invoke(Method.java:616)
2013-02-08 19:22:17,182 ERROR [java.logging] (main) at org.jclouds.concurrent.internal.SyncProxy.invoke(SyncProxy.java:137)
2013-02-08 19:22:17,182 ERROR [java.logging] (main) at $Proxy75.authorizeSecurityGroupIngressInRegion(Unknown Source)
2013-02-08 19:22:17,182 ERROR [java.logging] (main) at backtype.storm.security$authorize_group.invoke(security.clj:116)
2013-02-08 19:22:17,182 ERROR [java.logging] (main) at backtype.storm.security$authorize_group.invoke(security.clj:111)
2013-02-08 19:22:17,182 ERROR [java.logging] (main) at backtype.storm.provision$start_with_nodes_BANG_.invoke(provision.clj:81)
2013-02-08 19:22:17,182 ERROR [java.logging] (main) at backtype.storm.provision$start_BANG_.invoke(provision.clj:95)
2013-02-08 19:22:17,183 ERROR [java.logging] (main) at backtype.storm.provision$_main.doInvoke(provision.clj:147)
2013-02-08 19:22:17,183 ERROR [java.logging] (main) at clojure.lang.RestFn.invoke(RestFn.java:483)
2013-02-08 19:22:17,183 ERROR [java.logging] (main) at clojure.lang.Var.invoke(Var.java:381)
2013-02-08 19:22:17,183 ERROR [java.logging] (main) at user$eval8251.invoke(NO_SOURCE_FILE:1)
2013-02-08 19:22:17,183 ERROR [java.logging] (main) at clojure.lang.Compiler.eval(Compiler.java:5424)
2013-02-08 19:22:17,183 ERROR [java.logging] (main) at clojure.lang.Compiler.eval(Compiler.java:5415)
2013-02-08 19:22:17,183 ERROR [java.logging] (main) at clojure.lang.Compiler.eval(Compiler.java:5391)
2013-02-08 19:22:17,183 ERROR [java.logging] (main) at clojure.core$eval.invoke(core.clj:2382)
2013-02-08 19:22:17,184 ERROR [java.logging] (main) at clojure.main$eval_opt.invoke(main.clj:235)
2013-02-08 19:22:17,184 ERROR [java.logging] (main) at clojure.main$initialize.invoke(main.clj:254)
2013-02-08 19:22:17,184 ERROR [java.logging] (main) at clojure.main$null_opt.invoke(main.clj:279)
2013-02-08 19:22:17,184 ERROR [java.logging] (main) at clojure.main$main.doInvoke(main.clj:354)
2013-02-08 19:22:17,184 ERROR [java.logging] (main) at clojure.lang.RestFn.invoke(RestFn.java:422)
2013-02-08 19:22:17,184 ERROR [java.logging] (main) at clojure.lang.Var.invoke(Var.java:369)
2013-02-08 19:22:17,184 ERROR [java.logging] (main) at clojure.lang.AFn.applyToHelper(AFn.java:165)
2013-02-08 19:22:17,184 ERROR [java.logging] (main) at clojure.lang.Var.applyTo(Var.java:482)
2013-02-08 19:22:17,185 ERROR [java.logging] (main) at clojure.main.main(main.java:37)
2013-02-08 19:22:17,185 ERROR [java.logging] (main) at org.jclouds.concurrent.config.ExecutorServiceModule$DescribingExecutorService.submit(ExecutorServiceModule.java:184)
2013-02-08 19:22:17,185 ERROR [java.logging] (main) at org.jclouds.http.internal.BaseHttpCommandExecutorService.submit(BaseHttpCommandExecutorService.java:127)
2013-02-08 19:22:17,185 ERROR [java.logging] (main) at org.jclouds.http.TransformingHttpCommandExecutorServiceImpl.submit(TransformingHttpCommandExecutorServiceImpl.java:54)
2013-02-08 19:22:17,185 ERROR [java.logging] (main) at org.jclouds.http.TransformingHttpCommandImpl.execute(TransformingHttpCommandImpl.java:73)
2013-02-08 19:22:17,185 ERROR [java.logging] (main) at org.jclouds.rest.internal.AsyncRestClientProxy.createListenableFutureForHttpRequestMappedToMethodAndArgs(AsyncRestClientProxy.java:198)
2013-02-08 19:22:17,185 ERROR [java.logging] (main) at org.jclouds.rest.internal.AsyncRestClientProxy.invoke(AsyncRestClientProxy.java:134)
2013-02-08 19:22:17,185 ERROR [java.logging] (main) at $Proxy74.authorizeSecurityGroupIngressInRegion(Unknown Source)
2013-02-08 19:22:17,186 ERROR [java.logging] (main) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2013-02-08 19:22:17,186 ERROR [java.logging] (main) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
2013-02-08 19:22:17,186 ERROR [java.logging] (main) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2013-02-08 19:22:17,186 ERROR [java.logging] (main) at java.lang.reflect.Method.invoke(Method.java:616)
2013-02-08 19:22:17,186 ERROR [java.logging] (main) at org.jclouds.concurrent.internal.SyncProxy.invoke(SyncProxy.java:137)
2013-02-08 19:22:17,186 ERROR [java.logging] (main) at $Proxy75.authorizeSecurityGroupIngressInRegion(Unknown Source)
2013-02-08 19:22:17,186 ERROR [java.logging] (main) at backtype.storm.security$authorize_group.invoke(security.clj:116)
2013-02-08 19:22:17,187 ERROR [java.logging] (main) at backtype.storm.security$authorize_group.invoke(security.clj:111)
2013-02-08 19:22:17,187 ERROR [java.logging] (main) at backtype.storm.provision$start_with_nodes_BANG_.invoke(provision.clj:81)
2013-02-08 19:22:17,187 ERROR [java.logging] (main) at backtype.storm.provision$start_BANG_.invoke(provision.clj:95)
2013-02-08 19:22:17,187 ERROR [java.logging] (main) at backtype.storm.provision$_main.doInvoke(provision.clj:147)
2013-02-08 19:22:17,187 ERROR [java.logging] (main) at clojure.lang.RestFn.invoke(RestFn.java:483)
2013-02-08 19:22:17,187 ERROR [java.logging] (main) at clojure.lang.Var.invoke(Var.java:381)
2013-02-08 19:22:17,187 ERROR [java.logging] (main) at user$eval8251.invoke(NO_SOURCE_FILE:1)
2013-02-08 19:22:17,188 ERROR [java.logging] (main) at clojure.lang.Compiler.eval(Compiler.java:5424)
2013-02-08 19:22:17,188 ERROR [java.logging] (main) ... 12 more
2013-02-08 19:22:17,188 ERROR [java.logging] (main) Caused by: org.jclouds.aws.AWSResponseException: request POST https://ec2.us-east-1.amazonaws.com/ HTTP/1.1 failed with code 400, error: AWSError{requestId='31001939-f631-402a-a040-7ce66a25f41a', requestToken='null', code='InvalidGroup.NotFound', message='Unable to find group 'jclouds#supervisor-amplcluster#us-east-1'', context='{Response=, Errors=}'}
2013-02-08 19:22:17,189 ERROR [java.logging] (main) at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:78)
2013-02-08 19:22:17,189 ERROR [java.logging] (main) at org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:69)
2013-02-08 19:22:17,189 ERROR [java.logging] (main) at org.jclouds.http.internal.BaseHttpCommandExecutorService$HttpResponseCallable.shouldContinue(BaseHttpCommandExecutorService.java:191)
2013-02-08 19:22:17,189 ERROR [java.logging] (main) at org.jclouds.http.internal.BaseHttpCommandExecutorService$HttpResponseCallable.call(BaseHttpCommandExecutorService.java:161)
2013-02-08 19:22:17,189 ERROR [java.logging] (main) at org.jclouds.http.internal.BaseHttpCommandExecutorService$HttpResponseCallable.call(BaseHttpCommandExecutorService.java:130)
2013-02-08 19:22:17,189 ERROR [java.logging] (main) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
2013-02-08 19:22:17,189 ERROR [java.logging] (main) at java.util.concurrent.FutureTask.run(FutureTask.java:166)
2013-02-08 19:22:17,189 ERROR [java.logging] (main) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
2013-02-08 19:22:17,190 ERROR [java.logging] (main) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
2013-02-08 19:22:17,190 ERROR [java.logging] (main) at java.lang.Thread.run(Thread.java:679)
guarly commented 11 years ago

I'm able to solve it with the following code snippet:

--- a/src/clj/backtype/storm/security.clj
+++ b/src/clj/backtype/storm/security.clj
@@ -116,7 +116,7 @@
         (sg-service compute)
         region
         to-group
-        (UserIdGroupPair. user-id from-group)
+        (UserIdGroupPair. "" from-group)
         )
     (catch IllegalStateException _)
     )))
txbm commented 11 years ago

You should open a pull request with your fix for this and the other issue.

guarly commented 11 years ago

Done, although not sure if complying with the necessary netiquette...

On Sun, Feb 10, 2013 at 9:34 PM, petermelias notifications@github.comwrote:

You should open a pull request with your fix for this and the other issue.

— Reply to this email directly or view it on GitHubhttps://github.com/nathanmarz/storm-deploy/issues/24#issuecomment-13360501..

lorcan commented 11 years ago

This issue should be closed, it was resolved by ec9c1251224765c76e7aa59842500834b2ce258b.