search

nathanmarz / storm-deploy

One click deploy for Storm clusters on AWS
Other
516 stars 148 forks source link

Cannot connect to the Storm UI with Storm 0.9.x #54

Open spisarski opened 10 years ago

spisarski commented 10 years ago

I have encountered many issues attempting to deploy storm to EC2. I have tried several different branches with varying failing results. My last attempt was without specifying a branch (apache-storm-0.9.1-incubating-SNAPSHOT). It appears that the UI daemon is running on the nimbus although it appears that port 8080 is not open for communications. According to the running processes, nimbus appears to be fine but it is difficult to verify without the UI. Additionally, it appears that only one of the two supervisor machines actually is running the daemon.

I noticed 6 "java.lang.ClassNotFoundException: backtype.storm.LocalDRPC" being thrown during the build process, but it really does not seem like this would be the root cause. Please see my commands and config which may help determine whether or not I have made some error.

Lein commands: lein deploy-storm --start --name test lein deploy-storm --attach --name test

.pallet/config.clj: (defpallet :services { :default { :blobstore-provider "aws-s3" :provider "aws-ec2" :environment {:user {:username "storm" ; this must be "storm" :private-key-path "~/.ssh/aws_rsa" :public-key-path "~/.ssh/aws_rsa.pub"} :aws-user-id "xxxx-xxxx-xxxx"} :identity "xxxxxxxxxxxxxxxxxxxxx" :credential "xxxxxxxxxxxxxxxxxxxxxxxxxxxxx" :jclouds.regions "us-west-2" } })

conf/clusters.yaml: nimbus.image: "us-west-2/ami-xxxxxxxx" #64-bit ubuntu nimbus.hardware: "t1.micro" supervisor.count: 2 supervisor.image: "us-west-2/ami-xxxxxxxx" #64-bit ubuntu supervisor.hardware: "t1.micro" zookeeper.count: 1 zookeeper.image: "us-west-2/ami-xxxxxxxx" #64-bit ubuntu zookeeper.hardware: "t1.micro"

conf/storm.yaml storm.local.dir: "/mnt/storm"

tbatchelli commented 10 years ago

Wondering if this is an issue with the port restrictions at the EC2 level. Can you reach port 8080 if you do an wget locally to the UI?

—

Antoni Batchelli,

From: Steve Pisarski notifications@github.com Reply: nathanmarz/storm-deploy reply@reply.github.com Date: January 9, 2014 at 2:57:26 PM To: nathanmarz/storm-deploy storm-deploy@noreply.github.com Subject:  [storm-deploy] Cannot connect to the Storm UI (#54)

I have encountered many issues attempting to deploy storm to
EC2. I have tried several different branches with varying failing
results. My last attempt was without specifying a branch (apache-storm-0.9.1-incubating-SNAPSHOT).
It appears that the UI daemon is running on the nimbus although
it appears that port 8080 is not open for communications. According
to the running processes, nimbus appears to be fine but it is difficult
to verify without the UI. Additionally, it appears that only
one of the two supervisor machines actually is running the daemon.

I noticed 6 "java.lang.ClassNotFoundException: backtype.storm.LocalDRPC"
being thrown during the build process, but it really does not
seem like this would be the root cause. Please see my commands
and config which may help determine whether or not I have made
some error.

Lein commands: lein deploy-storm --start --name test lein deploy-storm --attach --name test

.pallet/config.clj: (defpallet :services { :default { :blobstore-provider "aws-s3" :provider "aws-ec2" :environment {:user {:username "storm" ; this must be "storm"
:private-key-path "~/.ssh/aws_rsa" :public-key-path "~/.ssh/aws_rsa.pub"} :aws-user-id "xxxx-xxxx-xxxx"} :identity "xxxxxxxxxxxxxxxxxxxxx" :credential "xxxxxxxxxxxxxxxxxxxxxxxxxxxxx" :jclouds.regions "us-west-2" } })

conf/clusters.yaml: nimbus.image: "us-west-2/ami-xxxxxxxx" #64-bit ubuntu nimbus.hardware: "t1.micro" supervisor.count: 2 supervisor.image: "us-west-2/ami-xxxxxxxx" #64-bit ubuntu
supervisor.hardware: "t1.micro" zookeeper.count: 1 zookeeper.image: "us-west-2/ami-xxxxxxxx" #64-bit ubuntu
zookeeper.hardware: "t1.micro"

conf/storm.yaml storm.local.dir: "/mnt/storm"

Reply to this email directly or view it on GitHub: https://github.com/nathanmarz/storm-deploy/issues/54

spisarski commented 10 years ago

Thanks for your quick reply!

No luck. I tried with and without the protocol "http://"

storm@ip-172-31-6-67:~$ wget http://localhost:8080 --2014-01-09 23:06:46-- http://localhost:8080/ Resolving localhost (localhost)... 127.0.0.1 Connecting to localhost (localhost)|127.0.0.1|:8080... failed: Connection refused.

storm@ip-172-31-6-67:~$ wget localhost:8080 --2014-01-09 23:07:20-- http://localhost:8080/ Resolving localhost (localhost)... 127.0.0.1 Connecting to localhost (localhost)|127.0.0.1|:8080... failed: Connection refused.

Also tried with curl and it returned "couldn't connect to host"

spisarski commented 10 years ago

BTW, I have had no issues accessing Ganglia...

msegel commented 10 years ago

You can reset the ports in the storm.yaml that gets sent.

However if you don't, you can ssh to the nimbus node, and see if storm UI is running, and then check the logs for errors.

On Jan 9, 2014, at 5:03 PM, Antoni Batchelli notifications@github.com wrote:

Wondering if this is an issue with the port restrictions at the EC2 level. Can you reach port 8080 if you do an wget locally to the UI?

—

Antoni Batchelli,

From: Steve Pisarski notifications@github.com Reply: nathanmarz/storm-deploy reply@reply.github.com Date: January 9, 2014 at 2:57:26 PM To: nathanmarz/storm-deploy storm-deploy@noreply.github.com Subject: [storm-deploy] Cannot connect to the Storm UI (#54)

I have encountered many issues attempting to deploy storm to EC2. I have tried several different branches with varying failing results. My last attempt was without specifying a branch (apache-storm-0.9.1-incubating-SNAPSHOT). It appears that the UI daemon is running on the nimbus although it appears that port 8080 is not open for communications. According to the running processes, nimbus appears to be fine but it is difficult to verify without the UI. Additionally, it appears that only one of the two supervisor machines actually is running the daemon.

I noticed 6 "java.lang.ClassNotFoundException: backtype.storm.LocalDRPC" being thrown during the build process, but it really does not seem like this would be the root cause. Please see my commands and config which may help determine whether or not I have made some error.

Lein commands: lein deploy-storm --start --name test lein deploy-storm --attach --name test

.pallet/config.clj: (defpallet :services { :default { :blobstore-provider "aws-s3" :provider "aws-ec2" :environment {:user {:username "storm" ; this must be "storm" :private-key-path "~/.ssh/aws_rsa" :public-key-path "~/.ssh/aws_rsa.pub"} :aws-user-id "xxxx-xxxx-xxxx"} :identity "xxxxxxxxxxxxxxxxxxxxx" :credential "xxxxxxxxxxxxxxxxxxxxxxxxxxxxx" :jclouds.regions "us-west-2" } })

conf/clusters.yaml: nimbus.image: "us-west-2/ami-xxxxxxxx" #64-bit ubuntu nimbus.hardware: "t1.micro" supervisor.count: 2 supervisor.image: "us-west-2/ami-xxxxxxxx" #64-bit ubuntu supervisor.hardware: "t1.micro" zookeeper.count: 1 zookeeper.image: "us-west-2/ami-xxxxxxxx" #64-bit ubuntu zookeeper.hardware: "t1.micro"

conf/storm.yaml storm.local.dir: "/mnt/storm"

Reply to this email directly or view it on GitHub: https://github.com/nathanmarz/storm-deploy/issues/54 — Reply to this email directly or view it on GitHub.

spisarski commented 10 years ago

That is another problem. I cannot find any logs whatsoever. The ~storm/logs/ directory is empty and could not find the files "ui.log" or "nimbus.log". Here is the contents of my built storm.yaml:

storm.local.dir: "/mnt/storm" storm.zookeeper.servers:

The /mnt/storm directory is empty. Is this correct? Storm is deployed to "/home/storm/apache-storm-0.9.1-incubating-SNAPSHOT". Could the multiple "storm.local.dir" entries be problematic?

skillflip commented 10 years ago

I had lots of problems deploying storm as well similar to what you describe. What eventually worked for me was deploying 0.8.2. The exact same config wouldn't work on the 0.9.0 release candidates, 0.9.0.1 or the latest snapshot. Either the management UI wouldn't load at all - or it would load but show no workers configured etc.

0.8.2 worked perfectly first time.

msegel commented 10 years ago

Ok..

With storm deploy, if you look at ~storm, is that your main directory? Meaning you see the conf/ , bin/ and logs/ directories?

What do you get if you type 'which storm' ?

On Jan 10, 2014, at 9:47 AM, Steve Pisarski notifications@github.com wrote:

That is another problem. I cannot find any logs whatsoever. The ~storm/logs/ directory is empty and could not find the files "ui.log" or "nimbus.log". Here is the contents of my built storm.yaml:

storm.local.dir: "/mnt/storm" storm.zookeeper.servers:

"54.201.32.199" nimbus.host: "54.200.193.86" drpc.servers: "172.31.6.67" storm.local.dir: "/mnt/storm" The /mnt/storm directory is empty. Is this correct? Storm is deployed to "/home/storm/apache-storm-0.9.1-incubating-SNAPSHOT". Could the multiple "storm.local.dir" entries be problematic?

— Reply to this email directly or view it on GitHub.

spisarski commented 10 years ago

Thanks skillflip, 0.8.2 seems to have deployed nicely!

spisarski commented 10 years ago

Michael, that is indeed the directory structure observed with each version I had attempted; however, 0.8.2 was the only one that contained any log content, which is where I found the Storm UI port number 8772. I had been attempting 8080 in previous attempts of other versions but had no log output that told me that the UI was using a different port (if at all).

spisarski commented 10 years ago

Michael, sorry about the 8772, I had forgotten that I changed this myself in the storm.yaml to help with my debugging so ignore my other comments about the port number meaning that I definitely never had luck deploying any storm version other than 0.8.2...

dfcarney commented 10 years ago

My bet is that the AMI has Java 1.6 installed by default, and storm-0.9.x needs 1.7.

rahul8590 commented 8 years ago

Is this issue fixed ? I seemed to face similar problem while setting up storm 0.9.6 . The only difference I see, is that in my case nimbus logs do get updated . But the access and metrics log remain the same.