search

hortonworks / cloudbreak-deployer

Cloudbreak Deployer Tool
https://goo.gl/NbswU2
Apache License 2.0
35 stars 45 forks source link

CBD hangs after login #137

Closed afederic closed 8 years ago

afederic commented 9 years ago

Hi, our CBD installation on Centos 7.1.1503 hangs after user login as shown in the following picture

4feecd6a-799c-11e5-838f-58ccd9414a57

CBD logs from uluwatu shows that 

uluwatu_1 | GET / 304 1.012 ms - -
uluwatu_1 | GET /sultans/permission - - ms - -
uluwatu_1 | GET /sultans/account/details - - ms - -
uluwatu_1 | Sultans request to: http://130.186.13.93:3001/permission
uluwatu_1 | Sultans request to: http://130.186.13.93:3001/account/details
uluwatu_1 | GET /user?get=%7B%22method%22:%22GET%22%7D 304 45.189 ms - -
uluwatu_1 | GET /account/stacks 200 257.833 ms - 2
uluwatu_1 | GET /events 200 259.710 ms - 2
uluwatu_1 | GET /account/securitygroups 200 124.151 ms - 912
uluwatu_1 | GET /account/networks 200 221.742 ms - 838
uluwatu_1 | GET /account/templates 200 247.919 ms - 930
uluwatu_1 | GET /account/credentials 200 314.418 ms - 588
uluwatu_1 | GET /account/blueprints 200 371.855 ms - 433435
uluwatu_1 | Periscope request to: http://backend:8085/clusters
uluwatu_1 | GET /periscope/clusters 200 119.820 ms - 2

It seems it cannot get a response for account details and permissions. Thanks Ale

lalyos commented 9 years ago

hi @afederic

By checking the screenshot, it seems that authorization was successful, even more uluwatu was able to get permission to query cloudbreak api (for example 3 blueprints are received).

Could you please check if any container reports some error, if you try to login again

cbd logs
afederic commented 9 years ago

hi @lalyos I'm out of office now. I'll upload cbd logs on monday. But I'm quite sure that I didn't see any errors from the other container logs. Thanks

afederic commented 9 years ago

hi @lalyos you will find CBD logs here. Thanks Ale

lalyos commented 9 years ago

hi Ale,

Hmm i don't see any unusual in the logs ... If you could please join the support channel at hipchat: https://www.hipchat.com/gohd6v0n1 we could have a more interactive troubleshooting.

Fo hipchat: no installation/registration is needed, just a browser.

afederic commented 9 years ago

hi @lalyos I joined. Thanks Ale

afederic commented 9 years ago

hi @lalyos thank you for support. I will resize my CBD Openstack VM following the minimum recommendations (2 VCPU, 4GB RAM) that you gave me in the chat room. I will let you know asap. Thanks Ale

afederic commented 9 years ago

hi @lalyos I have resized the VM: 2 VCPU, 6GB RAM. Started cbd I still get "Client cannot access resource server."

[root@cbd ~]# cbd logs
[...]
cloudbreak_1    | 2015-10-26 15:02:40,357 [main] logStarted:61 INFO  c.s.c.CloudbreakApplication - [owner:spring] [type:springLog] [id:] [name:] Started CloudbreakApplication in 59.815 seconds (JVM running for 63.83)
sultans_1       | POST login.do - Client cannot access resource server. Check UAA address.
ambassador_1    | 2015/10/26 15:04:30 172.17.0.13:48906 -> node1.node.dc1.consul.:5433
ambassador_1    | 2015/10/26 15:04:30 172.17.0.13:48907 -> node1.node.dc1.consul.:5433
ambassador_1    | 2015/10/26 15:04:40 172.17.0.13:48912 -> node1.node.dc1.consul.:5433
cloudbreak_1    | 2015-10-26 15:04:48,418 [taskScheduler-1] send:36 INFO  c.s.c.s.n.HttpNotificationSender - [owner:spring] [type:springLog] [id:] [name:] Could not send notification to the specified endpoint: 'http://XXX.XXX.XXX.XXX:3000/notifications' Cause: I/O error on POST request for "http://XXX.XXX.XXX.XXX:3000/notifications":Connect to XXX.XXX.XXX.XXX:3000 [/XXX.XXX.XXX.XXX] failed: Connection timed out; nested exception is org.apache.http.conn.ConnectTimeoutException: Connect to XXX.XXX.XXX.XXX:3000 [/XXX.XXX.XXX.XXX] failed: Connection timed out
ambassador_1    | 2015/10/26 15:04:50 172.17.0.13:48915 -> node1.node.dc1.consul.:5433
ambassador_1    | 2015/10/26 15:04:54 172.17.0.14:51422 -> node1.node.dc1.consul.:8089
ambassador_1    | 2015/10/26 15:05:00 172.17.0.13:48921 -> node1.node.dc1.consul.:5433
ambassador_1    | 2015/10/26 15:05:00 172.17.0.13:48922 -> node1.node.dc1.consul.:5433
sultans_1       | POST login.do - Client cannot access resource server. Check UAA address.

[root@cbd ~]# uptime
 16:08:27 up 8 min,  4 users,  load average: 0.10, 0.34, 0.25
[root@cbd ~]# free -m
              total        used        free      shared  buff/cache   available
Mem:           5807        2238        2808          52         760        3301
Swap:             0           0           0
afederic commented 9 years ago

UPDATE: after some time the previous error disappeared and I'm able to login but the GUI is still loading :-(

[root@cbd ~]# uptime
 16:13:25 up 13 min,  4 users,  load average: 0.19, 0.19, 0.21

[root@cbd ~]# free -m
              total        used        free      shared  buff/cache   available
Mem:           5807        2588        2435          54         783        2948
Swap:             0           0           0
afederic commented 9 years ago

These is (I guess) the relevant log while the GUI is in the loading state

cloudbreak_1    | 2015-10-26 15:29:47,554 [taskScheduler-1] send:36 INFO  c.s.c.s.n.HttpNotificationSender - [owner:spring] [type:springLog] [id:] [name:] Could not send notification to the specified endpoint: 'http://XXX.XXX.XXX.XXX:3000/notifications' Cause: I/O error on POST request for "http://XXX.XXX.XXX.XXX:3000/notifications":Connect to XXX.XXX.XXX.XXX:3000 [/XXX.XXX.XXX.XXX] failed: Connection timed out; nested exception is org.apache.http.conn.ConnectTimeoutException: Connect to XXX.XXX.XXX.XXX:3000 [/XXX.XXX.XXX.XXX] failed: Connection timed out
lalyos commented 9 years ago

If the "Loading ..." keeps up, it means that the client side js (angular) doesn't get a response from one of the backend (uluwatu/sultans/cloudbreak/periscope). I suppose uluwatu,sultans, and cloudbreak connection is fine (because in the grayed background you see 3 blueprints)

Pleas with on the developer toolbar in your browser, and check the "network" tab, about which request takes so long.

It might be that you opened up the ports 3000, 3001 so you can reach cloudbreak ui from your browser but the micro services itself also use other ports: 8080, 8089 between each other. So please make sure that the cloudbreak container can curl for example uluwatu.

You can test the connectivity between cloudbreak and uluwatu, by

docker exec -it cbreak_cloudbreak_1 bash

curl -I http://XXX.XXX.XXX.XXX:3000/info

PRIVATE_IP

If it happens that you have a PRIVATE_IP set in your Profile, please remove it, its not necessary.

afederic commented 9 years ago

hi @lalyos SOLVED! THANK YOU VERY MUCH! :+1: the problem was that the CBD Openstack VM should connect to itself via its PUBLIC_IP (the Openstack floating IP assigned to the VM). I have added a security rule to allow all TCP incoming connections from the PUBLIC_IP and the loading has gone ;-) Do you think I should allow all UDP as well?

Thanks Ale

akanto commented 8 years ago

hi,

You don't need to open UDP.

Attila