search

cloudfoundry-community-attic / cf-services-contrib-release

release repository for community contributed services
Apache License 2.0
46 stars 60 forks source link

Error 400007: `rabbit_node/0' is not running after update #124

Closed betht1220 closed 10 years ago

betht1220 commented 10 years ago

I tried deploying multiple times, and rabbit_node/0 always failed to start - Can you please take a look - Let me know if you need more info. Thanks.

Started updating job rabbit_gateway > rabbit_gateway/0 (canary) Started updating job rabbit_node > rabbit_node/0 (canary) Done updating job rabbit_gateway > rabbit_gateway/0 (canary) (00:00:12) Done updating job rabbit_node > rabbit_node/0 (canary) (00:04:00)

Error 400007: `rabbit_node/0' is not running after update

rkoster commented 10 years ago

@betht1220 which version of cf are you using? and can you share your deployment manifest used to deploy rabbit via a gist (with redacted password, etc)?

betht1220 commented 10 years ago

Ruben Sure. Followed is the info. I don't know how to use gist, if you can let me know how to do this. In the meanwhile, I'll just attach the file, and find out more about gist. Thanks, Beth cf-warden | cf/169

On Wed, May 7, 2014 at 4:23 AM, Ruben Koster notifications@github.comwrote:

@betht1220https://urldefense.proofpoint.com/v1/url?u=https://github.com/betht1220&k=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0A&r=EH73v7FJlcM19VsLmWgWSA%3D%3D%0A&m=5DV2AuegcDAB7n0mBjjXJaI%2Bw2%2FP2eO%2Fc5DDL%2B9Az8E%3D%0A&s=f5d19933bc22bf5877a197d861f50ab3c504eec90dd4d6e0779b523c4e07bae8which version of cf are you using? and can you share your deployment manifest used to deploy rabbit via a gist (with redacted password, etc)?

— Reply to this email directly or view it on GitHubhttps://urldefense.proofpoint.com/v1/url?u=https://github.com/cloudfoundry-community/cf-services-contrib-release/issues/124%23issuecomment-42415467&k=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0A&r=EH73v7FJlcM19VsLmWgWSA%3D%3D%0A&m=5DV2AuegcDAB7n0mBjjXJaI%2Bw2%2FP2eO%2Fc5DDL%2B9Az8E%3D%0A&s=b48ad004e8d3104686bfd409e4ef165a145ff8c0c9769115b6cd9d3f249456d6 .

rkoster commented 10 years ago

@betht1220 you can create a gist by going to https://gist.github.com.

betht1220 commented 10 years ago

Here's the gist - https://gist.github.com/betht1220/089a88e6d8a2cab9df7a

Thanks Beth

On Wed, May 7, 2014 at 4:34 AM, Beth Tran btran@gopivotal.com wrote:

Ruben Sure. Followed is the info. I don't know how to use gist, if you can let me know how to do this. In the meanwhile, I'll just attach the file, and find out more about gist. Thanks, Beth cf-warden | cf/169

On Wed, May 7, 2014 at 4:23 AM, Ruben Koster notifications@github.comwrote:

@betht1220https://urldefense.proofpoint.com/v1/url?u=https://github.com/betht1220&k=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0A&r=EH73v7FJlcM19VsLmWgWSA%3D%3D%0A&m=5DV2AuegcDAB7n0mBjjXJaI%2Bw2%2FP2eO%2Fc5DDL%2B9Az8E%3D%0A&s=f5d19933bc22bf5877a197d861f50ab3c504eec90dd4d6e0779b523c4e07bae8which version of cf are you using? and can you share your deployment manifest used to deploy rabbit via a gist (with redacted password, etc)?

— Reply to this email directly or view it on GitHubhttps://urldefense.proofpoint.com/v1/url?u=https://github.com/cloudfoundry-community/cf-services-contrib-release/issues/124%23issuecomment-42415467&k=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0A&r=EH73v7FJlcM19VsLmWgWSA%3D%3D%0A&m=5DV2AuegcDAB7n0mBjjXJaI%2Bw2%2FP2eO%2Fc5DDL%2B9Az8E%3D%0A&s=b48ad004e8d3104686bfd409e4ef165a145ff8c0c9769115b6cd9d3f249456d6 .

betht1220 commented 10 years ago

Ruben Thanks, I've just sent you the gist. Cheers Beth

On Wed, May 7, 2014 at 4:35 AM, Ruben Koster notifications@github.comwrote:

@betht1220https://urldefense.proofpoint.com/v1/url?u=https://github.com/betht1220&k=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0A&r=EH73v7FJlcM19VsLmWgWSA%3D%3D%0A&m=mZKPlUZXUqWomTID87aYKy2Hhqqu7SI8UpUs9KRbpVk%3D%0A&s=f2ce1eeab3cc68ba18893218f83eb89bef11cfb81b303742fbd760445e33a59byou can create a gist by going to https://gist.github.comhttps://urldefense.proofpoint.com/v1/url?u=https://gist.github.com&k=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0A&r=EH73v7FJlcM19VsLmWgWSA%3D%3D%0A&m=mZKPlUZXUqWomTID87aYKy2Hhqqu7SI8UpUs9KRbpVk%3D%0A&s=65527909c2326c0d85b9f3e5281d8cd4e1ffa5378646918c83c1e89c7f45f606 .

— Reply to this email directly or view it on GitHubhttps://urldefense.proofpoint.com/v1/url?u=https://github.com/cloudfoundry-community/cf-services-contrib-release/issues/124%23issuecomment-42416409&k=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0A&r=EH73v7FJlcM19VsLmWgWSA%3D%3D%0A&m=mZKPlUZXUqWomTID87aYKy2Hhqqu7SI8UpUs9KRbpVk%3D%0A&s=8283a9303c51ca6da96d6fa7ca12014f911fedf97960fd0c45714c318a3ff47e .

rkoster commented 10 years ago

Services-contrib has not yet been tested with the go_agent stemcell. This could be related. Could you try and create a manifest with ./templates/make_manifest warden. Also when this does not work can you do a bosh ssh rabbit_node and tail the logs in /var/vcap/sys/log/* for errors?

betht1220 commented 10 years ago
  1. I did generate the manifest using "./templates/make_manifest warden", and just modified the file to include only rabbitmq in this deploy. I was using the full manifest before this bosh deploy. Both times, I got all the *_node failing
  2. About your recommendation - bosh ssh to rabbit_node (the fail node) returns ssh: connect to host 10.244.3.86 port 22: Connection timed out Even when I bosh ssh to rabbit_gateway, I also got connection timed out

+------------------+---------+---------------+-------------+ | Job/index | State | Resource Pool | IPs | +------------------+---------+---------------+-------------+ | rabbit_gateway/0 | running | gateway_z1 | 10.244.3.6 | | rabbit_node/0 | failing | node_z1 | 10.244.3.86 | +------------------+---------+---------------+-------------+

-Beth

On Wed, May 7, 2014 at 4:54 AM, Ruben Koster notifications@github.comwrote:

Services-contrib has not yet been tested with the go_agent stemcellhttps://urldefense.proofpoint.com/v1/url?u=https://gist.github.com/betht1220/089a88e6d8a2cab9df7a%23file-contrib-services-warden-manifest-yml-L35&k=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0A&r=EH73v7FJlcM19VsLmWgWSA%3D%3D%0A&m=AS8ovXVfntCKgbHgabxqYSkPfVCQe9qq0vgQmhyVX4c%3D%0A&s=89c6d089a8f72d395b01a75f205d8f750634ffdb1d3a8fb3d05511b4d7e35784. This could be related. Could you try and create a manifest with ./templates/make_manifest warden. Also when this does not work can you do a bosh ssh rabbit_nodeand tail the logs in /var/vcap/sys/log/* for errors?

— Reply to this email directly or view it on GitHubhttps://urldefense.proofpoint.com/v1/url?u=https://github.com/cloudfoundry-community/cf-services-contrib-release/issues/124%23issuecomment-42417831&k=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0A&r=EH73v7FJlcM19VsLmWgWSA%3D%3D%0A&m=AS8ovXVfntCKgbHgabxqYSkPfVCQe9qq0vgQmhyVX4c%3D%0A&s=bd313e69b57be10339ffa3e994fb6a9ec2f18f2e197373c4bf4a489e53ef5e26 .

rkoster commented 10 years ago

Please follow the instructions here (run: scripts/add-route)

betht1220 commented 10 years ago

Ruben Yes, I've done that too, and still got the same issue. BTW, I have deployed mysql on cf-169 using bosh-lite, and it's working just fine-. I've tried to look at the output "bosh task 433 --debug", but could not find any helping errors. Here's the gist for the output of "bosh task..." - https://gist.github.com/betht1220/e92cf0ea235a98aef190 -Beth

On Wed, May 7, 2014 at 5:05 AM, Ruben Koster notifications@github.comwrote:

Please follow the instructions herehttps://urldefense.proofpoint.com/v1/url?u=https://github.com/cloudfoundry/bosh-lite%23ssh-into-deployment-jobs&k=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0A&r=EH73v7FJlcM19VsLmWgWSA%3D%3D%0A&m=qxucwe%2BA%2BUaxOpqgFcEiO6UWZaJ5pumanfY%2FmvGtWsM%3D%0A&s=5d77ff3430c42d786d93d95707211fed60a96edbc015343c47b44711b229a2c9(run: scripts/add-route)

— Reply to this email directly or view it on GitHubhttps://urldefense.proofpoint.com/v1/url?u=https://github.com/cloudfoundry-community/cf-services-contrib-release/issues/124%23issuecomment-42418675&k=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0A&r=EH73v7FJlcM19VsLmWgWSA%3D%3D%0A&m=qxucwe%2BA%2BUaxOpqgFcEiO6UWZaJ5pumanfY%2FmvGtWsM%3D%0A&s=8c2f04b03743b5279dec522b3e8bc1f230a35da5fec9fda64e17fe3d1b2d61bd .

rkoster commented 10 years ago

I just realised I forgot to ask which services-contrib release version you are using. Because all the efforts on getting cf-services-contrib to work with warden happend after the last final release. If you indeed are using the final release you have to options:

  1. wait for a the v5 final release (will create one after I have tested and merged https://github.com/cloudfoundry-community/cf-services-contrib-release/pull/120)
  2. create a dev release by following the instructions I just added to the readme.
betht1220 commented 10 years ago

Ruben I am able to get into the failing rabbit node, and here's the log file - https://gist.github.com/betht1220/6ab8119bffab8d03c900 Thanks

On Wed, May 7, 2014 at 10:02 AM, Beth Tran btran@gopivotal.com wrote:

Ruben Yes, I've done that too, and still got the same issue. BTW, I have deployed mysql on cf-169 using bosh-lite, and it's working just fine-. I've tried to look at the output "bosh task 433 --debug", but could not find any helping errors. Here's the gist for the output of "bosh task..." - https://gist.github.com/betht1220/e92cf0ea235a98aef190 -Beth

On Wed, May 7, 2014 at 5:05 AM, Ruben Koster notifications@github.comwrote:

Please follow the instructions herehttps://urldefense.proofpoint.com/v1/url?u=https://github.com/cloudfoundry/bosh-lite%23ssh-into-deployment-jobs&k=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0A&r=EH73v7FJlcM19VsLmWgWSA%3D%3D%0A&m=qxucwe%2BA%2BUaxOpqgFcEiO6UWZaJ5pumanfY%2FmvGtWsM%3D%0A&s=5d77ff3430c42d786d93d95707211fed60a96edbc015343c47b44711b229a2c9(run: scripts/add-route)

— Reply to this email directly or view it on GitHubhttps://urldefense.proofpoint.com/v1/url?u=https://github.com/cloudfoundry-community/cf-services-contrib-release/issues/124%23issuecomment-42418675&k=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0A&r=EH73v7FJlcM19VsLmWgWSA%3D%3D%0A&m=qxucwe%2BA%2BUaxOpqgFcEiO6UWZaJ5pumanfY%2FmvGtWsM%3D%0A&s=8c2f04b03743b5279dec522b3e8bc1f230a35da5fec9fda64e17fe3d1b2d61bd .

rkoster commented 10 years ago

@betht1220 This really looks like an issue which has been solved in master. Have you tried using a dev-release (as suggested in my previous comment)?

betht1220 commented 10 years ago

Ruben I tried to create a dev release, but the generated manifest for the dev release needs to be modified before I can do "bosh deploy" Here's the gist for the generated manifest - and it looks very different from the one generated by "make_manifest warden". Generated manifest - https://gist.github.com/betht1220/664c623f661c667f793f Can you give me some tips? Thanks, Beth

On Fri, May 9, 2014 at 3:34 AM, Ruben Koster notifications@github.comwrote:

@betht1220https://urldefense.proofpoint.com/v1/url?u=https://github.com/betht1220&k=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0A&r=EH73v7FJlcM19VsLmWgWSA%3D%3D%0A&m=a9rbp2WEfnQ5p41CCAIigylIjAiSErfaMsniE3xHTEs%3D%0A&s=9c8b6ad4b91fd85f959c8b83341845c497f0ad1e907ab13aad9143b0cf0e590aThis really looks like an issue which has been solved in master. Have you tried using a dev-release (as suggested in my previous comment)?

— Reply to this email directly or view it on GitHubhttps://urldefense.proofpoint.com/v1/url?u=https://github.com/cloudfoundry-community/cf-services-contrib-release/issues/124%23issuecomment-42652926&k=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0A&r=EH73v7FJlcM19VsLmWgWSA%3D%3D%0A&m=a9rbp2WEfnQ5p41CCAIigylIjAiSErfaMsniE3xHTEs%3D%0A&s=5ab82bda40fadd96e994dfdf85f75f400417d6e77724f1132dc6f968233cc844 .

rkoster commented 10 years ago

@betht1220 you can upload the generated release by running bosh upload release. Optionally you can pass in the path to the generated release manifest. After the release has been uploaded you can deploy with the deployment manifest you generated with make_manifest warden.

betht1220 commented 10 years ago

Ruben Thanks for helping me out, it's working now with the release created from the master branch. Is this release written in ruby, or ? I'm learning to develop in bosh, can you kindly give me some background info, URLs, etc. Thank you Ruben. Beth

On Fri, May 9, 2014 at 11:47 AM, Ruben Koster notifications@github.comwrote:

@betht1220https://urldefense.proofpoint.com/v1/url?u=https://github.com/betht1220&k=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0A&r=EH73v7FJlcM19VsLmWgWSA%3D%3D%0A&m=qO7x9arWd2lTakyhVRtBo9VgBSHWizExAsetUAZn884%3D%0A&s=1d5cce79e926b970fe5903df362e6ba3dcb49912d82dc681e6922ed6c22808b3you can upload the generated release by running bosh upload release. Optionally you can pass in the path to the generated release manifesthttps://urldefense.proofpoint.com/v1/url?u=https://gist.github.com/betht1220/664c623f661c667f793f&k=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0A&r=EH73v7FJlcM19VsLmWgWSA%3D%3D%0A&m=qO7x9arWd2lTakyhVRtBo9VgBSHWizExAsetUAZn884%3D%0A&s=d719f31babc6891f5b381641bf7c1d0c7d53d12634ac535e8c92a273cb7c238a. After the release has been uploaded you can deploy with the deployment manifest you generated with make_manifest warden.

— Reply to this email directly or view it on GitHubhttps://urldefense.proofpoint.com/v1/url?u=https://github.com/cloudfoundry-community/cf-services-contrib-release/issues/124%23issuecomment-42700219&k=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0A&r=EH73v7FJlcM19VsLmWgWSA%3D%3D%0A&m=qO7x9arWd2lTakyhVRtBo9VgBSHWizExAsetUAZn884%3D%0A&s=d034f8701784b93a6330317b9511264bc831f0b9bc852e29fa4e7c87fb8073d2 .

rkoster commented 10 years ago

@betht1220 I'm glad its working. Bosh releases are mostly written in shell script + ruby erb templates. You can find documentation on how to write bosh-releases here.

betht1220 commented 10 years ago

Ruben I'm now working with mongodb. The VCAP_SERVICES for my application does show this json. But when I use mongodb client to login using the credentials in VCAP_SERVICES, it has error exception: login failed. Thanks, Beth

  1. mongo client command generates error mongo 10.244.3.82:10002/3067972e-7618-4850-8433-8fc9400df278 -u 81e24c03-8455-4f4b-ad9d-17d7e3a83b62 -p 21702bad-c512-4516-a1f6-d9c753c032cc
  2. VCAP_SERVICES for mongo db

"credentials": { "hostname": "10.244.3.82", "host": "10.244.3.82", "port": 10002, "username": "81e24c03-8455-4f4b-ad9d-17d7e3a83b62", "password": "21702bad-c512-4516-a1f6-d9c753c032cc", "name": "3067972e-7618-4850-8433-8fc9400df278", "db": "db", "url": "mongodb://81e24c03-8455-4f4b-ad9d-17d7e3a83b62:21702bad-c512-4516-a1f6-d9c753c032cc@10.244.3.82:10002/db" }

betht1220 commented 10 years ago

Ruben Also, if I use this command (replacing the db string and use "db") - I got error command= mongo 10.244.3.82:10002/db -u 81e24c03-8455-4f4b-ad9d-17d7e3a83b62 -p 21702bad-c512-4516-a1f6-d9c753c032cc error = Error while trying to show server startup warnings: need to login -Beth