search

cloudfoundry-attic / bosh-init

bosh-init is a tool used to create and update the Director VM
Apache License 2.0
31 stars 33 forks source link

Error 450001: Action Failed get_task: Failed Jobs: cloud_controller_ng. #128

Closed gln-gln closed 6 years ago

gln-gln commented 6 years ago

Hi Team ,

I am deploying CF on Openstack using bosh . When I execute : $bosh deploy , I get the following error 

  Started updating instance consul_z1 > consul_z1/72ded18d-f9d5-4a00-ade9-d0f1d78e17c8 (0) (canary). Done (00:01:17)
  Started updating instance nats_z1 > nats_z1/847b06df-bacf-47f5-8c4a-ed7345389651 (0) (canary). Done (00:00:52)
  Started updating instance etcd_z1 > etcd_z1/12add3d4-170f-4395-b38a-b4b5eaa37a8e (0) (canary). Done (00:01:32)
  Started updating instance stats_z1 > stats_z1/58308a67-4891-4534-8891-8af3436a6f95 (0) (canary). Done (00:00:58)
  Started updating instance blobstore_z1 > blobstore_z1/931475a0-e187-430d-9ee8-23e6acc1f70d (0) (canary). Done (00:01:21)
  Started updating instance postgres_z1 > postgres_z1/8c3d2491-e691-4fdd-967f-0f158e8b3bb5 (0) (canary). Done (00:01:22)
  Started updating instance uaa_z1 > uaa_z1/bc12c108-bd00-4486-9661-fbad7ecc28da (0) (canary). Done (00:06:44)
  Started updating instance api_z1 > api_z1/284322ca-4abc-4c41-8005-599426ca71c0 (0) (canary). Failed: Action Failed get_task: Task 0cac0355-8be0-4b43-6555-03a6d4ed3ecb result: 1 of 5 pre-start scripts failed. Failed Jobs: cloud_controller_ng. Successful Jobs: route_registrar, cloud_controller_worker, cloud_controller_clock, consul_agent. (00:01:01)

Error 450001: Action Failed get_task: Task 0cac0355-8be0-4b43-6555-03a6d4ed3ecb result: 1 of 5 pre-start scripts failed. Failed Jobs: cloud_controller_ng. Successful Jobs: route_registrar, cloud_controller_worker, cloud_controller_clock, consul_agent.

Task 115 error
Error log -
{"time":1503554454,"stage":"Updating instance","tags":["uaa_z1"],"total":1,"task":"uaa_z1/bc12c108-bd00-4486-9661-fbad7ecc28da (0) (canary)","index":1,"state":"started","progress":0}
{"time":1503554858,"stage":"Updating instance","tags":["uaa_z1"],"total":1,"task":"uaa_z1/bc12c108-bd00-4486-9661-fbad7ecc28da (0) (canary)","index":1,"state":"finished","progress":100}
{"time":1503554858,"stage":"Updating instance","tags":["api_z1"],"total":1,"task":"api_z1/284322ca-4abc-4c41-8005-599426ca71c0 (0) (canary)","index":1,"state":"started","progress":0}
{"time":1503554919,"stage":"Updating instance","tags":["api_z1"],"total":1,"task":"api_z1/284322ca-4abc-4c41-8005-599426ca71c0 (0) (canary)","index":1,"state":"failed","progress":100,"data":{"error":"Action Failed get_task: Task 0cac0355-8be0-4b43-6555-03a6d4ed3ecb result: 1 of 5 pre-start scripts failed. Failed Jobs: cloud_controller_ng. Successful Jobs: route_registrar, cloud_controller_worker, cloud_controller_clock, consul_agent."}}
{"time":1503554919,"error":{"code":450001,"message":"Action Failed get_task: Task 0cac0355-8be0-4b43-6555-03a6d4ed3ecb result: 1 of 5 pre-start scripts failed. Failed Jobs: cloud_controller_ng. Successful Jobs: route_registrar, cloud_controller_worker, cloud_controller_clock, consul_agent."}}
', "result_output" = '', "context_id" = '' WHERE ("id" = 115)
D, [2017-08-24 06:08:39 #29640] [task:115] DEBUG -- DirectorJobRunner: (0.009765s) COMMIT
I, [2017-08-24 06:08:39 #29640] []  INFO -- DirectorJobRunner: Task took 19 minutes 38.601041800000075 seconds to process.

Manifest stub used: https://gist.github.com/gowriln/936dd0f488bfd981c999ed56c9016579

Any suggestions?

dpb587-pivotal commented 6 years ago

Hi - the error message suggests that the cloud_controller_ng pre-start script failed, so I suggest you start by bosh ssh'ing onto api_z1/284322ca-4abc-4c41-8005-599426ca71c0 to take a look at what was output in /var/vcap/sys/log/cloud_controller_ng/pre-start.*.log. Those logs should provide you with better information about what went wrong.

gln-gln commented 6 years ago

Hi , @dpb587-pivotal Thanks for reply. I have the logs from api_z1/284322ca-4abc-4c41-8005-599426ca71c0 in the gist - https://gist.github.com/gowriln/ed762ecabe670564a80dcc23d1e388a4 . It shows error as SchemaValidationError: { tls_port => Expected instance of Integer, given an instance of String }. Any suggestions ?

gln-gln commented 6 years ago

hi , Just an update - I noticed that variable tls_port: CC_MUTUAL_TLS_PORT in stub (https://gist.github.com/gowriln/936dd0f488bfd981c999ed56c9016579) is not configured with any value. I tried removing this and redeployed but results in a different error 

[2017-09-06 06:44:16+0000] /var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/cache/delayed_job_sequel-9ac6191b473b/lib/delayed/backend/sequel.rb:2:in `<top (required)>'
[2017-09-06 06:44:16+0000] /var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.7.1/lib/active_support/dependencies.rb:274:in `require'
[2017-09-06 06:44:16+0000] /var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.7.1/lib/active_support/dependencies.rb:274:in `block in require'
[2017-09-06 06:44:16+0000] /var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.7.1/lib/active_support/dependencies.rb:240:in `load_dependency'
[2017-09-06 06:44:16+0000] /var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.7.1/lib/active_support/dependencies.rb:274:in `require'
[2017-09-06 06:44:16+0000] SEQUEL DEPRECATION WARNING: Sequel::Model.def_dataset_method is deprecated and will be removed in Sequel 5.  Define the method inside a dataset_module block, or use the def_dataset_method_plugin.
[2017-09-06 06:44:16+0000] /var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.3.0/gems/sequel-4.49.0/lib/sequel/model/base.rb:467:in `def_dataset_method'
[2017-09-06 06:44:16+0000] /var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/cache/delayed_job_sequel-9ac6191b473b/lib/delayed/backend/sequel.rb:27:in `<class:Job>'
[2017-09-06 06:44:16+0000] /var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/cache/delayed_job_sequel-9ac6191b473b/lib/delayed/backend/sequel.rb:7:in `<module:Sequel>'
[2017-09-06 06:44:16+0000] /var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/cache/delayed_job_sequel-9ac6191b473b/lib/delayed/backend/sequel.rb:4:in `<module:Backend>'
[2017-09-06 06:44:16+0000] /var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/cache/delayed_job_sequel-9ac6191b473b/lib/delayed/backend/sequel.rb:3:in `<module:Delayed>'
[2017-09-06 06:44:16+0000] /var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/cache/delayed_job_sequel-9ac6191b473b/lib/delayed/backend/sequel.rb:2:in `<top (required)>'
[2017-09-06 06:44:16+0000] /var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.7.1/lib/active_support/dependencies.rb:274:in `require'
[2017-09-06 06:44:16+0000] /var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.7.1/lib/active_support/dependencies.rb:274:in `block in require'
[2017-09-06 06:44:16+0000] /var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.7.1/lib/active_support/dependencies.rb:240:in `load_dependency'
[2017-09-06 06:44:16+0000] /var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.3.0/gems/activesupport-4.2.7.1/lib/active_support/dependencies.rb:274:in `require'
[2017-09-06 06:44:16+0000] rake aborted!
[2017-09-06 06:44:16+0000] CloudController::Blobstore::BlobstoreError: Could not create object, 403/<html>^M
[2017-09-06 06:44:16+0000] <head><title>403 Forbidden</title></head>^M
[2017-09-06 06:44:16+0000] <body bgcolor="white">^M
[2017-09-06 06:44:16+0000] <center><h1>403 Forbidden</h1></center>^M
[2017-09-06 06:44:16+0000] <hr><center>nginx</center>^M
[2017-09-06 06:44:16+0000] </body>^M
[2017-09-06 06:44:16+0000] </html>^M
[2017-09-06 06:44:16+0000] /var/vcap/data/packages/cloud_controller_ng/c82f0888574d3acc2edeba9d00fb612341254c19/cloud_controller_ng/lib/cloud_controller/blobstore/webdav/dav_client.rb:101:in `block in cp_to_blobstore'

I am not sure if tls_port was the issue .Any suggestions ?

dpb587-pivotal commented 6 years ago

This is fairly release specific, so it is a bit outside the scope of BOSH itself. Based on the error message though, it sounds like your cloud controller may be using the wrong credentials to connect to your blobstore. I'd suggest you review the relevant settings for the capi jobs/blobstore in your deployment manifest.

I'm not sure how you're managing your release, but you may be interested in bbl or cf-deployment which is a greatly simplified way of bringing both BOSH and CloudFoundry for testing. It will probably help you avoid issues if you are manually managing your deployment manifest.

If you need further help investigating this particular issue, I'd recommend you follow up with the CAPI team responsible for the capi-release - they may be able to help via GitHub issue on that repo, or, more likely, should be able to help from the cloudfoundry#capi Slack channel if you'd like to join.