search

cloudfoundry / bosh-openstack-cpi-release

BOSH OpenStack CPI
Apache License 2.0
36 stars 59 forks source link

bosh deploy fails while updating job blobstore_z1 > blobstore_z1/0 #48

Closed MoizArif closed 8 years ago

MoizArif commented 8 years ago

Hi all, I am deploying CF using BOSH and i am getting this error:

Error 400007: 'blobstore_z1/0 (0eb27491-096c-4506-98b8-e06763d7f996)' is not running after update. Review logs for failed jobs: blobstore_nginx

Here is the log from bosh task:

/var/vcap/packages/director/gem_home/ruby/2.3.0/gems/bosh-director-1.3262.3.0/lib/bosh/director/instance_updater/state_applier.rb:48:in `post_start'
/var/vcap/packages/director/gem_home/ruby/2.3.0/gems/bosh-director-1.3262.3.0/lib/bosh/director/instance_updater/state_applier.rb:30:in `apply'
/var/vcap/packages/director/gem_home/ruby/2.3.0/gems/bosh-director-1.3262.3.0/lib/bosh/director/instance_updater.rb:109:in `block in update'
/var/vcap/packages/director/gem_home/ruby/2.3.0/gems/bosh-director-1.3262.3.0/lib/bosh/director/instance_updater/instance_state.rb:5:in `with_instance_update'
/var/vcap/packages/director/gem_home/ruby/2.3.0/gems/bosh-director-1.3262.3.0/lib/bosh/director/instance_updater/instance_state.rb:11:in `with_instance_update_and_event_creation'
/var/vcap/packages/director/gem_home/ruby/2.3.0/gems/bosh-director-1.3262.3.0/lib/bosh/director/instance_updater.rb:111:in `update'
/var/vcap/packages/director/gem_home/ruby/2.3.0/gems/bosh-director-1.3262.3.0/lib/bosh/director/job_updater.rb:113:in `block (2 levels) in update_canary_instance'
/var/vcap/packages/director/gem_home/ruby/2.3.0/gems/bosh_common-1.3262.3.0/lib/common/thread_formatter.rb:49:in `with_thread_name'
/var/vcap/packages/director/gem_home/ruby/2.3.0/gems/bosh-director-1.3262.3.0/lib/bosh/director/job_updater.rb:111:in `block in update_canary_instance'
/var/vcap/packages/director/gem_home/ruby/2.3.0/gems/bosh-director-1.3262.3.0/lib/bosh/director/event_log.rb:99:in `advance_and_track'
/var/vcap/packages/director/gem_home/ruby/2.3.0/gems/bosh-director-1.3262.3.0/lib/bosh/director/job_updater.rb:110:in `update_canary_instance'
/var/vcap/packages/director/gem_home/ruby/2.3.0/gems/bosh-director-1.3262.3.0/lib/bosh/director/job_updater.rb:105:in `block (2 levels) in update_canaries'
/var/vcap/packages/director/gem_home/ruby/2.3.0/gems/bosh_common-1.3262.3.0/lib/common/thread_pool.rb:77:in `block (2 levels) in create_thread'
/var/vcap/packages/director/gem_home/ruby/2.3.0/gems/bosh_common-1.3262.3.0/lib/common/thread_pool.rb:63:in `loop'
/var/vcap/packages/director/gem_home/ruby/2.3.0/gems/bosh_common-1.3262.3.0/lib/common/thread_pool.rb:63:in `block in create_thread'
/var/vcap/packages/director/gem_home/ruby/2.3.0/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:323:in `block in create_with_logging_context'
E, [2016-07-28 22:29:47 #19665] [] ERROR -- DirectorJobRunner: Worker thread raised exception: 'blobstore_z1/0 (0eb27491-096c-4506-98b8-e06763d7f996)' is not running after update. Review logs for failed jobs: blobstore_nginx - /var/vcap/packages/director/gem_home/ruby/2.3.0/gems/bosh-director-1.3262.3.0/lib/bosh/director/instance_updater/state_applier.rb:48:in `post_start'
/var/vcap/packages/director/gem_home/ruby/2.3.0/gems/bosh-director-1.3262.3.0/lib/bosh/director/instance_updater/state_applier.rb:30:in `apply'
/var/vcap/packages/director/gem_home/ruby/2.3.0/gems/bosh-director-1.3262.3.0/lib/bosh/director/instance_updater.rb:109:in `block in update'
/var/vcap/packages/director/gem_home/ruby/2.3.0/gems/bosh-director-1.3262.3.0/lib/bosh/director/instance_updater/instance_state.rb:5:in `with_instance_update'
/var/vcap/packages/director/gem_home/ruby/2.3.0/gems/bosh-director-1.3262.3.0/lib/bosh/director/instance_updater/instance_state.rb:11:in `with_instance_update_and_event_creation'
/var/vcap/packages/director/gem_home/ruby/2.3.0/gems/bosh-director-1.3262.3.0/lib/bosh/director/instance_updater.rb:111:in `update'
/var/vcap/packages/director/gem_home/ruby/2.3.0/gems/bosh-director-1.3262.3.0/lib/bosh/director/job_updater.rb:113:in `block (2 levels) in update_canary_instance'
/var/vcap/packages/director/gem_home/ruby/2.3.0/gems/bosh_common-1.3262.3.0/lib/common/thread_formatter.rb:49:in `with_thread_name'
/var/vcap/packages/director/gem_home/ruby/2.3.0/gems/bosh-director-1.3262.3.0/lib/bosh/director/job_updater.rb:111:in `block in update_canary_instance'
/var/vcap/packages/director/gem_home/ruby/2.3.0/gems/bosh-director-1.3262.3.0/lib/bosh/director/event_log.rb:99:in `advance_and_track'
/var/vcap/packages/director/gem_home/ruby/2.3.0/gems/bosh-director-1.3262.3.0/lib/bosh/director/job_updater.rb:110:in `update_canary_instance'
/var/vcap/packages/director/gem_home/ruby/2.3.0/gems/bosh-director-1.3262.3.0/lib/bosh/director/job_updater.rb:105:in `block (2 levels) in update_canaries'
/var/vcap/packages/director/gem_home/ruby/2.3.0/gems/bosh_common-1.3262.3.0/lib/common/thread_pool.rb:77:in `block (2 levels) in create_thread'
/var/vcap/packages/director/gem_home/ruby/2.3.0/gems/bosh_common-1.3262.3.0/lib/common/thread_pool.rb:63:in `loop'
/var/vcap/packages/director/gem_home/ruby/2.3.0/gems/bosh_common-1.3262.3.0/lib/common/thread_pool.rb:63:in `block in create_thread'
/var/vcap/packages/director/gem_home/ruby/2.3.0/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:323:in `block in create_with_logging_context'
D, [2016-07-28 22:29:47 #19665] [] DEBUG -- DirectorJobRunner: Thread is no longer needed, cleaning up

output of bosh vms command:


+---------------------------------------------------------------------------+---------+-----+-----------+---------------+
| VM                                                                        | State   | AZ  | VM Type   | IPs           |
+---------------------------------------------------------------------------+---------+-----+-----------+---------------+
| api_z1/0 (c26b3732-7294-43d6-8eb7-e773dd84824f)                           | running | n/a | large_z1  | 10.10.0.104   |
| blobstore_z1/0 (0eb27491-096c-4506-98b8-e06763d7f996)                     | failing | n/a | medium_z1 | 10.10.0.102   |
| consul_z1/0 (517d10bf-637a-470f-97ac-48ecc5549e9c)                        | running | n/a | small_z1  | 10.10.0.137   |
| doppler_z1/0 (6c672830-e1c0-4998-8d20-06e29be5a733)                       | running | n/a | medium_z1 | 10.10.0.107   |
| etcd_z1/0 (28e6d628-5b77-4d3f-8882-865a377e9946)                          | running | n/a | medium_z1 | 10.10.0.133   |
| ha_proxy_z1/0 (77732d60-ac2e-4be0-bfa1-59ff5f75ce37)                      | running | n/a | router_z1 | 192.168.14.90 |
|                                                                           |         |     |           | 10.10.0.125   |
| hm9000_z1/0 (e2f9d941-e8a2-49cb-a3ed-59c8f8cecf30)                        | running | n/a | medium_z1 | 10.10.0.105   |
| loggregator_trafficcontroller_z1/0 (39ab5e4f-0a8a-4cbd-bc5d-25c35420a362) | running | n/a | small_z1  | 10.10.0.108   |
| nats_z1/0 (d5829968-5616-4d51-bb32-d5f906273aae)                          | running | n/a | medium_z1 | 10.10.0.127   |
| postgres_z1/0 (3065a829-4f96-433d-a76b-dcc1ecf30c03)                      | running | n/a | medium_z1 | 10.10.0.129   |
| router_z1/0 (b3c8d635-47ee-41e7-b38b-28690eb1167d)                        | running | n/a | router_z1 | 10.10.0.130   |
| runner_z1/0 (2fd481fa-dc8b-458c-8d68-ae9f7a542d71)                        | running | n/a | runner_z1 | 10.10.0.106   |
| stats_z1/0 (6279e637-7c4d-4355-8752-c6f45641dc2e)                         | running | n/a | small_z1  | 10.10.0.101   |
| uaa_z1/0 (17488c94-2fa5-4085-8f2f-54bf67ec5059)                           | running | n/a | medium_z1 | 10.10.0.103   |
+---------------------------------------------------------------------------+---------+-----+-----------+---------------+

output of bosh instances --ps

+----------------------------------------------------------------------------+---------+-----+-----------+---------------+
| Instance                                                                   | State   | AZ  | VM Type   | IPs           |
+----------------------------------------------------------------------------+---------+-----+-----------+---------------+
| api_z1/0 (c26b3732-7294-43d6-8eb7-e773dd84824f)*                           | running | n/a | large_z1  | 10.10.0.104   |
+----------------------------------------------------------------------------+---------+-----+-----------+---------------+
| blobstore_z1/0 (0eb27491-096c-4506-98b8-e06763d7f996)*                     | failing | n/a | medium_z1 | 10.10.0.102   |
|   consul_agent                                                             | running |     |           |               |
|   metron_agent                                                             | running |     |           |               |
|   blobstore_nginx                                                          | unknown |     |           |               |
|   blobstore_url_signer                                                     | running |     |           |               |
|   route_registrar                                                          | running |     |           |               |
+----------------------------------------------------------------------------+---------+-----+-----------+---------------+
| consul_z1/0 (517d10bf-637a-470f-97ac-48ecc5549e9c)*                        | running | n/a | small_z1  | 10.10.0.137   |
|   consul_agent                                                             | running |     |           |               |
|   metron_agent                                                             | running |     |           |               |
+----------------------------------------------------------------------------+---------+-----+-----------+---------------+
| doppler_z1/0 (6c672830-e1c0-4998-8d20-06e29be5a733)*                       | running | n/a | medium_z1 | 10.10.0.107   |
+----------------------------------------------------------------------------+---------+-----+-----------+---------------+
| etcd_z1/0 (28e6d628-5b77-4d3f-8882-865a377e9946)*                          | running | n/a | medium_z1 | 10.10.0.133   |
|   etcd                                                                     | running |     |           |               |
|   etcd_metrics_server                                                      | running |     |           |               |
|   metron_agent                                                             | running |     |           |               |
+----------------------------------------------------------------------------+---------+-----+-----------+---------------+
| ha_proxy_z1/0 (77732d60-ac2e-4be0-bfa1-59ff5f75ce37)*                      | running | n/a | router_z1 | 192.168.14.90 |
|                                                                            |         |     |           | 10.10.0.125   |
|   consul_template                                                          | running |     |           |               |
|   haproxy_config                                                           | running |     |           |               |
|   haproxy                                                                  | running |     |           |               |
|   metron_agent                                                             | running |     |           |               |
|   consul_agent                                                             | running |     |           |               |
+----------------------------------------------------------------------------+---------+-----+-----------+---------------+
| hm9000_z1/0 (e2f9d941-e8a2-49cb-a3ed-59c8f8cecf30)*                        | running | n/a | medium_z1 | 10.10.0.105   |
+----------------------------------------------------------------------------+---------+-----+-----------+---------------+
| loggregator_trafficcontroller_z1/0 (39ab5e4f-0a8a-4cbd-bc5d-25c35420a362)* | running | n/a | small_z1  | 10.10.0.108   |
+----------------------------------------------------------------------------+---------+-----+-----------+---------------+
| nats_z1/0 (d5829968-5616-4d51-bb32-d5f906273aae)*                          | running | n/a | medium_z1 | 10.10.0.127   |
|   nats                                                                     | running |     |           |               |
|   nats_stream_forwarder                                                    | running |     |           |               |
|   metron_agent                                                             | running |     |           |               |
+----------------------------------------------------------------------------+---------+-----+-----------+---------------+
| postgres_z1/0 (3065a829-4f96-433d-a76b-dcc1ecf30c03)*                      | running | n/a | medium_z1 | 10.10.0.129   |
+----------------------------------------------------------------------------+---------+-----+-----------+---------------+
| router_z1/0 (b3c8d635-47ee-41e7-b38b-28690eb1167d)*                        | running | n/a | router_z1 | 10.10.0.130   |
+----------------------------------------------------------------------------+---------+-----+-----------+---------------+
| runner_z1/0 (2fd481fa-dc8b-458c-8d68-ae9f7a542d71)*                        | running | n/a | runner_z1 | 10.10.0.106   |
+----------------------------------------------------------------------------+---------+-----+-----------+---------------+
| stats_z1/0 (6279e637-7c4d-4355-8752-c6f45641dc2e)*                         | running | n/a | small_z1  | 10.10.0.101   |
|   collector                                                                | running |     |           |               |
|   metron_agent                                                             | running |     |           |               |
+----------------------------------------------------------------------------+---------+-----+-----------+---------------+
| uaa_z1/0 (17488c94-2fa5-4085-8f2f-54bf67ec5059)*                           | running | n/a | medium_z1 | 10.10.0.103   |
+----------------------------------------------------------------------------+---------+-----+-----------+---------------+

(*) Bootstrap node

Instances total: 14

can anyone help me with this issue?

cf-gitbot commented 8 years ago

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/127329291

The labels on this github issue will be updated when the story is started.

MoizArif commented 8 years ago

From monit logs inside blobstore_z1/0 VM:

[UTC Jul 29 03:49:56] info     : 'blobstore_nginx' start: /var/vcap/jobs/blobstore/bin/nginx_ctl
[UTC Jul 29 03:49:58] error    : 'blobstore_nginx' failed to start
[UTC Jul 29 03:50:02] error    : 'blobstore_nginx' process is not running
[UTC Jul 29 03:50:02] info     : 'blobstore_nginx' trying to restart

and starting it manually gives: /var/vcap/jobs/blobstore/bin/nginx_ctl: line 15: /nginx.pid: Permission denied

MoizArif commented 8 years ago

Solved this, changed tls port from 443 to 4443 as per release docs for cf release 239.