search

cloudfoundry / cf-mysql-release

Cloud Foundry MySQL Release
Apache License 2.0
58 stars 106 forks source link

Can't create-service on bosh-lite #138

Closed Omnipresent closed 7 years ago

Omnipresent commented 8 years ago

I've setup boshlite and CF on AWS. I've setup the mysql service by following the README. I've gotten some success but I can't seem to create-service.

I can see the service-brokers

ubuntu@agent-id-bosh-0:~$ cf service-brokers
Getting service brokers as admin...

name         url
p-mysql      https://p-mysql.bosh-lite.com:443
p-rabbitmq   https://pivotal-rabbitmq-broker.bosh-lite.com

I can see the services in marketplace

ubuntu@agent-id-bosh-0:~$ cf marketplace
Getting services from marketplace in org aerstone / space testing as admin...
OK

service      plans        description
p-mysql      10mb, 20mb   MySQL databases on demand
p-rabbitmq   standard     RabbitMQ is a robust and scalable high-performance multi-protocol messaging broker.

TIP:  Use 'cf marketplace -s SERVICE' to view descriptions of individual plans of a given service.

However, when I try to create-service I get an error

ubuntu@agent-id-bosh-0:~$ cf create-service p-mysql 20mb mysql
Creating service instance mysql in org aerstone / space testing as admin...
FAILED
Server error, status code: 502, error code: 10001, message: The service broker rejected the request to https://p-mysql.bosh-lite.com/v2/service_instances/6f040f45-6e61-4bec-9381-78d298ec57ce?accepts_incomplete=true. Status Code: 404 Not Found, Body: 404 Not Found: Requested route ('p-mysql.bosh-lite.com') does not exist.

I've also created security-group with rule.json

ubuntu@agent-id-bosh-0:~$ cf security-groups
Getting security groups as admin
OK

     Name                    Organization   Space
#0   public_networks
#1   dns
#2   services
#3   load_balancer
#4   user_bosh_deployments
#5   p-mysql
ubuntu@agent-id-bosh-0:~$ more rule.json
[
      {
          "destination": "10.10.163.1-10.10.163.255",
          "protocol": "all"
      },
      {
          "destination": "10.10.164.1-10.10.164.255",
          "protocol": "all"
      },
      {
          "destination": "10.10.165.1-10.10.165.255",
          "protocol": "all"
      }
  ]

Some more details about my environment

ubuntu@agent-id-bosh-0:~$ bosh releases
RSA 1024 bit CA certificates are loaded due to old openssl compatibility
Acting as user 'admin' on 'Bosh Lite Director'

+-------------+----------------+-------------+
| Name        | Versions       | Commit Hash |
+-------------+----------------+-------------+
| cf          | 248+dev.4      | 594e989f    |
|             | 248+dev.5      | 594e989f    |
|             | 248+dev.6*     | 594e989f    |
| cf-mysql    | 32*            | 6c0314b     |
| cf-rabbitmq | 224.0.0+dev.1* | 108d05b2+   |
| routing     | 0.136.0*       | d29132da+   |
+-------------+----------------+-------------+
(*) Currently deployed
(+) Uncommitted changes

Releases total: 4

BOSH vms shows everything as running except the following

+-------------------------------------------------------------+--------------------+-----+--------------------+------------+
| VM                                                          | State              | AZ  | VM Type            | IPs        |
+-------------------------------------------------------------+--------------------+-----+--------------------+------------+
| arbitrator_z3/0 (39066f55-3515-4098-adc5-7bfc7ca6b9d2)      | unresponsive agent | n/a | arbitrator_z3      |            |
| cf-mysql-broker_z1/0 (5632d437-d0b2-4688-93a1-a3a2b4126390) | unresponsive agent | n/a | cf-mysql-broker_z1 |            |
| cf-mysql-broker_z2/0 (0241e416-3bde-48ed-b51e-4d032b14640a) | unresponsive agent | n/a | cf-mysql-broker_z2 |            |
| mysql_z1/0 (a8e4916e-0258-446e-9404-330684e72fab)           | unresponsive agent | n/a | mysql_z1           |            |
| mysql_z2/0 (5bcd3148-0144-4cfe-b7e8-b0ee90abab2e)           | running            | n/a | mysql_z2           | 10.244.8.2 |
| proxy_z1/0 (ea646869-b8e4-4a44-92c2-1cfb25a18b4b)           | unresponsive agent | n/a | proxy_z1           |            |
| proxy_z2/0 (51ebc12c-2b6d-44c3-9f3d-09c049a4e032)           | unresponsive agent | n/a | proxy_z2           |            |
+-------------------------------------------------------------+--------------------+-----+--------------------+------------+

VMs total: 7

Should I start the above? If so, how?

cf-gitbot commented 8 years ago

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/135479623

The labels on this github issue will be updated when the story is started.

stickfigure commented 8 years ago

Hey @Omnipresent,

The 404 is coming from the cf router, basically saying it can't find a route to the service broker. The service broker instances don't seem to be running, so that would be the expected behavior.

I'm not sure how you only have one of the mysql vms up and running, but that's probably the root of the problem. Can you do a bosh cloudcheck and see if recreating those vms with unresponsive agent brings everything back online?

Omnipresent commented 8 years ago

Hi @stickfigure

I am on AWS with bosh lite. The VMs probably didn't come back up after I did vagrant halt and then vagrant up. However, to eliminate any start/stop issues I set everything up on a new EC2 instance and have not stopped the machine. I followed the README again and it looks like cf-mysql-broker_z1/0 still fails to start

$ bosh vms

Deployment 'cf-warden-mysql'

Director task 25

Task 25 done

+-------------------------------------------------------------+---------+-----+--------------------+------------+
| VM                                                          | State   | AZ  | VM Type            | IPs        |
+-------------------------------------------------------------+---------+-----+--------------------+------------+
| arbitrator_z3/0 (69ac28a5-4175-445a-8330-da497ed5d7bd)      | running | n/a | arbitrator_z3      | 10.244.9.6 |
| cf-mysql-broker_z1/0 (5dccc948-9ecc-4e9a-bb61-f98cb409daba) | failing | n/a | cf-mysql-broker_z1 | 10.244.7.4 |
| cf-mysql-broker_z2/0 (8d0695d9-e48c-48f8-b388-627a587b77f4) | running | n/a | cf-mysql-broker_z2 | 10.244.8.4 |
| mysql_z1/0 (16c2179e-ed20-404f-a73d-2cef8f268943)           | running | n/a | mysql_z1           | 10.244.7.2 |
| mysql_z2/0 (eed4a501-c124-4404-bd2a-1a1ee7b472ad)           | running | n/a | mysql_z2           | 10.244.8.2 |
| proxy_z1/0 (559b5c85-2a79-4c95-9f9e-f55acfd3ebf1)           | running | n/a | proxy_z1           | 10.244.7.3 |
| proxy_z2/0 (0e8f4fa9-0fb5-4eb2-b513-f0020d822a3c)           | running | n/a | proxy_z2           | 10.244.8.3 |
+-------------------------------------------------------------+---------+-----+--------------------+------------+

The error is: 

E, [2016-12-06 04:11:35 #17758] [task:17] ERROR -- DirectorJobRunner: 'cf-mysql-broker_z1/0 (5dccc948-9ecc-4e9a-bb61-f98cb409daba)' is not running after update. Review logs for failed jobs: cf-mysql-broker, quota-enforcer
/var/vcap/packages/director/gem_home/ruby/2.3.0/gems/bosh-director-1.3262.3.0/lib/bosh/director/instance_updater/state_applier.rb:48:in `post_start'
/var/vcap/packages/director/gem_home/ruby/2.3.0/gems/bosh-director-1.3262.3.0/lib/bosh/director/instance_updater/state_applier.rb:30:in `apply'
/var/vcap/packages/director/gem_home/ruby/2.3.0/gems/bosh-director-1.3262.3.0/lib/bosh/director/instance_updater.rb:109:in `block in update'

Any idea what this means and how to resolve it? Running cloudcheck or cck doesn't find anything. 

~/workspace/cf-mysql-release$ bosh cck
Acting as user 'admin' on deployment 'cf-warden-mysql' on 'Bosh Lite Director'
Performing cloud check...

Director task 26
  Started scanning 7 vms
  Started scanning 7 vms > Checking VM states. Done (00:00:00)
  Started scanning 7 vms > 7 OK, 0 unresponsive, 0 missing, 0 unbound. Done (00:00:00)
     Done scanning 7 vms (00:00:00)

  Started scanning 2 persistent disks
  Started scanning 2 persistent disks > Looking for inactive disks. Done (00:00:00)
  Started scanning 2 persistent disks > 2 OK, 0 missing, 0 inactive, 0 mount-info mismatch. Done (00:00:00)
     Done scanning 2 persistent disks (00:00:00)

Task 26 done

Started     2016-12-06 04:21:45 UTC
Finished    2016-12-06 04:21:45 UTC
Duration    00:00:00

Scan is complete, checking if any problems found...
No problems found
~/workspace/cf-mysql-release$ bosh cloudcheck
Acting as user 'admin' on deployment 'cf-warden-mysql' on 'Bosh Lite Director'
Performing cloud check...

Director task 27
  Started scanning 7 vms
  Started scanning 7 vms > Checking VM states. Done (00:00:00)
  Started scanning 7 vms > 7 OK, 0 unresponsive, 0 missing, 0 unbound. Done (00:00:00)
     Done scanning 7 vms (00:00:00)

  Started scanning 2 persistent disks
  Started scanning 2 persistent disks > Looking for inactive disks. Done (00:00:00)
  Started scanning 2 persistent disks > 2 OK, 0 missing, 0 inactive, 0 mount-info mismatch. Done (00:00:00)
     Done scanning 2 persistent disks (00:00:00)

Task 27 done

Started     2016-12-06 04:21:55 UTC
Finished    2016-12-06 04:21:55 UTC
Duration    00:00:00

Scan is complete, checking if any problems found...
No problems found

Additionally, running bosh run errand broker-registrar produces the error below

~/workspace/cf-mysql-release$ bosh run errand broker-registrar
Acting as user 'admin' on deployment 'cf-warden-mysql' on 'Bosh Lite Director'

Director task 28
Error 100: invalid deployment:

Task 28 error

Errand 'broker-registrar' did not complete

For a more detailed error report, run: bosh task 28 --debug

My deployment seems to be correct

~/workspace/cf-mysql-release$ bosh status
Config
             /home/bhaarat/.bosh_config

Director
  Name       Bosh Lite Director
  URL        https://24.17.94.106:25555
  Version    1.3262.3.0 (00000000)
  User       admin
  UUID       c9ff6012-3899-4540-bca1-77e05f5a32d0
  CPI        warden_cpi
  dns        disabled
  compiled_package_cache disabled
  snapshots  disabled

Deployment
  Manifest   /home/omni/workspace/cf-mysql-release/cf-mysql.yml

Some more details to help troubleshoot the issue.

cf-mysql.yml: https://gist.github.com/Omnipresent/4f7f7c4e27ec3000ae9355439fcb9674

jpalermo commented 7 years ago

Unfortunately BOSH is not able to surface the details of why jobs are failing to start. You'll need to download the logs to gather more info.

Run: bosh download logs cf-mysql-broker_z1 0 and extract the tgz that is downloaded.

Your output above says that both cf-mysql-broker and quota-enforcer jobs are failing.

Have a look at: ./cf-mysql-broker/cf-mysql-broker.log ./cf-mysql-broker/quota_enforcer.stderr.log ./cf-mysql-broker/quota_enforcer.stdout.log

and see if there are any clues to why the jobs are failing to start.

robdimsdale commented 7 years ago

We're closing this as there's been no response in a week; please feel free to reopen if you have logs.

Thanks,

Rob & @jpalermo