search

cloudfoundry-attic / cf-docs

A well lit place for docs
Apache License 2.0
46 stars 111 forks source link

cf target failed: 401 Unauthorized; CC: Authorization is required' (NATS::ServerError) #182

Closed iwinoto closed 11 years ago

iwinoto commented 11 years ago

Hi. I followed the steps in http://docs.cloudfoundry.com/docs/running/deploying-cf/openstack/install_cf_openstack.html. I have OpenStack Folsom with Nova networking. Deployment with micro-bosh works, but when I try to target CF, I get:

    $ cf target http://api.192.168.200.2.xip.io:8080 
    Setting target to http://api.192.168.200.2.xip.io:8080... FAILED
    CFoundry::Unauthorized: 401: 401 Unauthorized

Checking vms, the cloud_controller job is showing failing, but intermittently.

$ bosh vms
Deployment `cf-demo'

Director task 191

Task 191 done

+---------------------+---------+---------------+-------------------------------+
| Job/index           | State   | Resource Pool | IPs                           |
+---------------------+---------+---------------+-------------------------------+
| cloud_controller/0  | failing | common        | 192.168.100.8                 |
| dea/0               | running | large         | 192.168.100.11                |
| health_manager/0    | running | common        | 192.168.100.9                 |
| nats/0              | running | common        | 192.168.100.4                 |
| nfs_server/0        | running | common        | 192.168.100.6                 |
| postgres/0          | running | common        | 192.168.100.2                 |
| router/0            | running | common        | 192.168.100.10, 192.168.200.2 |
| syslog_aggregator/0 | running | common        | 192.168.100.5                 |
| uaa/0               | running | common        | 192.168.100.7                 |
+---------------------+---------+---------------+-------------------------------+

VMs total: 9

At other times I run bosh vms and all jobs show as running. I downloaded the logs for cloud_controller and in cloud_controller_ng/cloud_controller_ng.stderr.log I see a lot of entries of:

/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/nats-0.4.26/lib/nats/client.rb:567:in `block in connection_completed': 'Authorization is required' (NATS::ServerError)
        from /var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/nats-0.4.26/lib/nats/client.rb:506:in `call'
        from /var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/nats-0.4.26/lib/nats/client.rb:506:in `receive_data'
        from /var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/eventmachine-1.0.3/lib/eventmachine.rb:187:in `run_machine'
        from /var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/eventmachine-1.0.3/lib/eventmachine.rb:187:in `run'
        from /var/vcap/packages/cloud_controller_ng/cloud_controller_ng/lib/cloud_controller/runner.rb:92:in `run!'
        from /var/vcap/packages/cloud_controller_ng/cloud_controller_ng/bin/cloud_controller:12:in `<main>'

I saw an entry from Dr Nic with the same log output (https://groups.google.com/a/cloudfoundry.org/forum/#!topicsearch/Authorization$20is$20required/vcap-dev/sjPlwv1km-U). The suggested fix was to remove dns from the network in the deployment yml. Dr. Nic's post didn't specify which job, so I tried for just cloud_controller as well as all jobs. Neither changes made any difference.

cloud_controller_ng/cloud_controller_ng.log showing NATS registration log:

{"timestamp":1379299306.676946,"message":"reusing default serving domain: 192.168.200.2.xip.io","log_level":"info","source":"cc.db.domain","data":{},"thread_id":14223020,"fiber_id":37402740,"process_id":26794,"file":"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/app/models/core/domain.rb","lineno":149,"method":"block in find_or_create_shared_domain"}
{"timestamp":1379299306.9081013,"message":"Connected to NATS - router registration","log_level":"info","source":"cf.registrar","data":{},"thread_id":14223020,"fiber_id":37402740,"process_id":26794,"file":"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/vcap_common-2.2.1/lib/cf/registrar.rb","lineno":62,"method":"register_with_router"}
{"timestamp":1379299306.9115846,"message":"Sending registration: {:host=>\"192.168.100.8\", :port=>9022, :uris=>[\"ccng.192.168.200.2.xip.io\", \"api.192.168.200.2.xip.io\"], :tags=>{:component=>\"CloudController\"}, :index=>0, :private_instance_id=>nil}","log_level":"debug","source":"cf.registrar","data":{},"thread_id":14223020,"fiber_id":37402740,"process_id":26794,"file":"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/1.9.1/gems/vcap_common-2.2.1/lib/cf/registrar.rb","lineno":96,"method":"send_registration_message"}

Stemcells I've used:

$ bosh stemcells

+---------------+---------+--------------------------------------+
| Name          | Version | CID                                  |
+---------------+---------+--------------------------------------+
| bosh-stemcell | 877     | 2302e4fc-38ae-4d69-9319-ea9afde70770 |
| bosh-stemcell | 939     | 3ca5db47-6ac0-4dc5-9f0b-0127b08cfcb5 |
| bosh-stemcell | 962     | f717056b-ba51-4cf4-aea0-910157e63f4a |
| bosh-stemcell | 991     | 9581def9-66e7-47c2-950b-a586dded7898 |
+---------------+---------+--------------------------------------+

CF release:

$ bosh releases

+------+----------+-------------+
| Name | Versions | Commit Hash |
+------+----------+-------------+
| cf   | 138*     | adca9c45+   |
+------+----------+-------------+
(*) Currently deployed
(+) Uncommitted changes

Releases total: 1

Deployment yaml (as per instructions from 13/09/13), except persistent disks are 15GB):

$ cat ../../deployments/cf/demo.yml
<%
director_uuid = "c75204b0-0315-4a2c-95ce-ac7ef2ecc39c"
protocol = "http"
ip_address = "192.168.200.2"
common_password = "passw0rd"
root_domain = "#{ip_address}.xip.io"
deployment_name = "cf-demo"
%>
name: <%= deployment_name %>
director_uuid: <%= director_uuid %>

releases:
 - name: cf
   version: 138

compilation:
  workers: 3
  network: default
  reuse_compilation_vms: true
  cloud_properties:
    instance_type: m1.small

update:
  canaries: 1
  canary_watch_time: 30000-300000
  update_watch_time: 30000-300000
  max_in_flight: 4

networks:
  - name: floating
    type: vip
    cloud_properties: {}
  - name: default
    type: dynamic
    cloud_properties:
      security_groups:
      - cf-public
      - cf-private

resource_pools:
  - name: common
    network: default
    size: 8
    stemcell:
      name: bosh-stemcell
      version: latest
    cloud_properties:
      instance_type: m1.small

  - name: large
    network: default
    size: 1
    stemcell:
      name: bosh-stemcell
      version: latest
    cloud_properties:
      instance_type: m1.large

jobs:
  - name: nats
    template:
      - nats
    instances: 1
    resource_pool: common
    networks:
      - name: default
        default: [dns, gateway]

  - name: syslog_aggregator
    template:
      - syslog_aggregator
    instances: 1
    resource_pool: common
    persistent_disk: 15360
    networks:
      - name: default
        default: [dns, gateway]

  - name: postgres
    template:
      - postgres
    instances: 1
    resource_pool: common
    persistent_disk: 15360
    networks:
      - name: default
        default: [dns, gateway]
    properties:
      db: databases

  - name: nfs_server
    template:
      - debian_nfs_server
    instances: 1
    resource_pool: common
    persistent_disk: 15360
    networks:
      - name: default
        default: [dns, gateway]

  - name: uaa
    template:
      - uaa
    instances: 1
    resource_pool: common
    networks:
      - name: default
        default: [dns, gateway]

  - name: cloud_controller
    template:
      - cloud_controller_ng
    instances: 1
    resource_pool: common
    networks:
      - name: default
        default: [dns, gateway]
    properties:
      ccdb: ccdb

  - name: router
    template:
      - gorouter
    instances: 1
    resource_pool: common
    networks:
      - name: default
        default: [dns, gateway]
      - name: floating
        static_ips:
          - <%= ip_address %>

  - name: health_manager
    template:
      - health_manager_next
    instances: 1
    resource_pool: common
    networks:
      - name: default
        default: [dns, gateway]

  - name: dea
    template: dea_next
    instances: 1
    resource_pool: large
    networks:
      - name: default
        default: [dns, gateway]

properties:
  domain: <%= root_domain %>
  system_domain: <%= root_domain %>
  system_domain_organization: "demo"
  app_domains:
    - <%= root_domain %>
  support_address: http://support.<%= root_domain %>
  description: "Cloud Foundry v2 sponsored by Pivotal"

  networks:
    apps: default
    management: default

  nats:
    address: 0.nats.default.<%= deployment_name %>.microbosh
    port: 4222
    user: nats
    password: <%= common_password %>
    authorization_timeout: 10

  router:
    status:
      port: 8080
      user: gorouter
      password: <%= common_password %>

  dea: &dea
    memory_mb: 4096
    disk_mb: 16384
    directory_server_protocol: <%= protocol %>

  dea_next: *dea

  syslog_aggregator:
    address: 0.syslog-aggregator.default.<%= deployment_name %>.microbosh
    port: 54321

  nfs_server:
    address: 0.nfs-server.default.<%= deployment_name %>.microbosh
    network: "*.<%= deployment_name %>.microbosh"
    idmapd_domain: dfw2

  debian_nfs_server:
    no_root_squash: true

  databases: &databases
    db_scheme: postgres
    address: 0.postgres.default.<%= deployment_name %>.microbosh
    port: 5524
    roles:
      - tag: admin
        name: ccadmin
        password: <%= common_password %>
      - tag: admin
        name: uaaadmin
        password: <%= common_password %>
    databases:
      - tag: cc
        name: ccdb
        citext: true
      - tag: uaa
        name: uaadb
        citext: true

  ccdb: &ccdb
    db_scheme: postgres
    address: 0.postgres.default.<%= deployment_name %>.microbosh
    port: 5524
    roles:
      - tag: admin
        name: ccadmin
        password: <%= common_password %>
    databases:
      - tag: cc
        name: ccdb
        citext: true

  ccdb_ng: *ccdb

  uaadb:
    db_scheme: postgresql
    address: 0.postgres.default.<%= deployment_name %>.microbosh
    port: 5524
    roles:
      - tag: admin
        name: uaaadmin
        password: <%= common_password %>
    databases:
      - tag: uaa
        name: uaadb
        citext: true

  cc_api_version: v2

  cc: &cc
    logging_level: debug
    external_host: ccng
    srv_api_uri: <%= protocol %>://api.<%= root_domain %>
    cc_partition: default
    db_encryption_key: <%= common_password %>
    bootstrap_admin_email: "frodenas@gopivotal.com"
    bulk_api_password: <%= common_password %>
    uaa_resource_id: cloud_controller
    staging_upload_user: upload
    staging_upload_password: <%= common_password %>
    resource_pool:
      resource_directory_key: cf-att-io-cc-resources
    packages:
      app_package_directory_key: cf-att-io-cc-packages
    droplets:
      droplet_directory_key: cf-att-io-cc-droplets
    default_quota_definition: runaway

  ccng: *cc

  login:
    enabled: false

  uaa:
    url: <%= protocol %>://uaa.<%= root_domain %>
    no_ssl: <%= protocol == "http" %>
    catalina_opts: -Xmx768m -XX:MaxPermSize=256m
    resource_id: account_manager
    jwt:
      signing_key: |
        -----BEGIN RSA PRIVATE KEY-----
        MIICXAIBAAKBgQDHFr+KICms+tuT1OXJwhCUmR2dKVy7psa8xzElSyzqx7oJyfJ1
        JZyOzToj9T5SfTIq396agbHJWVfYphNahvZ/7uMXqHxf+ZH9BL1gk9Y6kCnbM5R6
        0gfwjyW1/dQPjOzn9N394zd2FJoFHwdq9Qs0wBugspULZVNRxq7veq/fzwIDAQAB
        AoGBAJ8dRTQFhIllbHx4GLbpTQsWXJ6w4hZvskJKCLM/o8R4n+0W45pQ1xEiYKdA
        Z/DRcnjltylRImBD8XuLL8iYOQSZXNMb1h3g5/UGbUXLmCgQLOUUlnYt34QOQm+0
        KvUqfMSFBbKMsYBAoQmNdTHBaz3dZa8ON9hh/f5TT8u0OWNRAkEA5opzsIXv+52J
        duc1VGyX3SwlxiE2dStW8wZqGiuLH142n6MKnkLU4ctNLiclw6BZePXFZYIK+AkE
        xQ+k16je5QJBAN0TIKMPWIbbHVr5rkdUqOyezlFFWYOwnMmw/BKa1d3zp54VP/P8
        +5aQ2d4sMoKEOfdWH7UqMe3FszfYFvSu5KMCQFMYeFaaEEP7Jn8rGzfQ5HQd44ek
        lQJqmq6CE2BXbY/i34FuvPcKU70HEEygY6Y9d8J3o6zQ0K9SYNu+pcXt4lkCQA3h
        jJQQe5uEGJTExqed7jllQ0khFJzLMx0K6tj0NeeIzAaGCQz13oo2sCdeGRHO4aDh
        HH6Qlq/6UOV5wP8+GAcCQFgRCcB+hrje8hfEEefHcFpyKH+5g1Eu1k0mLrxK2zd+
        4SlotYRHgPCEubokb2S1zfZDWIXW3HmggnGgM949TlY=
        -----END RSA PRIVATE KEY-----
      verification_key: |
        -----BEGIN PUBLIC KEY-----
        MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDHFr+KICms+tuT1OXJwhCUmR2d
        KVy7psa8xzElSyzqx7oJyfJ1JZyOzToj9T5SfTIq396agbHJWVfYphNahvZ/7uMX
        qHxf+ZH9BL1gk9Y6kCnbM5R60gfwjyW1/dQPjOzn9N394zd2FJoFHwdq9Qs0wBug
        spULZVNRxq7veq/fzwIDAQAB
        -----END PUBLIC KEY-----
    cc:
      client_secret: <%= common_password %>
    admin:
      client_secret: <%= common_password %>
    batch:
      username: batch
      password: <%= common_password %>
    client:
      autoapprove:
        - cf
    clients:
      cf:
        override: true
        authorized-grant-types: password,implicit,refresh_token
        authorities: uaa.none
        scope: cloud_controller.read,cloud_controller.write,openid,password.write,cloud_controller.admin,scim.read,scim.write
        access-token-validity: 7200
        refresh-token-validity: 1209600
      admin:
        secret: <%= common_password %>
        authorized-grant-types: client_credentials
        authorities: clients.read,clients.write,clients.secret,password.write,scim.read,uaa.admin
    scim:
      userids_enabled: true
      users:
      - admin|<%= common_password %>|scim.write,scim.read,openid,cloud_controller.admin,uaa.admin,password.write
      - services|<%= common_password %>|scim.write,scim.read,openid,cloud_controller.admin
iwinoto commented 11 years ago

re posted to bosh-users