Post https://mbus:mbus-password@0.0.0.0:6868/agent: Forbidden port

[StBurcher]

Hi,

I got the following error during the bosh init create VM.

Command 'deploy' failed: Deploying: Creating instance 'bosh/0': Waiting until instance is ready: Sending ping to the agent: Performing request to agent endpoint 'https://mbus:mbus-password@139.25.25.219::6868/agent': Performing POST request: Post https://mbus:mbus-password@139.25.25.219::6868/agent: Forbidden port

SSH is possible to the maschin. Also a nc

nc -zv 139.25.25.219 6868 Connection to 139.25.25.219 6868 port [tcp/*] succeeded!

but wget provides a 401

https://mbus:_password_@139.25.25.219:6868/agent Connecting to 139.25.25.219:6868... connected. WARNING: cannot verify 139.25.25.219's certificate, issued by ‘/C=US/O=Pivotal/CN=localhost’: Self-signed certificate encountered. WARNING: certificate common name ‘localhost’ doesn't match requested host name ‘139.25.25.219’. HTTP request sent, awaiting response... 401 Unauthorized Reusing existing connection to 139.25.25.219:6868. HTTP request sent, awaiting response... 404 Not Found 2016-10-14 08:00:41 ERROR 404: Not Found.

Here are my bosh.yml

---
name: bosh

releases:
- name: bosh
  url: https://bosh.io/d/github.com/cloudfoundry/bosh?v=257.15
  sha1: f4cf3579bfac994cd3bde4a9d8cbee3ad189c8b2
- name: bosh-openstack-cpi
  url: https://bosh.io/d/github.com/cloudfoundry-incubator/bosh-openstack-cpi-release?v=27
  sha1: 85e6244978f775c888bbd303b874a2c158eb43c4

resource_pools:
- name: vms
  network: private
  stemcell:
    url: https://bosh.io/d/stemcells/bosh-openstack-kvm-ubuntu-trusty-go_agent?v=3263.7
    sha1: 079fd87e0e707cfa09bad781086983c25a605962
  cloud_properties:
    instance_type: m1.xlarge

disk_pools:
- name: disks
  disk_size: 20_000

networks:
- name: private
  type: manual
  subnets:
  - range: 10.0.0.0/24 # <--- Replace with a private subnet CIDR
    gateway: 10.0.0.1 # <--- Replace with a private subnet's gateway
    dns: [10.0.0.2] # <--- Replace with your DNS
    cloud_properties: {net_id: 99828b4e-cb2b-45bf-97e2-852388bc16c0} # <--- # Replace with private network UUID
- name: public
  type: vip

jobs:
- name: bosh
  instances: 1

  templates:
  - {name: nats, release: bosh}
  - {name: postgres, release: bosh}
  - {name: blobstore, release: bosh}
  - {name: director, release: bosh}
  - {name: health_monitor, release: bosh}
  - {name: registry, release: bosh}
  - {name: openstack_cpi, release: bosh-openstack-cpi}

  resource_pool: vms
  persistent_disk_pool: disks

  networks:
  - name: private
    static_ips: [10.0.0.101] # <--- Replace with a private IP
    default: [dns, gateway]
  - name: public
    static_ips: [139.25.25.219] # <--- Replace with a floating IP

  properties:
    nats:
      address: 127.0.0.1
      user: nats
      password: nats-password

    postgres: &db
      listen_address: 127.0.0.1
      host: 127.0.0.1
      user: postgres
      password: postgres-password
      database: bosh
      adapter: postgres

    registry:
      address: 10.0.0.101 # <--- Replace with a private IP
      host: 10.0.0.101 # <--- Replace with a private IP
      db: *db
      http: {user: admin, password: admin, port: 25777}
      username: admin
      password: admin
      port: 25777

    blobstore:
      address: 10.0.0.101 # <--- Replace with a private IP
      port: 25250
      provider: dav
      director: {user: director, password: director-password}
      agent: {user: agent, password: agent-password}

    director:
      address: 127.0.0.1
      name: my-bosh
      db: *db
      cpi_job: openstack_cpi
      max_threads: 3
      user_management:
        provider: local
        local:
          users:
          - {name: admin, password: admin}
          - {name: hm, password: hm-password}

    hm:
      director_account: {user: hm, password: hm-password}
      resurrector_enabled: true

    openstack: &openstack
      #auth_url: https://10.10.0.10:35357/v3/tokens # <--- Replace with OpenStack Identity API endpoint
      auth_url: https://10.10.0.10:5000/v2.0/tokens
      project: CF_Diego # <--- Replace with OpenStack project name
      tenant: CF_Diego
      domain: default # <--- Replace with OpenStack domain name
      username: STUSER # <--- Replace with OpenStack username
      api_key: MYPASS  # <--- Replace with OpenStack password
      default_key_name: bosh-manual-key
      default_security_groups: [bosh, Security]
      connection_options: # all connection options that are supported by Excon
       ssl_verify_peer: false # Boolean

    agent: {mbus: "nats://nats:nats-password@139.25.25.219:4222"} # <--- Replace with a private IP

    ntp: &ntp [0.pool.ntp.org, 1.pool.ntp.org]

cloud_provider:
  template: {name: openstack_cpi, release: bosh-openstack-cpi}

  ssh_tunnel:
    host: 139.25.25.219 # <--- Replace with a floating IP
    port: 22
    user: vcap
    private_key: ../bosh-manual-key # Path relative to this manifest file

  mbus: "https://mbus:mbus-password@139.25.25.219:6868" # <--- Replace with a floating IP

  properties:
    openstack: *openstack
    agent: {mbus: "https://mbus:mbus-password@0.0.0.0:6868"}
    blobstore: {provider: local, path: /var/vcap/micro_bosh/data/cache}
    ntp: *ntp

[voelzmo]

@StBurcher your attempt to connect with wget fails because the ssl certificate cannot be validated (and you seem to use the wrong password)

Could you try a curl -k -vv https://mbus:mbus-password@139.25.25.219:6868/agent/ instead when the you're bringing up the Director VM?

Another question: is there an http(s) proxy server configured on your machine? If netcat works with plain tcp packets, but http doesn't, that seems suspicious.

[StBurcher]

@voelzmo Unfortunately, there is a proxy involved. I'm using export no_proxy for the ips 139.25.25.219 and 10.0.0.101 Here are the curl -k -vv https://mbus:mbus-password@139.25.25.219:6868/agentz output

curl -k -vv https://mbus:mbus-password@139.25.25.219:6868/age                                                                                                             nt/
* Hostname was NOT found in DNS cache
*   Trying 139.25.25.219...
* Connected to 139.25.25.219 (139.25.25.219) port 6868 (#0)
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Server key exchange (12):
* SSLv3, TLS handshake, Server finished (14):
* SSLv3, TLS handshake, Client key exchange (16):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSL connection using ECDHE-RSA-AES128-GCM-SHA256
* Server certificate:
*        subject: C=US; O=Pivotal; CN=localhost
*        start date: 2013-12-01 22:11:32 GMT
*        expire date: 2016-12-01 22:11:32 GMT
*        issuer: C=US; O=Pivotal; CN=localhost
*        SSL certificate verify result: self signed certificate (18), continuing                                                                                                              anyway.
* Server auth using Basic with user 'mbus'
> GET /agent/ HTTP/1.1
> Authorization: Basic bWJ1czptYnVzLXBhc3N3b3Jk
> User-Agent: curl/7.35.0
> Host: 139.25.25.219:6868
> Accept: */*
>
< HTTP/1.1 404 Not Found
< Content-Type: text/plain; charset=utf-8
< X-Content-Type-Options: nosniff
< Date: Fri, 14 Oct 2016 11:06:18 GMT
< Content-Length: 19
<
404 page not found

[StBurcher]

Short update. If im login with ssh to the instance. I can see the port ss -nlput tcp LISTEN 0 128 :::6868 :::* The logs says agent started.

cat /var/vcap/bosh/log/current | grep -B2 "Starting agent"
2016-10-14_12:00:20.05121 [main] 2016/10/14 12:00:20 DEBUG - Starting agent

[voelzmo]

@cppforlife @dpb587-pivotal Does bosh-init cope well with proxy/no_proxy ENV vars? Seems like port 6868 isn't allowed for the proxy @StBurcher is using.

[dpb587-pivotal]

Is that the actual manifest you're deploying with? I see double colons in the connection attempts which may be throwing dials off. If it's a different manifest, perhaps double check there aren't double typos between IPs/hosts and the port numbers.

You also might have the wrong IP in jobs[0].properties.agent.mbus - I think you want 10.0.0.101.

[StBurcher]

@dpb587-pivotal Thanks for response. This is the actual manifest. I have changed the IP Adresse of mbus and nats to 10.0.0.101. Unfortunately, it does not work.

Performing request to agent endpoint 'https://mbus:mbus-password@10.0.0.101:6868/agent': Performing POST request: Post https://mbus:mbus-password@10.0.0.101:6868/agent: Forbidden port

The Linux VM, Im using for starting bosh init is in the same network like the BOSH Directory.

[StBurcher]

@dpb587-pivotal and @voelzmo Short Update: If I going with SSH to the BOSH Director VM and using curl to establish a connection to the mbus agent. The following Error occurs (HTTP/1.1 404 Not Found). curl -kv https://localhost:6868/agent

* Hostname was NOT found in DNS cache
*   Trying 127.0.0.1...
* Connected to localhost (127.0.0.1) port 6868 (#0)
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Server key exchange (12):
* SSLv3, TLS handshake, Server finished (14):
* SSLv3, TLS handshake, Client key exchange (16):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSL connection using ECDHE-RSA-AES128-GCM-SHA256
* Server certificate:
*        subject: C=US; O=Pivotal; CN=localhost
*        start date: 2013-12-01 22:11:32 GMT
*        expire date: 2016-12-01 22:11:32 GMT
*        issuer: C=US; O=Pivotal; CN=localhost
*        SSL certificate verify result: self signed certificate (18), continuing anyway.
> GET /agent HTTP/1.1
> User-Agent: curl/7.35.0
> Host: localhost:6868
> Accept: */*
>
< HTTP/1.1 401 Unauthorized
< Www-Authenticate: Basic realm=""
< Date: Tue, 18 Oct 2016 12:17:51 GMT
< Content-Length: 0
< Content-Type: text/plain; charset=utf-8
<
* Connection #0 to host localhost left intact
vcap@9af98c0b-eb41-49cc-49e4-a513ff69b14c:~$ curl -kv https://mbus:mbus-password@localhost:6868/agent
* Hostname was NOT found in DNS cache
*   Trying 127.0.0.1...
* Connected to localhost (127.0.0.1) port 6868 (#0)
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Server key exchange (12):
* SSLv3, TLS handshake, Server finished (14):
* SSLv3, TLS handshake, Client key exchange (16):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSL connection using ECDHE-RSA-AES128-GCM-SHA256
* Server certificate:
*        subject: C=US; O=Pivotal; CN=localhost
*        start date: 2013-12-01 22:11:32 GMT
*        expire date: 2016-12-01 22:11:32 GMT
*        issuer: C=US; O=Pivotal; CN=localhost
*        SSL certificate verify result: self signed certificate (18), continuing anyway.
* Server auth using Basic with user 'mbus'
> GET /agent HTTP/1.1
> Authorization: Basic bWJ1czptYnVzLXBhc3N3b3Jk
> User-Agent: curl/7.35.0
> Host: localhost:6868
> Accept: */*
>
< HTTP/1.1 404 Not Found
< Date: Tue, 18 Oct 2016 12:18:06 GMT
< Content-Length: 0
< Content-Type: text/plain; charset=utf-8
<
* Connection #0 to host localhost left intact

[StBurcher]

It's getting wired. using Curl curl -vX POST -k https://mbus:mbus-password@139.25.25.219:6868/agent -d @test.json --header "Content-Type: application/json"

Returns {"value":"pong"}

But in the bosh.log with level debug.

[httpClient] 2016/10/19 14:33:49 DEBUG - Sending POST request to endpoint 'https://mbus:mbus-password@139.25.25.219:6868/agent' with body '{"method":"ping","arguments":[],"reply_to":"bd88454e-f00d-4b32-7ef7-6b290e25000f"}'
[timeoutRetryStrategy] 2016/10/19 14:33:50 DEBUG - Making attempt #97

[StBurcher]

Now, I wrote my own go test program.

package main

import ( "bytes" "fmt" "io/ioutil" "net/http" "crypto/tls" )

func main() { tr := &http.Transport{ TLSClientConfig: &tls.Config{InsecureSkipVerify: true}, }

url := "https://mbus:mbus-password@139.25.25.219:6868/agent"
fmt.Println("URL:>", url)

var jsonStr = []byte(`

{ "method":"ping", "arguments":[], "reply_to":"bd88454e-f00d-4b32-7ef7-6b290e25000f"}`) req, err := http.NewRequest("POST", url, bytes.NewBuffer(jsonStr)) req.Header.Set("X-Custom-Header", "myvalue") req.Header.Set("Content-Type", "application/json")

client := &http.Client{Transport: tr}
resp, err := client.Do(req)
if err != nil {
    panic(err)
}
defer resp.Body.Close()

fmt.Println("response Status:", resp.Status)
fmt.Println("response Headers:", resp.Header)
body, _ := ioutil.ReadAll(resp.Body)
fmt.Println("response Body:", string(body))

}

It works only with this lines, which are disabling the security check. We have a self sign certificate.

tr := &http.Transport{
    TLSClientConfig: &tls.Config{InsecureSkipVerify: true},
}

Is it possible to active that in bosh-init?

[cklause]

Could you try importing the certificated used for Openstack into the CA certs of your Linux environment?

[StBurcher]

It works now. But I had to import the pivotal certificate from the stemcell. getting this with openssl s_client -showcerts -connect 139.25.25.219:6868 -prexit 2>/dev/null | sed -n -e '/BEGIN\ CERTIFICATE/,/END\ CERTIFICATE/ p'

Cp the cert to /usr/share/ca-certificates/ , update /etc/ca-certifications.conf and uses update-ca-certifications.

Then it works.

But why the pivotal cert?

[StBurcher]

Finally found it. We have a proxy problem. Further, the proxy was twice in the env. Unset all env solves the issue.

[dpb587-pivotal]

Glad you were able to figure out, and thanks for reporting back what the issue turned out to be!