search

hashicorp / vault

A tool for secrets management, encryption as a service, and privileged access management
https://www.vaultproject.io/
Other
31.05k stars 4.2k forks source link

vault keeps trying to connect to port localhost:9001 #3170

Closed dictvm closed 7 years ago

dictvm commented 7 years ago

Environment:

Vault Config File:

storage "consul" {
  address = "localhost:8500" # local agent
  path    = "vault"
}

listener "tcp" {
  address     = "0.0.0.0:8200"
  tls_disable = 1
}

listener "tcp" {
  address     = "0.0.0.0:8900"
  tls_disable = 1
}

listener "tcp" {
  address     = "0.0.0.0:9000"
  tls_disable = 1
}

telemetry {
  statsd_address = "127.0.0.1:9125"
  disable_hostname = true
}

disable_mlock = "true"

Expected Behavior: Vault should only use ports that are documented.

Actual Behavior: Vault is trying to connect to port 9001 on localhost via grpc and keeps throwing warnings into the log:

Aug 15 13:31:52 ip-172-21-70-75 vault[1872]: 2017/08/15 13:31:52.436220 [WARN ] grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:9001: getsockopt: connection refused"; Reconnecting to {localhost:9001 <nil>}
Aug 15 13:31:53 ip-172-21-70-75 vault[1872]: 2017/08/15 13:31:53.436535 [WARN ] grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:9001: getsockopt: connection refused"; Reconnecting to {localhost:9001 <nil>}
Aug 15 13:31:54 ip-172-21-70-75 vault[1872]: 2017/08/15 13:31:54.952632 [WARN ] grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:9001: getsockopt: connection refused"; Reconnecting to {localhost:9001 <nil>}
Aug 15 13:31:57 ip-172-21-70-75 vault[1872]: 2017/08/15 13:31:57.380560 [WARN ] grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:9001: getsockopt: connection refused"; Reconnecting to {localhost:9001 <nil>}
Aug 15 13:32:01 ip-172-21-70-75 vault[1872]: 2017/08/15 13:32:01.626311 [WARN ] grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:9001: getsockopt: connection refused"; Reconnecting to {localhost:9001 <nil>}
Aug 15 13:32:07 ip-172-21-70-75 vault[1872]: 2017/08/15 13:32:07.862752 [WARN ] grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:9001: getsockopt: connection refused"; Reconnecting to {localhost:9001 <nil>}
Aug 15 13:32:19 ip-172-21-70-75 vault[1872]: 2017/08/15 13:32:19.774664 [WARN ] grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:9001: getsockopt: connection refused"; Reconnecting to {localhost:9001 <nil>}
Aug 15 13:32:39 ip-172-21-70-75 vault[1872]: 2017/08/15 13:32:39.033187 [WARN ] grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:9001: getsockopt: connection refused"; Reconnecting to {localhost:9001 <nil>}
Aug 15 13:33:03 ip-172-21-70-75 vault[1872]: 2017/08/15 13:33:03.425986 [WARN ] grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:9001: getsockopt: connection refused"; Reconnecting to {localhost:9001 <nil>}
Aug 15 13:33:48 ip-172-21-70-75 vault[1872]: 2017/08/15 13:33:48.850988 [WARN ] grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:9001: getsockopt: connection refused"; Reconnecting to {localhost:9001 <nil>}
Aug 15 13:34:15 ip-172-21-70-75 vault[1872]: 2017/08/15 13:34:15.183871 [INFO ] core: acquired lock, enabling active operation
Aug 15 13:34:15 ip-172-21-70-75 vault[1872]: 2017/08/15 13:34:15.232203 [INFO ] core: post-unseal setup starting
Aug 15 13:34:15 ip-172-21-70-75 vault[1872]: 2017/08/15 13:34:15.232628 [WARN ] Failed to dial localhost:9001: context canceled; please retry.

Steps to Reproduce: I am not quite sure, to be honest. I can find no trace in my configuration of a ressource that is supposed to listen to port 9001.

I've created a Consul Cluster with 5 nodes, a Vault 3 cluster with 3 nodes and 3 Consul agents on the Vault nodes.

Important Factoids: I'm on AWS. On the Vault nodes I'm also running a consul agent, a node_exporter and a statsd_exporter. The latter two are for monitoring purposes. Each of the services is running as a systemd-unit.

jefferai commented 7 years ago

Your standby nodes are trying to connect to the active node; since you didn't specify an address directly it's using your api listener + 1 for the port.

The reason it's not working is likely related to #3168 so let's tackle that and I think this will clear up.