Unable to remove/leave Agent with unreachable Node-Address

`consul version` for both Client and Server

Client: 0.8.3 Server: 0.8.3

`consul info` for both Client and Server

Client:

agent:
        check_monitors = 0
        check_ttls = 0
        checks = 0
        services = 0
build:
        prerelease =
        revision = ea2a82b
        version = 0.8.3
consul:
        known_servers = 3
        server = false
runtime:
        arch = amd64
        cpu_count = 32
        goroutines = 117
        max_procs = 32
        os = linux
        version = go1.8.1
serf_lan:
        encrypted = true
        event_queue = 0
        event_time = 219
        failed = 0
        health_score = 0
        intent_queue = 0
        left = 0
        member_time = 223
        members = 11
        query_queue = 0
        query_time = 18

Server:

agent:
        check_monitors = 0
        check_ttls = 0
        checks = 1
        services = 2
build:
        prerelease =
        revision = ea2a82b
        version = 0.8.3
consul:
        bootstrap = false
        known_datacenters = 1
        leader = true
        leader_addr = [2a00:1f08:900:1::6]:8300
        server = true
raft:
        applied_index = 464656
        commit_index = 464656
        fsm_pending = 0
        last_contact = 0
        last_log_index = 464656
        last_log_term = 4967
        last_snapshot_index = 464324
        last_snapshot_term = 4966
        latest_configuration = [{Suffrage:Voter ID:[2a00:1f08:0900:3:0:0:0:1]:8300 Address:[2a00:1f08:0900:3:0:0:0:1]:8300} {Suffrage:Voter ID:[2a00:1f08:0900:3:0:0:0:2]:8300 Address:[2a00:1f08:0900:3:0:0:0:2]:8300} {Suffrage:Voter ID:[2a00:1f08:900:1::6]:8300 Address:[2a00:1f08:900:1::6]:8300} {Suffrage:Voter ID:[2a00:1f08:900:3::1]:8300 Address:[2a00:1f08:900:3::1]:8300} {Suffrage:Voter ID:[2a00:1f08:900:3::2]:8300 Address:[2a00:1f08:900:3::2]:8300}]
        latest_configuration_index = 193962
        num_peers = 4
        protocol_version = 2
        protocol_version_max = 3
        protocol_version_min = 0
        snapshot_version_max = 1
        snapshot_version_min = 0
        state = Leader
        term = 4967
runtime:
        arch = amd64
        cpu_count = 8
        goroutines = 275
        max_procs = 8
        os = linux
        version = go1.8.1
serf_lan:
        encrypted = true
        event_queue = 0
        event_time = 219
        failed = 0
        health_score = 0
        intent_queue = 0
        left = 0
        member_time = 223
        members = 11
        query_queue = 0
        query_time = 18
serf_wan:
        encrypted = true
        event_queue = 0
        event_time = 1
        failed = 0
        health_score = 0
        intent_queue = 0
        left = 0
        member_time = 46
        members = 3
        query_queue = 0
        query_time = 1

Operating system and Environment details

RedHat EL 7.3

Description of the Issue (and unexpected/desired result)

We had to reinstall one of the servers which was running as consul client. We did not leave/force-leave the client. During installation we did configure a wrong bind address (IPv4) for the re-installed agent. Somehow the agent was able to connect to the IPv6-Consul-Servers anyway. Since then all of the cluster agents show this wrong cluster member with the state alive. But is isn't alive - all agents are complaining about not beeing able to connect to the IPv4-Address. We are not able to leave/force-leave that wrong host (bind address has been corrected in the mean-time). When trying a force leave on some other agent B the faulty agent is stuck in state leaving on agent B (but not on Agent C for example, where it is still alive) ,until a restart of the agent B. Then the faulty agent goes back to alive on Agent B.

As a workaround we assigned a different name/id to the faulty node. It was able to join the cluster without problems.

Since the state of the orphaned agent is 'alive' i 'don't think it will be reaped automatically. Any chance to remove it manually ?

Reproduction steps

Uninstall Agent without leave / force leave
Configure wrong & unreachable bind-address, but correct server-addresss to join
start up agent
Agent appears with wrong ip-adress in memberlist of all other nodes in DC

Log Fragments or Link to gist

Memberlist - the first entry is wrong and definitive not alive ceph-dc-1-01-osd-05 10.60.0.36:8301 alive client 0.8.3 2 dc-witcom-cloud ceph-dc-1-01-osd-05-temp-02 [2aXY:1f08:900:1::ZZZ]:8301 alive client 0.8.3 2 dc-witcom-cloud

Log from same host 2017/07/11 23:04:54 [ERR] memberlist: Failed to send ping: write udp [2aXY:1f08:900:1::ZZY]:8301->10.60.0.36:8301: sendto: network is unreachable 2017/07/11 23:04:58 [ERR] memberlist: Push/Pull with ceph-dc-1-01-osd-05 failed: dial tcp 10.60.0.36:8301: getsockopt: connection refused 2017/07/11 23:05:03 [ERR] memberlist: Failed to send ping: write udp [2aXY:1f08:900:1::ZZY]:8301->10.60.0.36:8301: sendto: network is unreachable

hashicorp / consul