weaveworks / scope

Monitoring, visualisation & management for Docker & Kubernetes
https://www.weave.works/oss/scope/
Apache License 2.0
5.84k stars 708 forks source link

Don't have TCP established connection from client but still show connection on the map #3823

Open thaonguyen-ct opened 3 years ago

thaonguyen-ct commented 3 years ago

Let's say, I have two pods A and B are running on K8s. A(source) sends a TCP request to B(dest) and both have a TCP established connection, but somehow pod A is stopped suddenly and don't send FIN package to close this TCP connection, so the TCP connection is still established on pod B and on pod A has gone. But the response of weave-scope-app API of deployment B still has the inbound connection from A and the response of weave-scope-app API of deployment A still have the outbound connection to B even pod A doesn't send TCP request to B anymore.

What you expected to happen?

Don't see the connection between A and B or at least the don't have the outbound connection on deployment A

What happened?

The connection from A to B still exist on both deployment A and deployment B when call weave-scope-app api

How to reproduce it?

  1. Deploy two deployment A and B like Nginx for example
  2. From pod A telnet to pod B and you can see the TCP established connection on pod A and B by check netstat -nat
  3. From pod A install the iptables and drop outbound of FIN package iptables -I OUTPUT --protocol tcp --tcp-flags FIN FIN -j DROP
  4. Kill the telnet command from pod A or crtl-c
  5. You can see the TCP connection still established on pod B but not on pod A
  6. Call API of weave-scope-app to verify the inbound connection of pod B and outbound connection of pod A still there http:///api/topology/kube-controllers/;%3Cdeployment%3E

Versions:

Weave scope: 1.13.1 Kubernetes: 1.13.3 Docker on kubernetes node: 18.6.3

bboreham commented 3 years ago

It sounds like Scope is telling you what Linux is telling it.

The line on the graph goes away, right? Because one of the process endpoints is now missing.

thaonguyen-ct commented 3 years ago

@bboreham No. The line on the graph is still appear