search

weaveworks / scope

Monitoring, visualisation & management for Docker & Kubernetes
https://www.weave.works/oss/scope/
Apache License 2.0
5.86k stars 714 forks source link

Unconnected containers in sock-shop #3082

Open dholbach opened 6 years ago

dholbach commented 6 years ago

Looking at the Explore → Containers view of sock-shop in WC, I can see quite a few containers which are not connected to anything. In a quick conversation, @rade pointed out that #1491, #1846, #2129 and #2586 could possibly be related.

screenshot-unconnected-containers

dholbach commented 6 years ago

This seems to happen, both with "Show Uncontained" and "Hide Uncontained".

dholbach commented 6 years ago

screenshot-unconnected-containers-2

rade commented 6 years ago

At first I thought it might be missing all short-lived connections, but I do occasionally see connections from front-end to user, so the problem must be something else.

The Pod topology looks fine, with all expected connections showing.

dholbach commented 6 years ago

Toggling the filters did not help, FF 59.0b12 Ubuntu.

bboreham commented 6 years ago

I suspect the POD containers are visible because Kubernetes changed the image of the ”pause container”. See IsPauseImageName() in the code.

rade commented 6 years ago

Oh, that's yet another bug.

rade commented 6 years ago

To be clear, the issue here isn't all the unconnected POD containers - that is expected - but all the other unconnected containers, especially since for a good few of them the corresponding pods do show as connected.

rade commented 6 years ago

the POD containers are visible because Kubernetes changed the image of the ”pause container”.

Actually, they are visible because of the filtering bug. if you switch to show 'All' and then back to 'Application Containers' they go away.

bboreham commented 6 years ago

Checking more closely I see POD containers with image gcr.io/google_containers/pause-amd64:3.0 which would be matched by the current hard-coded string. Sorry for confusion.

bboreham commented 6 years ago

for a good few of them the corresponding pods do show as connected.

I think the difference is that Scope knows the pod IPs and doesn't know the container IPs, and doesn't join those pieces of information together.

Some connections have a pid, and Scope does map processes to containers when rendering, so you see those connections.

rade commented 6 years ago

Some connections have a pid, and Scope does map processes to containers when rendering, so you see those connections.

Why don't all the connections have a pid?

jrryjcksn commented 5 years ago

It looks like this is being seen in the field. See: https://weave-community.slack.com/archives/C985DLV6F/p1550588443054100

jrryjcksn commented 5 years ago

Follow-on from the same user: FYI - i re-deployed using Calico CNI vs. Weave Net and I do now see connections between the pods. I can't say for sure if I see them all as I don't appreciate exactly all the communications between components. I suspect that maybe with Calico since there is no enacp. the weave scope is able to decode the traffic somehow?