some weave status commands fail with "weave container is not present. Have you launched it?"

ensonic commented 4 years ago

What you expected to happen?

Try to diagnose failures of weave-net in a kubeadm cluster. Downloaded the weave tool as per: https://www.weave.works/docs/net/latest/install/installing-weave/ Weave net is already running:

kubectl get pods -n kube-system -lname=weave-net
NAME              READY   STATUS    RESTARTS   AGE
weave-net-kdxmc   2/2     Running   1          17m

Running weave status works

weave status

        Version: 2.6.4 (up to date; next check at 2020/07/29 14:30:28)

        Service: router
       Protocol: weave 1..2
           Name: 1e:3a:f0:9e:96:b8(giza-apps2-corp-guitar.ams.corp.google.com)
     Encryption: enabled
  PeerDiscovery: enabled
        Targets: 0
    Connections: 0
          Peers: 1
 TrustedSubnets: none

        Service: ipam
         Status: ready
          Range: 192.168.9.0/24
  DefaultSubnet: 192.168.9.0/24

Running other subcommands fails with the above error

What happened?

weave status connection
weave container is not present. Have you launched it?
Invalid 'weave status' sub-command: connection
Usage:

weave --help | help
...

It's also inconsistent - running weave status dns only print the error without the usage (although I don't find the usage helpful here).

How to reproduce it?

See above.

Anything else we need to know?

Versions:

$ weave version
weave script 2.6.5
weave 2.6.4
$ docker version
Client: Docker Engine - Community
 Version:           19.03.5
 API version:       1.40
 Go version:        go1.12.12
 Git commit:        633a0ea838
 Built:             Wed Nov 13 07:25:58 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.5
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.12
  Git commit:       633a0ea838
  Built:            Wed Nov 13 07:24:29 2019
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.2.10
  GitCommit:        b34a5c8af56e510852c35414db4c1f4fa6172339
 runc:
  Version:          1.0.0-rc8+dev
  GitCommit:        3e425f80a8c931f88e6d94a8c831b9d5aa481657
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683
$ uname -a
Linux xxxxx 5.6.14-2-amd64 #1 SMP Debian 5.6.14-2 (2020-06-20 > 2018) x86_64 GNU/Linux
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.6", GitCommit:"dff82dc0de47299ab66c83c626e08b245ab19037", GitTreeState:"clean", BuildDate:"2020-07-15T16:58:53Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.3", GitCommit:"06ad960bfd03b39c8310aaf92d1e7c12ce618213", GitTreeState:"clean", BuildDate:"2020-02-11T18:07:13Z", GoVersion:"go1.13.6", Compiler:"gc", Platform:"linux/amd64"}

Logs:

# $ kubectl logs -n kube-system <weave-net-pod> weave
# FYI: if you want to update the bug template to use '-lname=weave-net' people can copy'n'paste the command
$ kubectl logs -n kube-system -lname=weave-net -c weave
WARN: 2020/07/29 08:01:36.471967 Vetoed installation of hairpin flow FlowSpec{keys: [EthernetFlowKey{src: ba:fe:f2:35:89:0b, dst: 96:92:62:52:21:b6} InPortFlowKey{vport: 1}], actions: [OutputAction{vport: 1}]}
WARN: 2020/07/29 08:01:38.423402 Vetoed installation of hairpin flow FlowSpec{keys: [InPortFlowKey{vport: 1} EthernetFlowKey{src: 1e:3a:f0:9e:96:b8, dst: d2:4d:8d:26:fd:af}], actions: [OutputAction{vport: 1}]}
WARN: 2020/07/29 08:01:38.452213 Vetoed installation of hairpin flow FlowSpec{keys: [EthernetFlowKey{src: 1e:3a:f0:9e:96:b8, dst: 96:92:62:52:21:b6} InPortFlowKey{vport: 1}], actions: [OutputAction{vport: 1}]}
WARN: 2020/07/29 08:01:38.452271 Vetoed installation of hairpin flow FlowSpec{keys: [EthernetFlowKey{src: b6:bf:5c:9f:95:95, dst: ba:fe:f2:35:89:0b} InPortFlowKey{vport: 1}], actions: [OutputAction{vport: 1}]}
WARN: 2020/07/29 08:01:38.643380 Vetoed installation of hairpin flow FlowSpec{keys: [EthernetFlowKey{src: 1e:09:2e:df:98:67, dst: b6:bf:5c:9f:95:95} InPortFlowKey{vport: 1}], actions: [OutputAction{vport: 1}]}
WARN: 2020/07/29 08:01:38.643458 Vetoed installation of hairpin flow FlowSpec{keys: [EthernetFlowKey{src: aa:cd:f7:d7:af:05, dst: b6:bf:5c:9f:95:95} InPortFlowKey{vport: 1}], actions: [OutputAction{vport: 1}]}
WARN: 2020/07/29 08:04:31.251455 Vetoed installation of hairpin flow FlowSpec{keys: [EthernetFlowKey{src: 1e:3a:f0:9e:96:b8, dst: 8a:6d:d0:ab:66:36} InPortFlowKey{vport: 1}], actions: [OutputAction{vport: 1}]}
WARN: 2020/07/29 08:04:31.251533 Vetoed installation of hairpin flow FlowSpec{keys: [EthernetFlowKey{src: 1e:3a:f0:9e:96:b8, dst: 62:d9:90:09:56:d9} InPortFlowKey{vport: 1}], actions: [OutputAction{vport: 1}]}
WARN: 2020/07/29 08:04:31.283265 Vetoed installation of hairpin flow FlowSpec{keys: [EthernetFlowKey{src: 8a:02:af:38:b7:5b, dst: 26:30:06:a8:4c:2d} InPortFlowKey{vport: 1}], actions: [OutputAction{vport: 1}]}
WARN: 2020/07/29 08:04:31.571574 Vetoed installation of hairpin flow FlowSpec{keys: [EthernetFlowKey{src: ca:a6:c5:53:bb:f8, dst: a6:98:92:32:8c:53} InPortFlowKey{vport: 1}], actions: [OutputAction{vport: 1}]}

Network:

$ ip route
default via 100.109.39.254 dev ens13 proto dhcp metric 100 
100.109.39.0/24 dev ens13 proto kernel scope link src 100.109.39.7 metric 100 
192.168.9.0/24 dev weave proto kernel scope link src 192.168.9.1 metric 425 
192.168.10.0/24 dev docker0 proto kernel scope link src 192.168.10.1 linkdown 
$ ip -4 -o addr
1: lo    inet 127.0.0.1/8 scope host lo\       valid_lft forever preferred_lft forever
2: ens13    inet 100.109.39.7/24 brd 100.109.39.255 scope global dynamic noprefixroute ens13\       valid_lft 84870sec preferred_lft 84870sec
3: docker0    inet 192.168.10.1/24 brd 192.168.10.255 scope global docker0\       valid_lft forever preferred_lft forever
6: weave    inet 192.168.9.1/24 brd 192.168.9.255 scope global weave\       valid_lft forever preferred_lft forever
$ sudo iptables-save
https://gist.github.com/ensonic/e5f1b7146a20d7e8cb2424f2b32b080c

ensonic commented 4 years ago

same problem with weave env.

ensonic commented 4 years ago

erm it is weave status connection*s*, but that outputs an empty list

ensonic commented 4 years ago

ahh, this seems to be always printed when the "docker run" fails (util_op). So its mostly a UX issue.

weaveworks / weave