search

overnode-org / overnode

Predictable container deployment and management on top of automated multi-host docker-compose
https://overnode.org/
MIT License
45 stars 10 forks source link

Ubuntu 20.04: Installation of overnode fails: Weave fails with runtime error: slice bounds out of range [:9] with length 8 #56

Open jtagcat opened 3 years ago

jtagcat commented 3 years ago

Latest Ubuntu 20.04 does not work. Breakage caused by systemd 245.4-4ubuntu3.3.

Errors seen using overnode: weave status and other commands often return curl: (52) Empty reply from server

Errors after launching: [overnode] Error: weave is not running

$ sudo overnode launch --id 1 --token my-cluster-password                                                                                                                                                                            
[overnode] Launching weave ...                                                                                                                                                                                                                 
[overnode] => done: 58da42db3ce382c95e0aeb50ca46cbeb96ae92c07052d5584e2b8140a8e98ebb                                                                                                                                                           
[overnode] Launching agent ...                                                                                         
Creating volume "overnode" with local driver                                                                                                                                                                                                   
Creating overnode ... done                                                                                             
[overnode] => done                                                                                                     
[1] Node launched                                                                                                      
$ sudo overnode status                                                                                       
[overnode] Error: weave is not running                                                                                 
[overnode] Run '> overnode launch' to start the node                                                                   
[overnode] Run '> overnode resume' to restart the node                                                                 
[overnode] [action aborted]  
$ sudo docker ps
CONTAINER ID        IMAGE                     COMMAND                  CREATED             STATUS              PORTS               NAMES
96581298e474        alpine/socat:1.7.3.4-r0   "/w/w socat TCP-LIST…"   29 seconds ago      Up 26 seconds                           overnode
58da42db3ce3        weaveworks/weave:2.6.5    "/home/weave/weaver …"   44 seconds ago      Up 42 seconds                           weave
$ sudo docker logs weave
INFO: 2021/01/02 15:33:11.510465 Command line options: map[H:[unix:///var/run/weave/weave.sock] datapath:datapath dns-domain:weave.local. dns-listen-address:172.17.0.1:53 docker-bridge:docker0 host-root:/host http-addr:127.0.0.1:6784 ipalloc-default-subnet:10.32.0.0/12 ipalloc-init:seed=::1,::2,::3,::4 ipalloc-range:10.40.0.0/13 name:::1 nickname:rkon4 plugin:false port:6783 proxy:true resolv-conf:/var/run/weave/etc/stub-resolv.conf rewrite-inspect:true status-addr:127.0.0.1:6782 weave-bridge:weave]
INFO: 2021/01/02 15:33:11.510557 weave  2.6.5
INFO: 2021/01/02 15:33:11.547363 Docker API on unix:///var/run/docker.sock: &[ApiVersion=1.40 Os=linux KernelVersion=5.4.0-58-generic BuildTime=2020-06-01T09:11:07.000000000+00:00 Components=[{"Details":{"ApiVersion":"1.40","Arch":"amd64","BuildTime":"2020-06-01T09:11:07.000000000+00:00","Experimental":"false","GitCommit":"42e35e61f3","GoVersion":"go1.13.10","KernelVersion":"5.4.0-58-generic","MinAPIVersion":"1.12","Os":"linux"},"Name":"Engine","Version":"19.03.11"},{"Details":{"GitCommit":"269548fa27e0089a8b8278fc4fc781d7f65a939b"},"Name":"containerd","Version":"1.4.3"},{"Details":{"GitCommit":"ff819c7e9184c13b7c2607fe6c30ae19403a7aff"},"Name":"runc","Version":"1.0.0-rc92"},{"Details":{"GitCommit":"fec3683"},"Name":"docker-init","Version":"0.18.0"}] Version=19.03.11 MinAPIVersion=1.12 GitCommit=42e35e61f3 GoVersion=go1.13.10 Arch=amd64 Platform={"Name":"Docker Engine - Community"}]
INFO: 2021/01/02 15:33:11.568420 proxy listening on unix:///var/run/weave/weave.sock
INFO: 2021/01/02 15:33:11.883886 Bridge type is bridged_fastdp
INFO: 2021/01/02 15:33:11.884419 Communication between peers via untrusted networks is encrypted.
INFO: 2021/01/02 15:33:12.039300 Our name is 00:00:00:00:00:01(rkon4)
INFO: 2021/01/02 15:33:12.039366 Launch detected - using supplied peer list: []
INFO: 2021/01/02 15:33:12.069297 Docker API on unix:///var/run/docker.sock: &[GoVersion=go1.13.10 KernelVersion=5.4.0-58-generic BuildTime=2020-06-01T09:11:07.000000000+00:00 ApiVersion=1.40 GitCommit=42e35e61f3 Version=19.03.11 MinAPIVersion=1.12 Os=linux Arch=amd64 Platform={"Name":"Docker Engine - Community"} Components=[{"Details":{"ApiVersion":"1.40","Arch":"amd64","BuildTime":"2020-06-01T09:11:07.000000000+00:00","Experimental":"false","GitCommit":"42e35e61f3","GoVersion":"go1.13.10","KernelVersion":"5.4.0-58-generic","MinAPIVersion":"1.12","Os":"linux"},"Name":"Engine","Version":"19.03.11"},{"Details":{"GitCommit":"269548fa27e0089a8b8278fc4fc781d7f65a939b"},"Name":"containerd","Version":"1.4.3"},{"Details":{"GitCommit":"ff819c7e9184c13b7c2607fe6c30ae19403a7aff"},"Name":"runc","Version":"1.0.0-rc92"},{"Details":{"GitCommit":"fec3683"},"Name":"docker-init","Version":"0.18.0"}]]
INFO: 2021/01/02 15:33:12.084394 Checking for pre-existing addresses on weave bridge
INFO: 2021/01/02 15:33:12.090121 [allocator 00:00:00:00:00:01] No valid persisted data
INFO: 2021/01/02 15:33:12.141734 [allocator 00:00:00:00:00:01] Initialising with supplied IPAM seed
INFO: 2021/01/02 15:33:12.189180 Listening for DNS queries on 172.17.0.1:53
INFO: 2021/01/02 15:33:12.195768 Sniffing traffic on datapath (via ODP)
INFO: 2021/01/02 15:33:12.213149 Listening for HTTP control messages on 127.0.0.1:6784
INFO: 2021/01/02 15:33:12.213673 Listening for status+metrics requests on 127.0.0.1:6782
2021/01/02 15:33:12 http: panic serving 127.0.0.1:39144: runtime error: slice bounds out of range [:9] with length 8
goroutine 167 [running]:
net/http.(*conn).serve.func1(0xc0003da8c0)
        /usr/local/go/src/net/http/server.go:1767 +0x139
panic(0x18db0e0, 0xc00044cce0)
        /usr/local/go/src/runtime/panic.go:679 +0x1b2
github.com/weaveworks/weave/vendor/github.com/miekg/dns.ClientConfigFromReader(0x1c5daa0, 0xc0000ca5e0, 0x0, 0x0, 0xc0000ca5e0)
        /go/src/github.com/weaveworks/weave/vendor/github.com/miekg/dns/clientconfig.go:94 +0x8d1
github.com/weaveworks/weave/vendor/github.com/miekg/dns.ClientConfigFromFile(0x7ffc3ef9be42, 0x23, 0x0, 0x0, 0x0)
        /go/src/github.com/weaveworks/weave/vendor/github.com/miekg/dns/clientconfig.go:29 +0xde
github.com/weaveworks/weave/nameserver.(*upstream).Config(0xc0004f32c0, 0x0, 0x0, 0x0)
        /go/src/github.com/weaveworks/weave/nameserver/dns.go:66 +0x1ac
github.com/weaveworks/weave/nameserver.NewStatus(0xc000294000, 0xc000294070, 0x0)
        /go/src/github.com/weaveworks/weave/nameserver/status.go:39 +0x2c9
main.HandleHTTP.func1(0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
        /go/src/github.com/weaveworks/weave/prog/weaver/http.go:315 +0x134
main.HandleHTTP.func4.1(0x1c95780, 0xc0002f2000, 0xc000024900)
        /go/src/github.com/weaveworks/weave/prog/weaver/http.go:359 +0x44
net/http.HandlerFunc.ServeHTTP(0xc00043c0a0, 0x1c95780, 0xc0002f2000, 0xc000024900)
        /usr/local/go/src/net/http/server.go:2007 +0x44
github.com/weaveworks/weave/vendor/github.com/gorilla/mux.(*Router).ServeHTTP(0xc00022e150, 0x1c95780, 0xc0002f2000, 0xc000024900)
        /go/src/github.com/weaveworks/weave/vendor/github.com/gorilla/mux/mux.go:159 +0x104
github.com/weaveworks/weave/common.(*loggingHandler).ServeHTTP(0xc000446eb0, 0x1c95780, 0xc0002f2000, 0xc000024700)
        /go/src/github.com/weaveworks/weave/common/http.go:13 +0x139
net/http.(*ServeMux).ServeHTTP(0x2a5fde0, 0x1c95780, 0xc0002f2000, 0xc000024700)
        /usr/local/go/src/net/http/server.go:2387 +0x1bd
net/http.serverHandler.ServeHTTP(0xc0002d4000, 0x1c95780, 0xc0002f2000, 0xc000024700)
        /usr/local/go/src/net/http/server.go:2802 +0xa4
net/http.(*conn).serve(0xc0003da8c0, 0x1c9c280, 0xc00009c680)
        /usr/local/go/src/net/http/server.go:1890 +0x875
created by net/http.(*Server).Serve
        /usr/local/go/src/net/http/server.go:2927 +0x38e
INFO: 2021/01/02 15:33:25.155960 Creating container with WEAVE_CIDR "10.39.240.1/12"
INFO: 2021/01/02 15:33:27.693423 Attaching container 96581298e4740dd1f065fa6ca2eb0113cdc047af9c9911bfb37219d6ef674b9a with WEAVE_CIDR "10.39.240.1/12" to weave network
INFO: 2021/01/02 15:33:27.695398 [allocator 00:00:00:00:00:01] Address 10.39.240.1/12 claimed by 96581298e4740dd1f065fa6ca2eb0113cdc047af9c9911bfb37219d6ef674b9a - not in our range
INFO: 2021/01/02 15:33:30.257984 [nameserver 00:00:00:00:00:01] adding entry for 96581298e4740dd1f065fa6ca2eb0113cdc047af9c9911bfb37219d6ef674b9a: overnode.weave.local. -> 10.39.240.1
2021/01/02 15:33:36 http: panic serving 127.0.0.1:39152: runtime error: slice bounds out of range [:9] with length 8
goroutine 322 [running]:
net/http.(*conn).serve.func1(0xc0000d43c0)
        /usr/local/go/src/net/http/server.go:1767 +0x139
panic(0x18db0e0, 0xc000047640)
        /usr/local/go/src/runtime/panic.go:679 +0x1b2
github.com/weaveworks/weave/vendor/github.com/miekg/dns.ClientConfigFromReader(0x1c5daa0, 0xc0000100e0, 0x0, 0x0, 0xc0000100e0)
        /go/src/github.com/weaveworks/weave/vendor/github.com/miekg/dns/clientconfig.go:94 +0x8d1
github.com/weaveworks/weave/vendor/github.com/miekg/dns.ClientConfigFromFile(0x7ffc3ef9be42, 0x23, 0x0, 0x0, 0x0)
        /go/src/github.com/weaveworks/weave/vendor/github.com/miekg/dns/clientconfig.go:29 +0xde
github.com/weaveworks/weave/nameserver.(*upstream).Config(0xc0004f32c0, 0x0, 0x0, 0x0)
        /go/src/github.com/weaveworks/weave/nameserver/dns.go:66 +0x1ac
github.com/weaveworks/weave/nameserver.NewStatus(0xc000294000, 0xc000294070, 0x0)
        /go/src/github.com/weaveworks/weave/nameserver/status.go:39 +0x2c9
main.HandleHTTP.func1(0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
        /go/src/github.com/weaveworks/weave/prog/weaver/http.go:315 +0x134
main.HandleHTTP.func4.1(0x1c95780, 0xc0009741c0, 0xc00037a600)
        /go/src/github.com/weaveworks/weave/prog/weaver/http.go:359 +0x44
net/http.HandlerFunc.ServeHTTP(0xc00043c0a0, 0x1c95780, 0xc0009741c0, 0xc00037a600)
        /usr/local/go/src/net/http/server.go:2007 +0x44
github.com/weaveworks/weave/vendor/github.com/gorilla/mux.(*Router).ServeHTTP(0xc00022e150, 0x1c95780, 0xc0009741c0, 0xc00037a600)
        /go/src/github.com/weaveworks/weave/vendor/github.com/gorilla/mux/mux.go:159 +0x104
github.com/weaveworks/weave/common.(*loggingHandler).ServeHTTP(0xc000446eb0, 0x1c95780, 0xc0009741c0, 0xc00037a400)
        /go/src/github.com/weaveworks/weave/common/http.go:13 +0x139
net/http.(*ServeMux).ServeHTTP(0x2a5fde0, 0x1c95780, 0xc0009741c0, 0xc00037a400)
        /usr/local/go/src/net/http/server.go:2387 +0x1bd
net/http.serverHandler.ServeHTTP(0xc0002d4000, 0x1c95780, 0xc0009741c0, 0xc00037a400)
        /usr/local/go/src/net/http/server.go:2802 +0xa4
net/http.(*conn).serve(0xc0000d43c0, 0x1c9c280, 0xc00009c100)
        /usr/local/go/src/net/http/server.go:1890 +0x875
created by net/http.(*Server).Serve
        /usr/local/go/src/net/http/server.go:2927 +0x38e
jtagcat commented 3 years ago

https://github.com/weaveworks/weave/issues/3868

avkonst commented 3 years ago

thanks for letting me know

On Sun, 3 Jan 2021, 04:59 jtagcat, notifications@github.com wrote:

weaveworks/weave#3868 https://github.com/weaveworks/weave/issues/3868

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/overnode-org/overnode/issues/56#issuecomment-753490285, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA6JSVIERMP5ZISGT2LH3SDSX47GVANCNFSM4VRELQIA .

snoby commented 3 years ago

I'm surprised this is still a thing with an LTS release that is over a year old. I guess the answer here is to pull down overnode.sh and change the version string for weave_version and run it locally on all my nodes? To work around you have to download the overnode.sh file and DO NOT DO an upgrade.

you have to chnag ehte weave version in the file and then do an ./overnode.sh --debug install --force

but the old version of weave is still running at this point. I'm sure there is a better way, but I just rebooted the node at this point. And the old version was still running, there is no "uninstall command"

As a side note, this bash script to do the install has reached a level of complexity that brings into question perhaps some other orchestration tool should be used such as ansible.

jtagcat commented 3 years ago

Did you get it working with 20.04 or not?

I'm surprised this is still a thing with an LTS release that is over a year old.

LTS stands for Long Time Support — your 18.04 installations continue working (during support time) without needing changes. 18.04 is fine till 2023 or 2028, depending on how you read it.

Of course, software development responds to current and upcoming demand. The demand here is in single digits, if even that.

For installations of overnode I support, jumping from 18.04 to 22.04 is fine.

I guess the answer here is to pull down overnode.sh and change the version string for weave_version and run it locally on all my nodes?

You could edit it once, and point it to yourself for the curl-bash.

but the old version of weave is still running at this point. I'm sure there is a better way, but I just rebooted the node at this point. And the old version was still running, there is no "uninstall command"

overnode reset

As a side note, this bash script to do the install has reached a level of complexity that brings into question perhaps some other orchestration tool should be used such as ansible.

The note expresses the superiority of bash to other orchestration tools, but I think you meant the other way around.
I agree, migrating from bash wouldn't be bad. Make a PR.

avkonst commented 3 years ago

Bash was an intentional choice to keep it simple. 2/3 of the script is verification of the command line arguments. If we need orchestration spanning further of docker-compose, then Kubernetes, Docker-Swarm, etc... If I had a lot of time and this tool had wide adoption and faced a new feature request which would be impossible in Bash, then I would rewrite it in Rust.

snoby commented 3 years ago

I hacked the install script to not download and run against itself and I changed versions of weave to 2.8.1 and it still crashes with the same result. I like the idea of overnode. It makes a lot of sense for small deployments that don't need all that kubernete offers. In your defense I was stating it's crazy that WEAVE doesn't support 21.04 after it being out for almost a year. The linked issue has a patch that has been available since November 2020, so 5 months later it still hasn't been merged in and a release made? Crazy man, crazy.

avkonst commented 3 years ago

Yes, weavenet issue is very disappointing. It makes me thinking if it is reliable enough to build on top of it? My current plan is to keep 18.04 and comeback to this later. Maybe I will need to switch to a different Linux flavour, which is actually used by weavenet itself, but what is it? Alpine? Core OS? If it is not reliable and should be abandoned, then, I am afraid, overnode is doomed.

jtagcat commented 3 years ago

To be honest, k8s is the same way — PRs are ready to merge, but there's not enough authority to do a final review and merge. :ˇ)

Daxcor69 commented 3 years ago

Is this still the case? It is broken for 20.04?

jtagcat commented 3 years ago

Yes.

karser commented 2 years ago

Encountered this issue on ubuntu 22.04 because of options edns0 trust-ad in my /etc/resolv.conf. What's weird is that on ubuntu 20.04 it was options edns0 and weave was working well.

jtagcat commented 2 years ago

I've moved on to k8s. I've abandoned my involvement in the project.