search

docker / for-linux

Docker Engine for Linux
https://docs.docker.com/engine/installation/
756 stars 86 forks source link

Docker swarm successfully joined the swarm cluster as a manager node, but frequently failed to start a container(pxc) using the swarm network #1448

Closed wiserK closed 1 year ago

wiserK commented 1 year ago

Description：

I run the docker server on the openEuler system. All three nodes have successfully joined the swarm as manager node, but when I use: 

docker run - d - p 3306:3306 - p 4567-4568:4567-4568 CLUSTER_ NAME=PXC -e CLUSTER_ JOIN=pn0001...... --name=pn0003       --net=swarm_ mysql pxc

to start the pxc container, it always fails to start.

Output of docker version:

Client:
 Version:           18.09.0
 EulerVersion:      18.09.0.243
 API version:       1.39
 Go version:        go1.15.7
 Git commit:        ce42130
 Built:             Wed Sep 28 03:21:31 2022
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          18.09.0
  EulerVersion:     18.09.0.243
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.15.7
  Git commit:       ce42130
  Built:            Wed Sep 28 03:20:45 2022
  OS/Arch:          linux/amd64
  Experimental:     false

RPM package of docker: docker-engine-18.09.0-243.oe1.x86_64

Output of docker node ls:

ID                                               HOSTNAME            STATUS              AVAILABILITY        MANAGER STATUS      ENGINE VERSION
z8h03jwvanegkj8nqg8cqut52     moss-0001             Ready                     Active                      Leader                          18.09.0
r3cmi4clslnjg2yceemly059n        moss-0002             Ready                     Active                      Reachable                    18.09.0
qb1hi7b3stvta8k6air156uw4 *    moss-0003             Ready                     Active                      Reachable                     18.09.0

Output of /etc/os-release:

NAME="openEuler"
VERSION="20.03 (LTS-SP3)"
ID="openEuler"
VERSION_ID="20.03"
PRETTY_NAME="openEuler 20.03 (LTS-SP3)"
ANSI_COLOR="0;31"

The log of /var/log/messages:

> Feb  8 10:34:24 moss-0003 dockerd[19904]: time="2023-02-08T10:34:24.686430006+08:00" level=error msg="Error in responding to bulk sync from node 172.30.0.11: failed to send a TCP message during bulk sync: write tcp 172.30.0.12:47180->172.30.0.11:7946: write: connection reset by peer"
Feb  8 10:34:25 moss-0003 dockerd[19904]: time="2023-02-08T10:34:25.413708227+08:00" level=error msg="Bulk sync to node 70561e9142e7 timed out"
Feb  8 10:34:25 moss-0003 dockerd[19904]: time="2023-02-08T10:34:25.413999313+08:00" level=warning msg="bulk sync to node 70561e9142e7 failed: failed to send a TCP message during bulk sync: write tcp 172.30.0.12:47182->172.30.0.11:7946: write: connection reset by peer"
Feb  8 10:34:56 moss-0003 dockerd[19904]: time="2023-02-08T10:34:56.142864118+08:00" level=error msg="Error in responding to bulk sync from node 172.30.0.10: failed to send a TCP message during bulk sync: write tcp 172.30.0.12:53394->172.30.0.10:7946: write: connection reset by peer"
Feb  8 10:35:24 moss-0003 dockerd[19904]: time="2023-02-08T10:35:24.689036121+08:00" level=error msg="Error in responding to bulk sync from node 172.30.0.11: failed to send a TCP message during bulk sync: write tcp 172.30.0.12:47242->172.30.0.11:7946: write: connection reset by peer"
Feb  8 10:35:26 moss-0003 dockerd[19904]: time="2023-02-08T10:35:26.144664216+08:00" level=error msg="Error in responding to bulk sync from node 172.30.0.10: failed to send a TCP message during bulk sync: write tcp 172.30.0.12:53428->172.30.0.10:7946: write: connection reset by peer"
Feb  8 10:36:01 moss-0003 systemd[1]: Started Session 19 of user root.
Feb  8 10:36:13 moss-0003 docker-client[23183]: time="2023-02-08T10:36:13+08:00" level=info msg="received command [docker rm pn0003] from parent [23170] cmdline [/usr/bin/sh /moss/ms_moss/set_pxc_cluster.sh --join 0001 ]"
Feb  8 10:36:14 moss-0003 systemd[1656]: mgt-docker-overlay2-f261fe8e54a8aedd92300d6e188bca0c848f90fb8cd5f5bf90ecf4a8ee939539\x2dinit-merged.mount: Succeeded.
Feb  8 10:36:14 moss-0003 systemd[1]: mgt-docker-overlay2-f261fe8e54a8aedd92300d6e188bca0c848f90fb8cd5f5bf90ecf4a8ee939539\x2dinit-merged.mount: Succeeded.
Feb  8 10:36:14 moss-0003 systemd[1]: mgt-docker-overlay2-f261fe8e54a8aedd92300d6e188bca0c848f90fb8cd5f5bf90ecf4a8ee939539-merged.mount: Succeeded.
Feb  8 10:36:14 moss-0003 systemd[1656]: mgt-docker-overlay2-f261fe8e54a8aedd92300d6e188bca0c848f90fb8cd5f5bf90ecf4a8ee939539-merged.mount: Succeeded.
Feb  8 10:36:14 moss-0003 dockerd[19904]: time="2023-02-08T10:36:14.283213674+08:00" level=info msg="Container create done(Name: /pn0003 ID: 19e2ae3229564a61e048b3a862424835ac08badfa1b3df32595c1beaa57f29af)"
Feb  8 10:36:25 moss-0003 dockerd[19904]: time="2023-02-08T10:36:25.413400511+08:00" level=error msg="Bulk sync to node 59a0d7a83cbf timed out"
Feb  8 10:36:26 moss-0003 dockerd[19904]: time="2023-02-08T10:36:26.143441429+08:00" level=error msg="Error in responding to bulk sync from node 172.30.0.10: failed to send a TCP message during bulk sync: write tcp 172.30.0.12:53508->172.30.0.10:7946: write: connection reset by peer"
Feb  8 10:36:34 moss-0003 systemd[1656]: mgt-docker-overlay2-f261fe8e54a8aedd92300d6e188bca0c848f90fb8cd5f5bf90ecf4a8ee939539-merged.mount: Succeeded.
Feb  8 10:36:34 moss-0003 systemd[1]: mgt-docker-overlay2-f261fe8e54a8aedd92300d6e188bca0c848f90fb8cd5f5bf90ecf4a8ee939539-merged.mount: Succeeded.
Feb  8 10:36:34 moss-0003 dockerd[19904]: time="2023-02-08T10:36:34.321524652+08:00" level=error msg="19e2ae3229564a61e048b3a862424835ac08badfa1b3df32595c1beaa57f29af cleanup: failed to delete container from containerd, try to delete directly: no such container"
Feb  8 10:36:34 moss-0003 docker-runc[23239]: time="2023-02-08T10:36:34+08:00" level=error msg="container \"19e2ae3229564a61e048b3a862424835ac08badfa1b3df32595c1beaa57f29af\" does not exist\n"
Feb  8 10:36:34 moss-0003 dockerd[19904]: time="2023-02-08T10:36:34.326832769+08:00" level=error msg="19e2ae3229564a61e048b3a862424835ac08badfa1b3df32595c1beaa57f29af cleanup: failed to directly delete container"
Feb  8 10:36:34 moss-0003 dockerd[19904]: time="2023-02-08T10:36:34.326862557+08:00" level=error msg="Handler for POST /v1.39/containers/19e2ae3229564a61e048b3a862424835ac08badfa1b3df32595c1beaa57f29af/start returned error: attaching to network failed, make sure your network options are correct and check manager logs: context deadline exceeded"
Feb  8 10:36:38 moss-0003 docker-client[23266]: time="2023-02-08T10:36:38+08:00" level=info msg="received command [docker rm pn0003] from parent [23170] cmdline [/usr/bin/sh /moss/ms_moss/set_pxc_cluster.sh --join 0001 ]"
Feb  8 10:36:38 moss-0003 systemd[1]: mgt-docker-overlay2-a38320de9a6805dea540b8cc215ad3519f80d1e29cdb400113716283eda871a7\x2dinit-merged.mount: Succeeded.
Feb  8 10:36:38 moss-0003 systemd[1656]: mgt-docker-overlay2-a38320de9a6805dea540b8cc215ad3519f80d1e29cdb400113716283eda871a7\x2dinit-merged.mount: Succeeded.
Feb  8 10:36:38 moss-0003 systemd[1656]: mgt-docker-overlay2-a38320de9a6805dea540b8cc215ad3519f80d1e29cdb400113716283eda871a7-merged.mount: Succeeded.
Feb  8 10:36:38 moss-0003 systemd[1]: mgt-docker-overlay2-a38320de9a6805dea540b8cc215ad3519f80d1e29cdb400113716283eda871a7-merged.mount: Succeeded.
Feb  8 10:36:38 moss-0003 dockerd[19904]: time="2023-02-08T10:36:38.654857494+08:00" level=info msg="Container create done(Name: /pn0003 ID: 04178db3a4deb059547c24a2dd8ae145aca415c842089125b15306c3fd3cf33c)"
Feb  8 10:36:44 moss-0003 dockerd[19904]: time="2023-02-08T10:36:44.468580696+08:00" level=error msg="Bulk sync to node 59a0d7a83cbf timed out"
Feb  8 10:36:55 moss-0003 dockerd[19904]: time="2023-02-08T10:36:55.413776495+08:00" level=error msg="Bulk sync to node 70561e9142e7 timed out"
Feb  8 10:36:55 moss-0003 dockerd[19904]: time="2023-02-08T10:36:55.414258687+08:00" level=warning msg="bulk sync to node 70561e9142e7 failed: failed to send a TCP message during bulk sync: write tcp 172.30.0.12:47362->172.30.0.11:7946: write: connection reset by peer"
Feb  8 10:37:14 moss-0003 dockerd[19904]: time="2023-02-08T10:37:14.469118873+08:00" level=error msg="Bulk sync to node 70561e9142e7 timed out"
Feb  8 10:37:14 moss-0003 dockerd[19904]: time="2023-02-08T10:37:14.552840963+08:00" level=error msg="error reading the kernel parameter net.ipv4.vs.expire_nodest_conn" error="open /proc/sys/net/ipv4/vs/expire_nodest_conn: no such file or directory"
Feb  8 10:37:14 moss-0003 kernel: [ 1474.701575] br0: renamed from ov-001001-06lgr
Feb  8 10:37:14 moss-0003 Keepalived_vrrp[5969]: Interface vx-001001-06lgr added
Feb  8 10:37:14 moss-0003 systemd-udevd[23407]: Using default interface naming scheme 'v246'.
Feb  8 10:37:14 moss-0003 Keepalived_vrrp[5969]: Interface vx-001001-06lgr deleted
Feb  8 10:37:14 moss-0003 systemd-udevd[23407]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.
Feb  8 10:37:14 moss-0003 kernel: [ 1474.723064] vxlan0: renamed from vx-001001-06lgr
Feb  8 10:37:14 moss-0003 systemd-udevd[23407]: ethtool: could not get ethtool features for vx-001001-06lgr
Feb  8 10:37:14 moss-0003 systemd-udevd[23407]: Could not set offload features of vx-001001-06lgr: No such device
Feb  8 10:37:14 moss-0003 kernel: [ 1474.733900] br0: port 1(vxlan0) entered blocking state
Feb  8 10:37:14 moss-0003 kernel: [ 1474.734388] br0: port 1(vxlan0) entered disabled state
Feb  8 10:37:14 moss-0003 kernel: [ 1474.734871] device vxlan0 entered promiscuous mode
Feb  8 10:37:14 moss-0003 kernel: [ 1474.735416] br0: port 1(vxlan0) entered blocking state
Feb  8 10:37:14 moss-0003 kernel: [ 1474.735851] br0: port 1(vxlan0) entered forwarding state
Feb  8 10:37:14 moss-0003 Keepalived_vrrp[5969]: Interface veth564cf2b added
Feb  8 10:37:14 moss-0003 Keepalived_vrrp[5969]: Interface vethe3e0780 added
Feb  8 10:37:14 moss-0003 systemd-udevd[23419]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.
Feb  8 10:37:14 moss-0003 systemd-udevd[23407]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.
Feb  8 10:37:14 moss-0003 systemd-udevd[23419]: Using default interface naming scheme 'v246'.
Feb  8 10:37:14 moss-0003 NetworkManager[1138]: <info>  [1675823834.6968] manager: (veth564cf2b): new Veth device (/org/freedesktop/NetworkManager/Devices/65)
Feb  8 10:37:14 moss-0003 NetworkManager[1138]: <info>  [1675823834.6977] manager: (vethe3e0780): new Veth device (/org/freedesktop/NetworkManager/Devices/66)
Feb  8 10:37:14 moss-0003 Keepalived_vrrp[5969]: Interface vethe3e0780 deleted
Feb  8 10:37:14 moss-0003 kernel: [ 1474.783900] veth0: renamed from vethe3e0780
Feb  8 10:37:14 moss-0003 kernel: [ 1474.791913] br0: port 2(veth0) entered blocking state
Feb  8 10:37:14 moss-0003 kernel: [ 1474.792429] br0: port 2(veth0) entered disabled state
Feb  8 10:37:14 moss-0003 kernel: [ 1474.792909] device veth0 entered promiscuous mode
Feb  8 10:37:14 moss-0003 kernel: [ 1474.793408] br0: port 2(veth0) entered blocking state
Feb  8 10:37:14 moss-0003 kernel: [ 1474.793840] br0: port 2(veth0) entered forwarding state
Feb  8 10:37:14 moss-0003 Keepalived_vrrp[5969]: (veth564cf2b) MAC Address changed from ba:10:bd:1a:ac:5b to 02:42:0a:00:00:0f
Feb  8 10:37:14 moss-0003 Keepalived_vrrp[5969]: Interface veth564cf2b deleted
Feb  8 10:37:14 moss-0003 kernel: [ 1474.877131] eth0: renamed from veth564cf2b
Feb  8 10:37:14 moss-0003 kernel: [ 1474.893959] br0: port 2(veth0) entered disabled state
Feb  8 10:37:14 moss-0003 kernel: [ 1474.894434] br0: port 1(vxlan0) entered disabled state
Feb  8 10:37:14 moss-0003 kernel: [ 1474.894992] ov-001001-06lgr: renamed from br0
Feb  8 10:37:14 moss-0003 kernel: [ 1474.905844] device veth0 left promiscuous mode
Feb  8 10:37:14 moss-0003 kernel: [ 1474.906425] ov-001001-06lgr: port 2(veth0) entered disabled state
Feb  8 10:37:14 moss-0003 kernel: [ 1474.916822] device vxlan0 left promiscuous mode
Feb  8 10:37:14 moss-0003 kernel: [ 1474.917396] ov-001001-06lgr: port 1(vxlan0) entered disabled state
Feb  8 10:37:14 moss-0003 kernel: [ 1474.945100] vx-001001-06lgr: renamed from vxlan0
Feb  8 10:37:14 moss-0003 Keepalived_vrrp[5969]: Interface vx-001001-06lgr added
Feb  8 10:37:14 moss-0003 systemd-udevd[23443]: Using default interface naming scheme 'v246'.
Feb  8 10:37:14 moss-0003 kernel: [ 1474.963950] vethe3e0780: renamed from veth0
Feb  8 10:37:14 moss-0003 systemd-udevd[23443]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.
Feb  8 10:37:14 moss-0003 Keepalived_vrrp[5969]: Interface vethe3e0780 added
Feb  8 10:37:14 moss-0003 Keepalived_vrrp[5969]: Interface vx-001001-06lgr deleted
Feb  8 10:37:14 moss-0003 systemd-udevd[23443]: ethtool: could not set ethtool features for vx-001001-06lgr
Feb  8 10:37:14 moss-0003 systemd-udevd[23443]: Could not set offload features of vx-001001-06lgr: Operation not permitted
Feb  8 10:37:14 moss-0003 systemd-udevd[23419]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.
Feb  8 10:37:14 moss-0003 NetworkManager[1138]: <info>  [1675823834.9073] manager: (vethe3e0780): new Veth device (/org/freedesktop/NetworkManager/Devices/67)
Feb  8 10:37:14 moss-0003 systemd[1656]: run-docker-netns-1\x2d06lgrttbrb.mount: Succeeded.
Feb  8 10:37:14 moss-0003 systemd[1]: run-docker-netns-1\x2d06lgrttbrb.mount: Succeeded.
Feb  8 10:37:14 moss-0003 kernel: [ 1475.000178] veth564cf2b: renamed from eth0
Feb  8 10:37:14 moss-0003 Keepalived_vrrp[5969]: Interface veth564cf2b added
Feb  8 10:37:14 moss-0003 systemd-udevd[23443]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.
Feb  8 10:37:14 moss-0003 NetworkManager[1138]: <info>  [1675823834.9413] manager: (veth564cf2b): new Veth device (/org/freedesktop/NetworkManager/Devices/68)
Feb  8 10:37:14 moss-0003 dockerd[19904]: time="2023-02-08T10:37:14.944249543+08:00" level=warning msg="Error (Unable to complete atomic operation, key modified) deleting object [endpoint 06lgrttbrbagkro7aje6kppyc e3f3208bd97c54f656d6e428c824e98981e7c9b81583bf29381ec335b7b2343f], retrying...."
Feb  8 10:37:14 moss-0003 Keepalived_vrrp[5969]: Interface veth564cf2b deleted
Feb  8 10:37:14 moss-0003 Keepalived_vrrp[5969]: Interface vethe3e0780 deleted
Feb  8 10:37:14 moss-0003 systemd[1656]: run-docker-netns-lb_06lgrttbr.mount: Succeeded.
Feb  8 10:37:14 moss-0003 systemd[1]: run-docker-netns-lb_06lgrttbr.mount: Succeeded.
Feb  8 10:37:14 moss-0003 dockerd[19904]: time="2023-02-08T10:37:14.972408829+08:00" level=error msg="failed removing container name resolution for 444b0136a591995b3d4be96b27225d50e9a351d5c8498299190d4c8ba8ef5d9f epRec:{pn0001   <nil> 10.0.0.2 [] [] [52068d83d7ad] false} err:network 06lgrttbrbagkro7aje6kppyc not found"
Feb  8 10:37:14 moss-0003 dockerd[19904]: time="2023-02-08T10:37:14.972438314+08:00" level=error msg="failed removing container name resolution for 9fc52d6ed27f5a297174570203d739c5a9cc145de4322e5253efe6a1b195c595 epRec:{pn0002   <nil> 10.0.0.6 [] [] [6df191edffb5] false} err:network 06lgrttbrbagkro7aje6kppyc not found"
Feb  8 10:37:14 moss-0003 systemd[1656]: mgt-docker-overlay2-a38320de9a6805dea540b8cc215ad3519f80d1e29cdb400113716283eda871a7-merged.mount: Succeeded.
Feb  8 10:37:14 moss-0003 systemd[1]: mgt-docker-overlay2-a38320de9a6805dea540b8cc215ad3519f80d1e29cdb400113716283eda871a7-merged.mount: Succeeded.
Feb  8 10:37:14 moss-0003 dockerd[19904]: time="2023-02-08T10:37:14.988609352+08:00" level=error msg="04178db3a4deb059547c24a2dd8ae145aca415c842089125b15306c3fd3cf33c cleanup: failed to delete container from containerd, try to delete directly: no such container"
Feb  8 10:37:14 moss-0003 docker-runc[23496]: time="2023-02-08T10:37:14+08:00" level=error msg="container \"04178db3a4deb059547c24a2dd8ae145aca415c842089125b15306c3fd3cf33c\" does not exist\n"
Feb  8 10:37:14 moss-0003 dockerd[19904]: time="2023-02-08T10:37:14.994074551+08:00" level=error msg="04178db3a4deb059547c24a2dd8ae145aca415c842089125b15306c3fd3cf33c cleanup: failed to directly delete container"
Feb  8 10:37:14 moss-0003 dockerd[19904]: time="2023-02-08T10:37:14.994103713+08:00" level=error msg="Handler for POST /v1.39/containers/04178db3a4deb059547c24a2dd8ae145aca415c842089125b15306c3fd3cf33c/start returned error: failed to get network during CreateEndpoint: network 06lgrttbrbagkro7aje6kppyc not found"
Feb  8 10:37:17 moss-0003 systemd[1]: session-19.scope: Succeeded.
Feb  8 10:37:55 moss-0003 dockerd[19904]: time="2023-02-08T10:37:55.413529281+08:00" level=error msg="Bulk sync to node 59a0d7a83cbf timed out"
Feb  8 10:38:01 moss-0003 systemd[1]: Started Session 20 of user root.
Feb  8 10:38:12 moss-0003 docker-client[23862]: time="2023-02-08T10:38:12+08:00" level=info msg="received command [docker rm pn0003] from parent [23849] cmdline [/usr/bin/sh /moss/ms_moss/set_pxc_cluster.sh --join 0001 ]"
Feb  8 10:38:13 moss-0003 systemd[1656]: mgt-docker-overlay2-20612d819544aa250e38f62831d2cc416a1f4f004b18cd89da6f31805d1f7966\x2dinit-merged.mount: Succeeded.
Feb  8 10:38:13 moss-0003 systemd[1]: mgt-docker-overlay2-20612d819544aa250e38f62831d2cc416a1f4f004b18cd89da6f31805d1f7966\x2dinit-merged.mount: Succeeded.
Feb  8 10:38:13 moss-0003 systemd[1656]: mgt-docker-overlay2-20612d819544aa250e38f62831d2cc416a1f4f004b18cd89da6f31805d1f7966-merged.mount: Succeeded.
Feb  8 10:38:13 moss-0003 systemd[1]: mgt-docker-overlay2-20612d819544aa250e38f62831d2cc416a1f4f004b18cd89da6f31805d1f7966-merged.mount: Succeeded.
Feb  8 10:38:13 moss-0003 dockerd[19904]: time="2023-02-08T10:38:13.248012802+08:00" level=info msg="Container create done(Name: /pn0003 ID: bd1186cb6ea74ee3a6509f4375320204e645f6054b46132ac427d37f66ed889b)"
Feb  8 10:38:33 moss-0003 systemd[1656]: mgt-docker-overlay2-20612d819544aa250e38f62831d2cc416a1f4f004b18cd89da6f31805d1f7966-merged.mount: Succeeded.
Feb  8 10:38:33 moss-0003 systemd[1]: mgt-docker-overlay2-20612d819544aa250e38f62831d2cc416a1f4f004b18cd89da6f31805d1f7966-merged.mount: Succeeded.
Feb  8 10:38:33 moss-0003 dockerd[19904]: time="2023-02-08T10:38:33.285492675+08:00" level=error msg="bd1186cb6ea74ee3a6509f4375320204e645f6054b46132ac427d37f66ed889b cleanup: failed to delete container from containerd, try to delete directly: no such container"
Feb  8 10:38:33 moss-0003 docker-runc[23921]: time="2023-02-08T10:38:33+08:00" level=error msg="container \"bd1186cb6ea74ee3a6509f4375320204e645f6054b46132ac427d37f66ed889b\" does not exist\n"
Feb  8 10:38:33 moss-0003 dockerd[19904]: time="2023-02-08T10:38:33.291226952+08:00" level=error msg="bd1186cb6ea74ee3a6509f4375320204e645f6054b46132ac427d37f66ed889b cleanup: failed to directly delete container"
Feb  8 10:38:33 moss-0003 dockerd[19904]: time="2023-02-08T10:38:33.291261309+08:00" level=error msg="Handler for POST /v1.39/containers/bd1186cb6ea74ee3a6509f4375320204e645f6054b46132ac427d37f66ed889b/start returned error: attaching to network failed, make sure your network options are correct and check manager logs: context deadline exceeded"
Feb  8 10:38:37 moss-0003 docker-client[23947]: time="2023-02-08T10:38:37+08:00" level=info msg="received command [docker rm pn0003] from parent [23849] cmdline [/usr/bin/sh /moss/ms_moss/set_pxc_cluster.sh --join 0001 ]"
Feb  8 10:38:37 moss-0003 systemd[1656]: mgt-docker-overlay2-91c6f6ee2e43f2f5025ade91bacc889a7ac4dccf69acd2b520471fbfa920c25b\x2dinit-merged.mount: Succeeded.
Feb  8 10:38:37 moss-0003 systemd[1]: mgt-docker-overlay2-91c6f6ee2e43f2f5025ade91bacc889a7ac4dccf69acd2b520471fbfa920c25b\x2dinit-merged.mount: Succeeded.
Feb  8 10:38:37 moss-0003 systemd[1]: mgt-docker-overlay2-91c6f6ee2e43f2f5025ade91bacc889a7ac4dccf69acd2b520471fbfa920c25b-merged.mount: Succeeded.
Feb  8 10:38:37 moss-0003 systemd[1656]: mgt-docker-overlay2-91c6f6ee2e43f2f5025ade91bacc889a7ac4dccf69acd2b520471fbfa920c25b-merged.mount: Succeeded.
Feb  8 10:38:37 moss-0003 dockerd[19904]: time="2023-02-08T10:38:37.626299027+08:00" level=info msg="Container create done(Name: /pn0003 ID: 68d7dd923743e83c1bc28f7381f2006dfb4aa78f4f694b17ea2bd4acdae024b0)"
Feb  8 10:38:43 moss-0003 dockerd[19904]: time="2023-02-08T10:38:43.437309477+08:00" level=error msg="Bulk sync to node 70561e9142e7 timed out"
Feb  8 10:38:43 moss-0003 dockerd[19904]: time="2023-02-08T10:38:43.515472991+08:00" level=error msg="error reading the kernel parameter net.ipv4.vs.expire_nodest_conn" error="open /proc/sys/net/ipv4/vs/expire_nodest_conn: no such file or directory"
Feb  8 10:38:43 moss-0003 kernel: [ 1563.667489] br0: renamed from ov-001001-06lgr
Feb  8 10:38:43 moss-0003 Keepalived_vrrp[5969]: Interface vx-001001-06lgr added
Feb  8 10:38:43 moss-0003 systemd-udevd[24082]: Using default interface naming scheme 'v246'.
Feb  8 10:38:43 moss-0003 Keepalived_vrrp[5969]: Interface vx-001001-06lgr deleted
Feb  8 10:38:43 moss-0003 systemd-udevd[24082]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.
Feb  8 10:38:43 moss-0003 kernel: [ 1563.692984] vxlan0: renamed from vx-001001-06lgr
Feb  8 10:38:43 moss-0003 systemd-udevd[24082]: ethtool: could not get ethtool features for vx-001001-06lgr
Feb  8 10:38:43 moss-0003 systemd-udevd[24082]: Could not set offload features of vx-001001-06lgr: No such device
Feb  8 10:38:43 moss-0003 kernel: [ 1563.700936] br0: port 1(vxlan0) entered blocking state
Feb  8 10:38:43 moss-0003 kernel: [ 1563.701701] br0: port 1(vxlan0) entered disabled state
Feb  8 10:38:43 moss-0003 kernel: [ 1563.702191] device vxlan0 entered promiscuous mode
Feb  8 10:38:43 moss-0003 kernel: [ 1563.702739] br0: port 1(vxlan0) entered blocking state
Feb  8 10:38:43 moss-0003 kernel: [ 1563.703169] br0: port 1(vxlan0) entered forwarding state
Feb  8 10:38:43 moss-0003 Keepalived_vrrp[5969]: Interface vetha513fc9 added
Feb  8 10:38:43 moss-0003 Keepalived_vrrp[5969]: Interface vethe3fd9c6 added
Feb  8 10:38:43 moss-0003 systemd-udevd[24082]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.
Feb  8 10:38:43 moss-0003 systemd-udevd[24094]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.
Feb  8 10:38:43 moss-0003 systemd-udevd[24094]: Using default interface naming scheme 'v246'.
Feb  8 10:38:43 moss-0003 NetworkManager[1138]: <info>  [1675823923.6652] manager: (vetha513fc9): new Veth device (/org/freedesktop/NetworkManager/Devices/69)
Feb  8 10:38:43 moss-0003 NetworkManager[1138]: <info>  [1675823923.6659] device (vethe3fd9c6): driver '(null)' does not support carrier detection.
Feb  8 10:38:43 moss-0003 Keepalived_vrrp[5969]: Interface vethe3fd9c6 deleted
Feb  8 10:38:43 moss-0003 NetworkManager[1138]: <info>  [1675823923.6893] device (vethe3fd9c6): driver 'veth' does not support carrier detection.
Feb  8 10:38:43 moss-0003 NetworkManager[1138]: <info>  [1675823923.6896] manager: (vethe3fd9c6): new Veth device (/org/freedesktop/NetworkManager/Devices/70)
Feb  8 10:38:43 moss-0003 kernel: [ 1563.764923] veth0: renamed from vethe3fd9c6
Feb  8 10:38:43 moss-0003 kernel: [ 1563.773796] br0: port 2(veth0) entered blocking state
Feb  8 10:38:43 moss-0003 kernel: [ 1563.774252] br0: port 2(veth0) entered disabled state
Feb  8 10:38:43 moss-0003 kernel: [ 1563.774746] device veth0 entered promiscuous mode
Feb  8 10:38:43 moss-0003 kernel: [ 1563.775252] br0: port 2(veth0) entered blocking state
Feb  8 10:38:43 moss-0003 kernel: [ 1563.775689] br0: port 2(veth0) entered forwarding state
Feb  8 10:38:43 moss-0003 Keepalived_vrrp[5969]: (vetha513fc9) MAC Address changed from fe:61:72:b6:2f:ee to 02:42:0a:00:00:0f
Feb  8 10:38:43 moss-0003 Keepalived_vrrp[5969]: Interface vetha513fc9 deleted
Feb  8 10:38:43 moss-0003 kernel: [ 1563.863187] eth0: renamed from vetha513fc9
Feb  8 10:38:43 moss-0003 kernel: [ 1563.885743] br0: port 2(veth0) entered disabled state
Feb  8 10:38:43 moss-0003 kernel: [ 1563.886224] br0: port 1(vxlan0) entered disabled state
Feb  8 10:38:43 moss-0003 kernel: [ 1563.886756] ov-001001-06lgr: renamed from br0
Feb  8 10:38:43 moss-0003 audit: ANOM_PROMISCUOUS dev=veth0 prom=0 old_prom=256 auid=4294967295 uid=0 gid=0 ses=4294967295
Feb  8 10:38:43 moss-0003 kernel: [ 1563.893754] device veth0 left promiscuous mode
Feb  8 10:38:43 moss-0003 kernel: [ 1563.894200] ov-001001-06lgr: port 2(veth0) entered disabled state
Feb  8 10:38:43 moss-0003 audit: ANOM_PROMISCUOUS dev=vxlan0 prom=0 old_prom=256 auid=4294967295 uid=0 gid=0 ses=4294967295
Feb  8 10:38:43 moss-0003 kernel: [ 1563.900819] device vxlan0 left promiscuous mode
Feb  8 10:38:43 moss-0003 kernel: [ 1563.901487] ov-001001-06lgr: port 1(vxlan0) entered disabled state
Feb  8 10:38:43 moss-0003 audit[19904]: SYSCALL arch=c000003e syscall=44 success=yes exit=32 a0=30 a1=c001424aa0 a2=20 a3=0 items=0 ppid=1 pid=19904 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="dockerd" exe="/usr/bin/dockerd" key=(null)
Feb  8 10:38:43 moss-0003 audit: PROCTITLE proctitle=2F7573722F62696E2F646F636B657264002D2D646174612D726F6F74002F6D67742F646F636B6572002D48007463703A2F2F302E302E302E303A32333735002D4800756E69783A2F2F2F7661722F72756E2F646F636B65722E736F636B002D48007463703A2F2F302E302E302E303A37363534
Feb  8 10:38:43 moss-0003 kernel: [ 1563.928994] vx-001001-06lgr: renamed from vxlan0
Feb  8 10:38:43 moss-0003 Keepalived_vrrp[5969]: Interface vx-001001-06lgr added
Feb  8 10:38:43 moss-0003 kernel: [ 1563.945161] vethe3fd9c6: renamed from veth0
Feb  8 10:38:43 moss-0003 systemd-udevd[24094]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.
Feb  8 10:38:43 moss-0003 Keepalived_vrrp[5969]: Interface vethe3fd9c6 added
Feb  8 10:38:43 moss-0003 Keepalived_vrrp[5969]: Interface vx-001001-06lgr deleted
Feb  8 10:38:43 moss-0003 systemd-udevd[24094]: ethtool: could not set ethtool features for vx-001001-06lgr
Feb  8 10:38:43 moss-0003 systemd-udevd[24094]: Could not set offload features of vx-001001-06lgr: Operation not permitted
Feb  8 10:38:43 moss-0003 systemd-udevd[24117]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.
Feb  8 10:38:43 moss-0003 systemd-udevd[24117]: Using default interface naming scheme 'v246'.
Feb  8 10:38:43 moss-0003 NetworkManager[1138]: <info>  [1675823923.8877] manager: (vethe3fd9c6): new Veth device (/org/freedesktop/NetworkManager/Devices/71)
Feb  8 10:38:43 moss-0003 systemd[1]: run-docker-netns-1\x2d06lgrttbrb.mount: Succeeded.
Feb  8 10:38:43 moss-0003 systemd[1656]: run-docker-netns-1\x2d06lgrttbrb.mount: Succeeded.
Feb  8 10:38:43 moss-0003 kernel: [ 1563.976422] vetha513fc9: renamed from eth0
Feb  8 10:38:43 moss-0003 Keepalived_vrrp[5969]: Interface vetha513fc9 added
Feb  8 10:38:43 moss-0003 systemd-udevd[24094]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.
Feb  8 10:38:43 moss-0003 NetworkManager[1138]: <info>  [1675823923.9273] manager: (vetha513fc9): new Veth device (/org/freedesktop/NetworkManager/Devices/72)
Feb  8 10:38:43 moss-0003 dockerd[19904]: time="2023-02-08T10:38:43.931499316+08:00" level=warning msg="Error (Unable to complete atomic operation, key modified) deleting object [endpoint 06lgrttbrbagkro7aje6kppyc be25b890d735898021f2301d8ef5c2bc97dcac68be9f97cb9773737dc52835a1], retrying...."
Feb  8 10:38:43 moss-0003 Keepalived_vrrp[5969]: Interface vetha513fc9 deleted
Feb  8 10:38:43 moss-0003 Keepalived_vrrp[5969]: Interface vethe3fd9c6 deleted
Feb  8 10:38:43 moss-0003 systemd[1656]: run-docker-netns-lb_06lgrttbr.mount: Succeeded.
Feb  8 10:38:43 moss-0003 systemd[1]: run-docker-netns-lb_06lgrttbr.mount: Succeeded.
Feb  8 10:38:43 moss-0003 systemd[1656]: mgt-docker-overlay2-91c6f6ee2e43f2f5025ade91bacc889a7ac4dccf69acd2b520471fbfa920c25b-merged.mount: Succeeded.
Feb  8 10:38:43 moss-0003 systemd[1]: mgt-docker-overlay2-91c6f6ee2e43f2f5025ade91bacc889a7ac4dccf69acd2b520471fbfa920c25b-merged.mount: Succeeded.
Feb  8 10:38:43 moss-0003 dockerd[19904]: time="2023-02-08T10:38:43.971042533+08:00" level=error msg="68d7dd923743e83c1bc28f7381f2006dfb4aa78f4f694b17ea2bd4acdae024b0 cleanup: failed to delete container from containerd, try to delete directly: no such container"
Feb  8 10:38:43 moss-0003 docker-runc[24168]: time="2023-02-08T10:38:43+08:00" level=error msg="container \"68d7dd923743e83c1bc28f7381f2006dfb4aa78f4f694b17ea2bd4acdae024b0\" does not exist\n"
Feb  8 10:38:43 moss-0003 dockerd[19904]: time="2023-02-08T10:38:43.977202096+08:00" level=error msg="68d7dd923743e83c1bc28f7381f2006dfb4aa78f4f694b17ea2bd4acdae024b0 cleanup: failed to directly delete container"
Feb  8 10:38:43 moss-0003 dockerd[19904]: time="2023-02-08T10:38:43.977239614+08:00" level=error msg="Handler for POST /v1.39/containers/68d7dd923743e83c1bc28f7381f2006dfb4aa78f4f694b17ea2bd4acdae024b0/start returned error: failed to get network during CreateEndpoint: network 06lgrttbrbagkro7aje6kppyc not found"
Feb  8 10:38:45 moss-0003 systemd[1]: session-20.scope: Succeeded.
Feb  8 10:38:53 moss-0003 sshd[24181]: Accepted password for root from 172.0.42.12 port 51208 ssh2
Feb  8 10:38:55 moss-0003 dockerd[19904]: time="2023-02-08T10:38:55.413439867+08:00" level=warning msg="bulk sync to node 59a0d7a83cbf failed: failed to send a TCP message during bulk sync: write tcp 172.30.0.12:53674->172.30.0.10:7946: write: connection reset by peer"
Feb  8 10:38:55 moss-0003 dockerd[19904]: time="2023-02-08T10:38:55.413673428+08:00" level=warning msg="bulk sync to node 70561e9142e7 failed: failed to send a TCP message during bulk sync: write tcp 172.30.0.12:47500->172.30.0.11:7946: write: connection reset by peer"
Feb  8 10:38:55 moss-0003 dockerd[19904]: time="2023-02-08T10:38:55.413693648+08:00" level=error msg="periodic bulk sync failure for network bi2kleuhr7zue93ml6twngn8q: bulk sync to node 70561e9142e7 failed: failed to send a TCP message during bulk sync: write tcp 172.30.0.12:47500->172.30.0.11:7946: write: connection reset by peer"
Feb  8 10:38:55 moss-0003 dockerd[19904]: time="2023-02-08T10:38:55.613091369+08:00" level=info msg="NetworkDB stats moss-0003(17dc767b4e1d) - netID:bi2kleuhr7zue93ml6twngn8q leaving:false netPeers:3 entries:6 Queue qLen:0 netMsg/s:0"
Feb  8 10:38:55 moss-0003 dockerd[19904]: time="2023-02-08T10:38:55.613120927+08:00" level=info msg="NetworkDB stats moss-0003(17dc767b4e1d) - netID:06lgrttbrbagkro7aje6kppyc leaving:true netPeers:2 entries:6 Queue qLen:0 netMsg/s:0"
Feb  8 10:39:24 moss-0003 dockerd[19904]: time="2023-02-08T10:39:24.690656809+08:00" level=error msg="Error in responding to bulk sync from node 172.30.0.11: failed to send a TCP message during bulk sync: write tcp 172.30.0.12:47534->172.30.0.11:7946: write: connection reset by peer"
wiserK commented 1 year ago

And docker often reports errors like:

moss-0003 dockerd[19904]: time="2023-02-08T17:00:55.414069338+08:00" level=warning msg="bulk sync to node 70561e9142e7 failed: failed to send a TCP message during bulk sync: write tcp 172.30.0.12:44530->172.30.0.11:7946: write: connection reset by peer"

Is this the problem of openEuler docker or the problem of system configuration?

thaJeztah commented 1 year ago

Docker 18.09 reached end of life 5 years ago, and is no longer maintained. In addition, OpenEuler is not a distro that is officially supported by the Docker Engine packages provided by Docker on download.docker.com.

The package you're running looks to be a build of the Docker Engine that is built and maintained by OpenEuler, and may have (likely has, based on the version output) patches that are not in upstream;

 Version:           18.09.0
 EulerVersion:      18.09.0.243

I would recommend opening a support ticket in the OpenEuler issue tracker for the initial triage.

I'm closing this ticket here because of the above, but feel free to continue the conversation.