search

KatharaFramework / NetworkPlugin

Docker Network Plugin used by Kathará to create pure L2 networks.
https://www.kathara.org/
Apache License 2.0
17 stars 7 forks source link

iptables rule is not created #8

Closed nsychev closed 2 years ago

nsychev commented 2 years ago

Hi!

I'm trying to launch docker-compose network with your plugin. But hosts can't interact with each other.

My host is Ubuntu 20.04, Docker is 20.10.14, docker-compose is 1.29.2. Compose file:

version: "3.6"
services:
  relay:
    image: "debian:11-slim"
    cap_add:
      - NET_ADMIN
    networks:
      - internal
  stub:
    image: "debian:11-slim"
    networks:
      - internal
    cap_add:
      - NET_ADMIN
networks:
  internal:
    driver: kathara/katharanp:latest
    ipam:
      driver: "null"

Then I'm running bash in each container and executing ip addr add 10.8.123.1/24 dev eth0 (and .2 for other container). Then I'm doing ping 10.8.123.1 from the second one.

Expected behaviour: successful ping.

Actual behaviour:

Packets are dropping somewhere. Containers receive each other mac addresses (they're shown in arp -a).

Latter I launch iptables -vL on host and found that there is no accept rule from kt-... interface to itself. After adding it manually (iptables -A FORWARD -i kt-... -o kt-... -j ACCEPT), it worked.

I don't know how to collect any debug information on plugin, but I'd be glad to provide any useful information.

Skazza94 commented 2 years ago

Hi @nsychev, the problem is due to the newer iptables/nftables version installed on Ubuntu 20, that requires a lock file. During the plugin installation, you should mount xtables.lock file of the host into the network plugin.

Kathará automatically configures katharanp following this behavior (see here).

However, you can manually achieve the same configuration by installing katharanp with this command:

docker plugin rm -f kathara/katharanp:latest
docker plugin install kathara/katharanp:amd64 xtables_lock.source="/var/run/xtables.lock"

Also, note that the latest tag is used for retrocompatibilty, now you should use the architecture as plugin tag.

If you have any other problem, please do not hesitate to contact us. Mariano.

nsychev commented 2 years ago

Oh, it works. Thanks a lot!

What about writing about it in readme? I really stuck on it like two weeks ago and at first thought Linux or Docker just don't allow to build configuration I want :)

Skazza94 commented 2 years ago

Hi @nsychev, I'll close the issue and add a note in the readme :smile:

nsychev commented 2 years ago

Hi @Skazza94!

After upgrading to 22.04, my Docker daemon crashed and didn't start :/ For some reason they removed xtables.lock (while still using nftables as backend), and so plugin crashed with “No such file or directory” error. I couldn't find a way to disable plugin without a need to start Docker daemon beforehand.

journalctl -xeu docker.service Aug 17 16:57:30 hostname dockerd[698118]: time="2022-08-17T16:57:30.264672136+03:00" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc Aug 17 16:57:30 hostname dockerd[698118]: time="2022-08-17T16:57:30.389281198+03:00" level=error msg="failed to enable plugin" error="failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting \"/var/run/xtables.lock\" to rootfs at \"/var/run/xtables.lock\": stat /var/run/xtables.lock: no such file or directory: unknown" id= Aug 17 16:57:30 hostname dockerd[698118]: time="2022-08-17T16:57:30.445305381+03:00" level=info msg="[graphdriver] using prior storage driver: overlay2" Aug 17 16:57:30 hostname dockerd[698118]: time="2022-08-17T16:57:30.512270414+03:00" level=info msg="Loading containers: start." Aug 17 16:57:30 hostname dockerd[698118]: panic: runtime error: invalid memory address or nil pointer dereference Aug 17 16:57:30 hostname dockerd[698118]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x560d0b610f2d] Aug 17 16:57:30 hostname dockerd[698118]: goroutine 1 [running]: Aug 17 16:57:30 hostname dockerd[698118]: github.com/docker/docker/pkg/plugins.(*Client).callWithRetry(0x0, {0xc00005b200, 0x0}, {0x560d0d2b36a0, 0xc000b9a210}, 0x1, {0x0, 0x0, 0xc00139e840}) Aug 17 16:57:30 hostname dockerd[698118]: /go/src/github.com/docker/docker/pkg/plugins/client.go:160 +0x1cd Aug 17 16:57:30 hostname dockerd[698118]: github.com/docker/docker/pkg/plugins.(*Client).CallWithOptions(0x560d0c9b3890, {0xc00005b200, 0x1d}, {0x0, 0x0}, {0x560d0cf0c8e0, 0xc000b9a1e0}, {0x0, 0x0, 0x0}) Aug 17 16:57:30 hostname dockerd[698118]: /go/src/github.com/docker/docker/pkg/plugins/client.go:113 +0x185 Aug 17 16:57:30 hostname dockerd[698118]: github.com/docker/docker/pkg/plugins.(*Client).Call(...) Aug 17 16:57:30 hostname dockerd[698118]: /go/src/github.com/docker/docker/pkg/plugins/client.go:102 Aug 17 16:57:30 hostname dockerd[698118]: github.com/docker/docker/vendor/github.com/docker/libnetwork/drivers/remote.(*driver).call(0xc0007bf158, {0x560d0c9b3890, 0xc000180800}, {0x0, 0x0}, {0x560d0d2b5320, 0xc000b9a1e0}) Aug 17 16:57:30 hostname dockerd[698118]: /go/src/github.com/docker/docker/vendor/github.com/docker/libnetwork/drivers/remote/driver.go:127 +0xbc Aug 17 16:57:30 hostname dockerd[698118]: github.com/docker/docker/vendor/github.com/docker/libnetwork/drivers/remote.(*driver).getCapabilities(0xc00139e9e8) Aug 17 16:57:30 hostname dockerd[698118]: /go/src/github.com/docker/docker/vendor/github.com/docker/libnetwork/drivers/remote/driver.go:90 +0x52 Aug 17 16:57:30 hostname dockerd[698118]: github.com/docker/docker/vendor/github.com/docker/libnetwork/drivers/remote.Init.func1({0xc0000586c0, 0x17}, 0x0) Aug 17 16:57:30 hostname dockerd[698118]: /go/src/github.com/docker/docker/vendor/github.com/docker/libnetwork/drivers/remote/driver.go:37 +0xb6 Aug 17 16:57:30 hostname dockerd[698118]: github.com/docker/docker/vendor/github.com/docker/libnetwork/drivers/remote.Init({0x560d0d2c48e8, 0xc000f80480}, 0xc000e97e60) Aug 17 16:57:30 hostname dockerd[698118]: /go/src/github.com/docker/docker/vendor/github.com/docker/libnetwork/drivers/remote/driver.go:57 +0x15e Aug 17 16:57:30 hostname dockerd[698118]: github.com/docker/docker/vendor/github.com/docker/libnetwork/drvregistry.(*DrvRegistry).AddDriver(...) Aug 17 16:57:30 hostname dockerd[698118]: /go/src/github.com/docker/docker/vendor/github.com/docker/libnetwork/drvregistry/drvregistry.go:72 Aug 17 16:57:30 hostname dockerd[698118]: github.com/docker/docker/vendor/github.com/docker/libnetwork.New({0xc0000ee880, 0xa, 0x10}) Aug 17 16:57:30 hostname dockerd[698118]: /go/src/github.com/docker/docker/vendor/github.com/docker/libnetwork/controller.go:221 +0x61c Aug 17 16:57:30 hostname dockerd[698118]: github.com/docker/docker/daemon.(*Daemon).initNetworkController(0xc00000c1e0, 0xc000474b00, 0xc00054acc0) Aug 17 16:57:30 hostname dockerd[698118]: /go/src/github.com/docker/docker/daemon/daemon_unix.go:856 +0x65 Aug 17 16:57:30 hostname dockerd[698118]: github.com/docker/docker/daemon.(*Daemon).restore(0xc00000c1e0) Aug 17 16:57:30 hostname dockerd[698118]: /go/src/github.com/docker/docker/daemon/daemon.go:490 +0x532 Aug 17 16:57:30 hostname dockerd[698118]: github.com/docker/docker/daemon.NewDaemon({0x560d0d305210, 0xc0001e2840}, 0xc000474b00, 0xc0008cf110) Aug 17 16:57:30 hostname dockerd[698118]: /go/src/github.com/docker/docker/daemon/daemon.go:1150 +0x3037 Aug 17 16:57:30 hostname dockerd[698118]: main.(*DaemonCli).start(0xc000591230, 0xc0008e6420) Aug 17 16:57:30 hostname dockerd[698118]: /go/src/github.com/docker/docker/cmd/dockerd/daemon.go:195 +0xa36 Aug 17 16:57:30 hostname dockerd[698118]: main.runDaemon(...) Aug 17 16:57:30 hostname dockerd[698118]: /go/src/github.com/docker/docker/cmd/dockerd/docker_unix.go:14 Aug 17 16:57:30 hostname dockerd[698118]: main.newDaemonCommand.func1(0xc000478000, {0xc0005911d0, 0x3, 0x3}) Aug 17 16:57:30 hostname dockerd[698118]: /go/src/github.com/docker/docker/cmd/dockerd/docker.go:34 +0x5c Aug 17 16:57:30 hostname dockerd[698118]: github.com/docker/docker/vendor/github.com/spf13/cobra.(*Command).execute(0xc000478000, {0xc00004e0d0, 0x3, 0x3}) Aug 17 16:57:30 hostname dockerd[698118]: /go/src/github.com/docker/docker/vendor/github.com/spf13/cobra/command.go:850 +0x60e Aug 17 16:57:30 hostname dockerd[698118]: github.com/docker/docker/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0xc000478000) Aug 17 16:57:30 hostname dockerd[698118]: /go/src/github.com/docker/docker/vendor/github.com/spf13/cobra/command.go:958 +0x3ad Aug 17 16:57:30 hostname dockerd[698118]: github.com/docker/docker/vendor/github.com/spf13/cobra.(*Command).Execute(...) Aug 17 16:57:30 hostname dockerd[698118]: /go/src/github.com/docker/docker/vendor/github.com/spf13/cobra/command.go:895 Aug 17 16:57:30 hostname dockerd[698118]: main.main() Aug 17 16:57:30 hostname dockerd[698118]: /go/src/github.com/docker/docker/cmd/dockerd/docker.go:97 +0x15d Aug 17 16:57:30 hostname systemd[1]: docker.service: Main process exited, code=exited, status=2/INVALIDARGUMENT

So, I had to do the following:

  1. Create dummy file: touch /var/run/xtables.lock.
  2. Start Docker daemon and immediately disable and remove plugin: 
    docker plugin disable kathara/katharanp:amd64
docker plugin remove kathara/katharanp:amd64
  3. Then install it, but without mount: 
    docker plugin install kathara/katharanp:amd64

After that, it worked.

(just leaving a note here in case someone will receive the same error after upgrading)

barthr commented 1 year ago

@nsychev Kathara is working for me, also with your solution (starting terminals), but they don't have any network at all. When pinging a host it keeps getting 100% packet loss.

iptables -vL does show the accept rule:


  175 14700 ACCEPT     all  --  kt-611fe0fe933e kt-611fe0fe933e  anywhere             anywhere            
    0     0 ACCEPT     all  --  kt-46116dea0b17 kt-46116dea0b17  anywhere             anywhere            
    5   420 ACCEPT     all  --  kt-bc62d9ab61fa kt-bc62d9ab61fa  anywhere             anywhere ```