--static-ip not respected when run with --network with docker driver

lbergesio commented 1 month ago

What Happened?

I have a docker network previously created:

$ docker network inspect minikube_net
[
    {
        "Name": "minikube_net",
        "Id": "92ce6fe7b171ecdfbdec5274d619f0d652d391d2bb86460d4a5ef2d1e51957ee",
        "Created": "2024-07-17T07:46:01.295275401-07:00",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": {},
            "Config": [
                {
                    "Subnet": "192.168.100.0/25"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {},
        "Options": {
            "com.docker.network.bridge.name": "br-minikube_net"
        },
        "Labels": {}
    }
]

If I run the following command the ip is not repected:

$ minikube start -p leotest --static-ip 192.168.100.20 --network minikube_net
😄  [leotest] minikube v1.33.1 on Ubuntu 22.04 (kvm/amd64)
✨  Using the docker driver based on existing profile
❗  You cannot change the static IP of an existing minikube cluster. Please first delete the cluster.
👍  Starting "leotest" primary control-plane node in "leotest" cluster
🚜  Pulling base image v0.0.44 ...
🏃  Updating the running docker "leotest" container ...
🐳  Preparing Kubernetes v1.30.0 on Docker 26.1.1 ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: storage-provisioner, default-storageclass
🏄  Done! kubectl is now configured to use "leotest" cluster and "default" namespace by default
$ minikube ip -p leotest
172.17.0.6

But if I do not specify the network then, a new one is created and the ip is respected:

$ minikube start -p leotest --static-ip 192.168.200.20
😄  [leotest] minikube v1.33.1 on Ubuntu 22.04 (kvm/amd64)
✨  Automatically selected the docker driver. Other choices: none, ssh
📌  Using Docker driver with root privileges
👍  Starting "leotest" primary control-plane node in "leotest" cluster
🚜  Pulling base image v0.0.44 ...
🔥  Creating docker container (CPUs=2, Memory=3900MB) ...
🐳  Preparing Kubernetes v1.30.0 on Docker 26.1.1 ...
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔗  Configuring bridge CNI (Container Networking Interface) ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: storage-provisioner, default-storageclass
🏄  Done! kubectl is now configured to use "leotest" cluster and "default" namespace by default
$ minikube ip -p leotest
192.168.200.20

Attach the log file

log.txt

Operating System

Ubuntu

Driver

Docker

lbergesio commented 1 month ago

I have seen that the problem is that the docker network was not created with --gatewayso, dockerNetworkInspect returns info.gateway = nil and then the StaticIP is not respected in func (d *Driver) Create() error because to add it to params the check is:

} else if gateway != nil && staticIP != "" {
        params.Network = networkName        
        params.IP = staticIP                
}

The output of the network inspect is

{"Name": "leotest","Driver": "bridge","Subnet": "192.168.100.0/24","Gateway": "","MTU": 0, "ContainerIPs": []}

imho the code should check for the Subnet, check if the staticIp is free and use it.

saidjawad commented 1 month ago

I'm facing similar issue:

Minikube version: v1.33.1 OS: Ubuntu 22.04, AMD64 Docker version: 26.1.4

medyagh commented 1 month ago

@ComradeProgrammer would you please take a look

gnanieswar195 commented 3 weeks ago

It works for me, with below configuration

Now, i am changing and re applying the minikube with another static ip "192.168.100.25" and this also works.

Let me know if you have any questions.

lbergesio commented 3 weeks ago

It works for me, with below configuration

Now, i am changing and re applying the minikube with another static ip "192.168.100.25" and this also works.

Let me know if you have any questions.

It works if you specify the gateway, please check https://github.com/kubernetes/minikube/issues/19284#issuecomment-2236013065

deenadayalans commented 2 weeks ago

/assign

xcarolan commented 1 week ago

Why would you use a static ip without a gateway? i get it for docker, but in Kubernetes you need the gateway for the nodes to communicate in a cluster. What would you expect to happen if you try and add nodes?

lbergesio commented 1 week ago

Why would you use a static ip without a gateway? i get it for docker, but in Kubernetes you need the gateway for the nodes to communicate in a cluster. What would you expect to happen if you try and add nodes?

I'm not sure I get your question, but why not? Afaik here you are configuring the docker network. The k8s nodes can use the static IP as default gw, but that's a different GW. In any case if I'm specifying the static IP and the network and that is not being respected I would expect the command to fail or at least warn about it

xcarolan commented 1 week ago

For my education - if you create a minikube control plane with a static IP, for example, I don't see how you can add any other nodes (how will the other nodes be allocated IP addresses?). I think you are correct about the error message,, "fixing it," by removing the gateway check - I suspect will cause other issues (Ingress etc.)

lbergesio commented 1 week ago

Again, not sure I get your point. But it is true that with current code you can not start a cluster with static-ip and >1 nodes, it fails with:

leo:~$ minikube start -p leo --network leo --static-ip 192.168.111.100 -n 2
....
👍  Starting worker node leo-m02 in cluster leo
🚜  Pulling base image ...
🔥  Creating docker container (CPUs=2, Memory=3950MB) ...

❌  Exiting due to GUEST_START: failed to start node: adding node: Failed to start host: can't create with that IP, address already in use

But that imho is a bug in the code, when you do not run with a static ip it assignes proper ips to the nodes in the network range:

leo:~$ minikube start -p leo --network leo -n 2
....

👍  Starting worker node leo-m02 in cluster leo
🚜  Pulling base image ...
🔥  Creating docker container (CPUs=2, Memory=3950MB) ...
🌐  Found network options:
    ▪ NO_PROXY=192.168.111.2
🐳  Preparing Kubernetes v1.28.3 on Docker 24.0.7 ...
    ▪ env NO_PROXY=192.168.111.2
🔎  Verifying Kubernetes components...
🏄  Done! kubectl is now configured to use "leo" cluster and "default" namespace by default

leo:~$ minikube ip -p leo -n leo
192.168.111.2
leo:~$ minikube ip -p leo -n leo-m02
192.168.111.3

So, it depends how you want to treat --static-ip since that is used as the first node ip you can say that you can't use --static-ip with -n >1 or you can look for the next free ip in the network.

Regarding routing, the default route is

default via 192.168.111.1 dev eth0

But the docker network is a bridge network, you do not need an ip there, you can do:

leo:~$ minikube ssh -p leo
docker@leo:~$ ip route
default via 192.168.111.1 dev eth0 
10.244.0.2 dev veth1990a1d1 scope host 
10.244.1.0/24 via 192.168.111.3 dev eth0 
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown 
192.168.111.0/24 dev eth0 proto kernel scope link src 192.168.111.2 
docker@leo:~$ sudo ip route del default via 192.168.111.1 dev eth0 
docker@leo:~$ ip route
10.244.0.2 dev veth1990a1d1 scope host 
10.244.1.0/24 via 192.168.111.3 dev eth0 
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown 
192.168.111.0/24 dev eth0 proto kernel scope link src 192.168.111.2 
docker@leo:~$ sudo ip route add default dev eth0 
docker@leo:~$ ip route
default dev eth0 scope link 
10.244.0.2 dev veth1990a1d1 scope host 
10.244.1.0/24 via 192.168.111.3 dev eth0 
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown 
192.168.111.0/24 dev eth0 proto kernel scope link src 192.168.111.2

All this was creating the docker network like this (no gateway):

docker network create --driver bridge --subnet=192.168.111.0/24 --opt com.docker.network.bridge.name=leo_br leo --gateway 192.168.111.1

leo:~$ ifconfig leo_br
leo_br: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 192.168.111.1  netmask 255.255.255.0  broadcast 192.168.111.255
        ether 02:42:46:aa:c2:dd  txqueuelen 0  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

But what I said about the routing is not needed anyway, if you do not specify the gateway when creating the docker network, still docker assigns an ip to the bridge interface, so minikube could retrieve the gateway ip from the system instead from docker network inspect and it will find it:

leo:~$ docker network create --driver bridge --subnet=192.168.111.0/24 --opt com.docker.network.bridge.name=leo_br leo
83ac3fdb2b5a2c4272cec41ef72758711bcbd7d23497591944f060a3481662a5
leo:~$ ifconfig leo_br
leo_br: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 192.168.111.1  netmask 255.255.255.0  broadcast 192.168.111.255
        ether 02:42:46:aa:c2:dd  txqueuelen 0  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

xcarolan commented 4 days ago

Thanks for taking the time to respond with all those details. - very much appreciated . I'm wondering what the initial intention was in adding the static ip option to Minikube in the first place. If you do as you suggest, retrieving IP addresses for additional nodes from the docker gateway - wouldn't that be a security issue for the cluster - as anything on the docker network (external to the Minikube cluster that is ) can access directly the nodes.?

lbergesio commented 4 days ago

I think you can already today even not using static-ip. Anyway, you can limit that with the appropriated up tables rules, don't know if any are added today.

kubernetes / minikube