search

containernetworking / cni

Container Network Interface - networking for Linux containers
https://cni.dev
Apache License 2.0
5.49k stars 1.08k forks source link

DHCP daemon error #398

Closed shouhong closed 4 years ago

shouhong commented 7 years ago

I followed the instructions on https://github.com/containernetworking/cni/blob/master/Documentation/dhcp.md to use DHCP ipam plugin and got error messages like below. Do you have any suggestion how to solve this problem? Thanks!

root@ubuntu:/opt/cni/bin# ./dhcp daemon 2017/03/14 20:05:39 fb179323bda61f220021764c8cb1da4c97c484f1ead73039bd28b377216eab31/mynet: acquiring lease 2017/03/14 20:05:39 Link "eth0" down. Attempting to set up 2017/03/14 20:05:39 network is down 2017/03/14 20:05:39 d6db6378dfa62518a3106bd897bfcd3b690d380e8a04ff11e972b384c52c0046/mynet: acquiring lease 2017/03/14 20:05:39 cae1189ec94dfc3687ad9852bd6f1961f15523a7ab173c16cc2ead8a303c6e53/mynet: acquiring lease 2017/03/14 20:05:39 Link "eth0" down. Attempting to set up 2017/03/14 20:05:39 network is down

phucvinh52 commented 7 years ago

i have a same issue. I have used macvlan with dhcp. Please help me. Thanks

squeed commented 7 years ago

The 'network down' messages are normal and to be expected; they're displayed even for successful operations. Can you share your CNI configuration?

Reminder, macvlan generally does not work on wifi interfaces.

phucvinh52 commented 7 years ago

This is my config network cni. Before I run dhcp daemon. NIC eth0 in host

2: eth0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether fa:16:3e:2a:1d:db brd ff:ff:ff:ff:ff:ff
    inet 10.10.14.208/24 brd 10.10.14.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet 10.10.14.209/24 scope global secondary eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fe2a:1ddb/64 scope link 
       valid_lft forever preferred_lft forever

/etc/cni/net.d/10-containernet.conf

{
    "name": "containernet",
    "type": "macvlan",
    "master": "eth0",
    "ipam": {
        "type": "dhcp"
    }
}

log dhcp

2017/03/17 10:03:09 2e5a379b106f655c9fd577b52def762c4bbf9458e30f546e44ea18b04e01ac26/containernet: acquiring lease
2017/03/17 10:03:09 Link "eth0" down. Attempting to set up
2017/03/17 10:03:09 network is down
2017/03/17 10:03:18 resource temporarily unavailable
2017/03/17 10:03:31 resource temporarily unavailable
2017/03/17 10:03:48 5a952daeff853c55a5dfeb0906c3fe2079680333150e7b46c4b423d909f4cf3e/containernet: acquiring lease
2017/03/17 10:03:48 Link "eth0" down. Attempting to set up
2017/03/17 10:03:48 network is down
2017/03/17 10:03:56 resource temporarily unavailable
2017/03/17 10:04:09 resource temporarily unavailable

and log kuberlet

E0317 10:03:47.168306   20250 pod_workers.go:184] Error syncing pod f9cf41f2-0a2d-11e7-ad4e-fa163eac654e, skipping: failed to "SetupNetwork" for "abc1-3224606261-l3kw4_default" with SetupNetworkError: "Failed to setup network for pod \"abc1-3224606261-l3kw4_default(f9cf41f2-0a2d-11e7-ad4e-fa163eac654e)\" using network plugins \"cni\": error calling DHCP.Allocate: no more tries; Skipping pod"
E0317 10:03:49.024953   20250 docker_manager.go:373] NetworkPlugin cni failed on the status hook for pod 'abc1-3224606261-l3kw4' - Unexpected address output  
E0317 10:04:26.669450   20250 cni.go:255] Error adding network: error calling DHCP.Allocate: no more tries
E0317 10:04:26.669504   20250 cni.go:209] Error while adding to cni network: error calling DHCP.Allocate: no more tries
E0317 10:04:26.669593   20250 docker_manager.go:2201] Failed to setup network for pod "abc1-3224606261-l3kw4_default(f9cf41f2-0a2d-11e7-ad4e-fa163eac654e)" using network plugins "cni": error calling DHCP.Allocate: no more tries; Skipping pod
E0317 10:04:26.766632   20250 pod_workers.go:184] Error syncing pod f9cf41f2-0a2d-11e7-ad4e-fa163eac654e, skipping: failed to "SetupNetwork" for "abc1-3224606261-l3kw4_default" with SetupNetworkError: "Failed to setup network for pod \"abc1-3224606261-l3kw4_default(f9cf41f2-0a2d-11e7-ad4e-fa163eac654e)\" using network plugins \"cni\": error calling DHCP.Allocate: no more tries; Skipping pod"
E0317 10:04:28.137335   20250 docker_manager.go:373] NetworkPlugin cni failed on the status hook for pod 'abc1-3224606261-l3kw4' - Unexpected address output  
E0317 10:05:05.597413   20250 cni.go:255] Error adding network: error calling DHCP.Allocate: no more tries
E0317 10:05:05.597458   20250 cni.go:209] Error while adding to cni network: error calling DHCP.Allocate: no more tries
E0317 10:05:05.597518   20250 docker_manager.go:2201] Failed to setup network for pod "abc1-3224606261-l3kw4_default(f9cf41f2-0a2d-11e7-ad4e-fa163eac654e)" using network plugins "cni": error calling DHCP.Allocate: no more tries; Skipping pod

Thanks

shouhong commented 7 years ago

My CNI configuration and error messages are also like above.

squeed commented 7 years ago

At a first guess, it seems like either your server is out of leases. What happens if you try to bring the network up with cnitool?

I will be offline for the next week, but hopefully someone else has another idea.

rosenhouse commented 7 years ago

Any progress on this issue or should it be closed?

rkamudhan commented 6 years ago

@squeed Hi, I am also having the same issue any solution for this bug ?

squeed commented 6 years ago

It's not a bug, per se. It means the plugin was unable to reach the DHCP server.

The error message should be clearer.

rkamudhan commented 6 years ago

@squeed Thanks, I will double check my DHCP server.

lubouski commented 5 years ago

@squeed any thoughts on my similar issue ? :) https://github.com/intel/multus-cni/issues/291

weikinhuang commented 4 years ago

Cross post from https://github.com/intel/multus-cni/issues/291#issuecomment-574451016

I'm getting a similar issue running dhcp daemon:

I'm running coreos stable, with pfsense as my router.

sudo ./dhcp daemon
2020/01/15 01:06:24 1943bec66d91cebd9734e45678883c2d2329b32648439433089a24927935442f/macvlan-conf/net1: acquiring lease
2020/01/15 01:06:24 Link "net1" down. Attempting to set up
2020/01/15 01:06:24 network is down
2020/01/15 01:06:32 no DHCP packet received within 5s
2020/01/15 01:06:45 no DHCP packet received within 5s
2020/01/15 01:07:08 2a097ab3ee84e359ac65e72c8c4500c72581b4c9b7a71c300059e5500692a822/macvlan-conf/net1: acquiring lease
2020/01/15 01:07:08 Link "net1" down. Attempting to set up
2020/01/15 01:07:08 network is down
2020/01/15 01:07:17 no DHCP packet received within 5s
2020/01/15 01:07:30 no DHCP packet received within 5s
2020/01/15 01:07:52 0a771ae626aba1a80c0f0b8a93b368cf11143b2681e0efc640599daf7ca3a51a/macvlan-conf/net1: acquiring lease
2020/01/15 01:07:52 Link "net1" down. Attempting to set up
2020/01/15 01:07:52 network is down
2020/01/15 01:08:01 no DHCP packet received within 5s
2020/01/15 01:08:14 no DHCP packet received within 5s

I see in my pfsense's dhcpd logs that it's offering addresses, but it doesn't seem like the dhcp cni daemon is picking it up.

Jan 14 20:06:27 | dhcpd |   | DHCPDISCOVER from 00:0c:29:73:9c:d8 via em1
Jan 14 20:06:28 | dhcpd |   | DHCPOFFER on 192.168.1.58 to 00:0c:29:73:9c:d8 via em1
Jan 14 20:06:40 | dhcpd |   | DHCPDISCOVER from 00:0c:29:73:9c:d8 via em1
Jan 14 20:06:40 | dhcpd |   | DHCPOFFER on 192.168.1.58 to 00:0c:29:73:9c:d8 via em1
Jan 14 20:07:12 | dhcpd |   | DHCPDISCOVER from 00:0c:29:73:9c:d8 via em1
Jan 14 20:07:13 | dhcpd |   | DHCPOFFER on 192.168.1.59 to 00:0c:29:73:9c:d8 via em1
Jan 14 20:07:25 | dhcpd |   | DHCPDISCOVER from 00:0c:29:73:9c:d8 via em1
Jan 14 20:07:25 | dhcpd |   | DHCPOFFER on 192.168.1.59 to 00:0c:29:73:9c:d8 via em1
Jan 14 20:07:56 | dhcpd |   | DHCPDISCOVER from 00:0c:29:73:9c:d8 via em1
Jan 14 20:07:57 | dhcpd |   | DHCPOFFER on 192.168.1.62 to 00:0c:29:73:9c:d8 via em1
Jan 14 20:08:09 | dhcpd |   | DHCPDISCOVER from 00:0c:29:73:9c:d8 via em1
Jan 14 20:08:09 | dhcpd |   | DHCPOFFER on 192.168.1.62 to 00:0c:29:73:9c:d8 via em1
apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
  name: ipvlan-conf
spec:
  config: |
    {
      "cniVersion": "0.3.1",
      "type": "ipvlan",
      "master": "ens190",
      "ipam": {
        "type": "dhcp",
        "routes": [
          { "dst": "192.168.0.0/16"}
        ]
      }
    }
apiVersion: v1
kind: Pod
metadata:
  name: samplepod
  annotations:
    k8s.v1.cni.cncf.io/networks: ipvlan-conf
spec:
  containers:
  - name: samplepod
    image: nginx:latest
bboreham commented 4 years ago

As far as we know the daemon functions; it passes CI tests frequently. The CNI project has no resources to troubleshoot the specifics of your installation, so I'm going to close this issue.

elgamal2020 commented 4 years ago

Still existing problem

nicolacdnll commented 3 years ago

Same here.

andrewm659 commented 3 years ago

This is happening on CentOS 8 Stream asl well.

nicolacdnll commented 3 years ago

We have reprovisioned the node, same OS, same k8s, and CNI version and it's now working. No idea what was going on.

What I know was that the DHCP server was working correctly, and the DHCP lease packets were reaching the node (I could see them with tcpdump) but the DHCP plugin was failing to use/see them.

dhamodaran-pandiyan commented 2 years ago

Any update on this issue? Anyone please share the working configurations for

  1. NetworkAttachmentDefinition
  2. Pod YAML
  3. How DHCP CNI should run?
AJMansfield commented 1 year ago

So, it's possible to resolve this issue temporarily by manually running the /opt/cni/bin/dhcp plugin binary. But that's not an acceptable solution for actual deployment. How should the cluster configuration be modified so that this gets run automatically?