search

cilium / cilium

eBPF-based Networking, Security, and Observability
https://cilium.io
Apache License 2.0
20.14k stars 2.95k forks source link

Conflicting tunnel peer for prefix warning log in latest cilium with native routing mode in AKS #26817

Closed tamilmani1989 closed 1 year ago

tamilmani1989 commented 1 year ago

Is there an existing issue for this?

What happened?

In AKS clusters, when I deployed latest cilium in native routing mode with delegated ipam, this warning message is logged by cilium agent:

level=warning msg="Detected conflicting tunnel peer for prefix. This may cause connectivity issues for this address." cidr=169.254.23.0/32 conflictingResource=node//aks-nodepool1-30053602-vmss000000 conflictingTunnelPeer=10.10.0.4 resource=node//aks-nodepool1-30053602-vmss000001 subsys=ipcache tunnelPeer=10.10.0.5

We found this started to happen after this commit : https://github.com/cilium/cilium/pull/23208 .

Cilium Version

Latest master

Kernel Version

5.15+

Kubernetes Version

1.25

Cilium Config

#### Read-only configurations ####
ARPPingKernelManaged              : false
ARPPingRefreshPeriod              : 30000000000
AddressScopeMax                   : 252
AgentHealthPort                   : 9879
AgentLabels                       : []
AgentNotReadyNodeTaintKey         : node.cilium.io/agent-not-ready
AllocatorListTimeout              : 180000000000
AllowICMPFragNeeded               : true
AllowLocalhost                    : always
AnnotateK8sNode                   : false
AuthMapEntries                    : 524288
AutoCreateCiliumNodeResource      : true
BGPAnnounceLBIP                   : false
BGPAnnouncePodCIDR                : false
BGPConfigPath                     : /var/lib/cilium/bgp/config.yaml
BPFCompileDebug                   : 
BPFMapEventBuffers                : <nil>
BPFMapsDynamicSizeRatio           : 0.0025
BPFRoot                           : /sys/fs/bpf
BPFSocketLBHostnsOnly             : false
BpfDir                            : /var/lib/cilium/bpf
BypassIPAvailabilityUponRestore   : false
CGroupRoot                        : /run/cilium/cgroupv2
CRDWaitTimeout                    : 300000000000
CTMapEntriesGlobalAny             : 129057
CTMapEntriesGlobalTCP             : 258114
CTMapEntriesTimeoutAny            : 60000000000
CTMapEntriesTimeoutFIN            : 10000000000
CTMapEntriesTimeoutSVCAny         : 60000000000
CTMapEntriesTimeoutSVCTCP         : 21600000000000
CTMapEntriesTimeoutSVCTCPGrace    : 60000000000
CTMapEntriesTimeoutSYN            : 60000000000
CTMapEntriesTimeoutTCP            : 21600000000000
CgroupPathMKE                     : 
ClockSource                       : 1
ClusterHealthPort                 : 4240
ClusterID                         : 0
ClusterMeshHealthPort             : 0
ClusterName                       : default
CompilerFlags                     : []
ConfigDir                         : /tmp/cilium/config-map
ConfigFile                        : 
ConntrackGCInterval               : 0
CreationTime                      : 2023-07-12T16:54:04.010686445Z
DNSMaxIPsPerRestoredRule          : 1000
DNSPolicyUnloadOnShutdown         : false
DNSProxyConcurrencyLimit          : 0
DNSProxyConcurrencyProcessingGracePeriod: 0
DNSProxyLockCount                 : 128
DNSProxyLockTimeout               : 500000000
DatapathMode                      : veth
Debug                             : false
DebugVerbose                      : []
DeriveMasqIPAddrFromDevice        : 
Devices                           : [eth0]
DirectRoutingDevice               : eth0
DisableCNPStatusUpdates           : true
DisableCiliumEndpointCRD          : false
DisableEnvoyVersionCheck          : true
DisableIptablesFeederRules        : []
DryMode                           : false
EgressMasqueradeInterfaces        : 
EgressMultiHomeIPRuleCompat       : false
EnableAutoDirectRouting           : false
EnableAutoProtectNodePortRange    : true
EnableBBR                         : false
EnableBGPControlPlane             : false
EnableBPFClockProbe               : true
EnableBPFMasquerade               : false
EnableBPFTProxy                   : false
EnableBandwidthManager            : false
EnableCiliumEndpointSlice         : false
EnableCustomCalls                 : false
EnableEncryptionStrictMode        : false
EnableEndpointHealthChecking      : false
EnableEndpointRoutes              : true
EnableEnvoyConfig                 : false
EnableExternalIPs                 : true
EnableGatewayAPI                  : false
EnableHealthCheckNodePort         : true
EnableHealthChecking              : true
EnableHealthDatapath              : false
EnableHighScaleIPcache            : false
EnableHostFirewall                : false
EnableHostIPRestore               : true
EnableHostLegacyRouting           : true
EnableHostPort                    : true
EnableHubble                      : false
EnableHubbleOpenMetrics           : false
EnableHubbleRecorderAPI           : true
EnableICMPRules                   : true
EnableIPMasqAgent                 : false
EnableIPSec                       : false
EnableIPsecKeyWatcher             : true
EnableIPv4                        : true
EnableIPv4BIGTCP                  : false
EnableIPv4EgressGateway           : false
EnableIPv4FragmentsTracking       : true
EnableIPv4Masquerade              : false
EnableIPv6                        : false
EnableIPv6BIGTCP                  : false
EnableIPv6Masquerade              : false
EnableIPv6NDP                     : false
EnableIdentityMark                : true
EnableIngressController           : false
EnableK8sNetworkPolicy            : true
EnableK8sTerminatingEndpoint      : true
EnableL2Announcements             : false
EnableL2NeighDiscovery            : true
EnableL7Proxy                     : false
EnableLocalNodeRoute              : false
EnableLocalRedirectPolicy         : false
EnableMKE                         : false
EnableNat46X64Gateway             : false
EnableNodePort                    : true
EnablePMTUDiscovery               : false
EnablePolicy                      : default
EnableRecorder                    : false
EnableRemoteNodeIdentity          : true
EnableRuntimeDeviceDetection      : false
EnableSCTP                        : false
EnableSRv6                        : false
EnableSVCSourceRangeCheck         : true
EnableServiceTopology             : false
EnableSessionAffinity             : true
EnableSocketLB                    : true
EnableSocketLBPeer                : true
EnableSocketLBTracing             : true
EnableStaleCiliumEndpointCleanup  : true
EnableTracing                     : false
EnableUnreachableRoutes           : false
EnableVTEP                        : false
EnableWellKnownIdentities         : false
EnableWireguard                   : false
EnableWireguardUserspaceFallback  : false
EnableXDPPrefilter                : false
EnableXTSocketFallback            : true
EncryptInterface                  : []
EncryptNode                       : false
EncryptionStrictModeAllowRemoteNodeIdentities: false
EncryptionStrictModeCIDR          : 
EndpointQueueSize                 : 25
EndpointStatus
EnvoyConfigTimeout                : 120000000000
EnvoyLog                          : 
EnvoyLogPath                      : 
EnvoySecretNamespaces             : []
ExcludeLocalAddresses             : <nil>
ExternalClusterIP                 : false
ExternalEnvoyProxy                : false
FQDNProxyResponseMaxDelay         : 100000000
FQDNRegexCompileLRUSize           : 1024
FQDNRejectResponse                : refused
FixedIdentityMapping
FragmentsMapEntries               : 8192
HTTP403Message                    : 
HTTPIdleTimeout                   : 0
HTTPMaxGRPCTimeout                : 0
HTTPNormalizePath                 : true
HTTPRequestTimeout                : 3600
HTTPRetryCount                    : 3
HTTPRetryTimeout                  : 0
HostV4Addr                        : 
HostV6Addr                        : 
HubbleEventBufferCapacity         : 4095
HubbleEventQueueSize              : 8192
HubbleExportFileCompress          : false
HubbleExportFileMaxBackups        : 5
HubbleExportFileMaxSizeMB         : 10
HubbleExportFilePath              : 
HubbleListenAddress               : 
HubbleMetrics                     : []
HubbleMetricsServer               : 
HubbleMonitorEvents               : []
HubblePreferIpv6                  : false
HubbleRecorderSinkQueueSize       : 1024
HubbleRecorderStoragePath         : /var/run/cilium/pcaps
HubbleRedact                      : []
HubbleSkipUnknownCGroupIDs        : true
HubbleSocketPath                  : /var/run/cilium/hubble.sock
HubbleTLSCertFile                 : 
HubbleTLSClientCAFiles            : []
HubbleTLSDisabled                 : false
HubbleTLSKeyFile                  : 
IPAM                              : delegated-plugin
IPAMCiliumNodeUpdateRate          : 15000000000
IPAMMultiPoolPreAllocation
    default                   : 8
IPAllocationTimeout               : 120000000000
IPMasqAgentConfigPath             : /etc/config/ip-masq-agent
IPSecKeyFile                      : 
IPTablesLockTimeout               : 5000000000
IPTablesRandomFully               : false
IPsecKeyRotationDuration          : 300000000000
IPv4NativeRoutingCIDR             : <nil>
IPv4NodeAddr                      : auto
IPv4PodSubnets                    : []
IPv4Range                         : auto
IPv4ServiceRange                  : auto
IPv6ClusterAllocCIDR              : f00d::/64
IPv6ClusterAllocCIDRBase          : f00d::
IPv6MCastDevice                   : 
IPv6NAT46x64CIDR                  : 64:ff9b::/96
IPv6NAT46x64CIDRBase              : 64:ff9b::
IPv6NativeRoutingCIDR             : <nil>
IPv6NodeAddr                      : auto
IPv6PodSubnets                    : []
IPv6Range                         : auto
IPv6ServiceRange                  : auto
IdentityAllocationMode            : crd
IdentityChangeGracePeriod         : 5000000000
IdentityRestoreGracePeriod        : 600000000000
InstallIptRules                   : true
InstallNoConntrackIptRules        : false
JoinCluster                       : false
K8sEnableK8sEndpointSlice         : true
K8sEnableLeasesFallbackDiscovery  : false
K8sEventHandover                  : false
K8sNamespace                      : kube-system
K8sRequireIPv4PodCIDR             : false
K8sRequireIPv6PodCIDR             : false
K8sServiceCacheSize               : 128
K8sServiceProxyName               : 
K8sSyncTimeout                    : 180000000000
K8sWatcherEndpointSelector        : metadata.name!=kube-scheduler,metadata.name!=kube-controller-manager,metadata.name!=etcd-operator,metadata.name!=gcp-controller-manager
KVStore                           : 
KVStoreOpt
KVstoreConnectivityTimeout        : 120000000000
KVstoreKeepAliveInterval          : 300000000000
KVstoreLeaseTTL                   : 900000000000
KVstoreMaxConsecutiveQuorumErrors : 2
KVstorePeriodicSync               : 300000000000
KeepConfig                        : false
KernelHz                          : 250
KubeProxyReplacement              : strict
KubeProxyReplacementHealthzBindAddr: 0.0.0.0:10256
L2AnnouncerLeaseDuration          : 15000000000
L2AnnouncerRenewDeadline          : 5000000000
L2AnnouncerRetryPeriod            : 2000000000
LBAffinityMapEntries              : 0
LBBackendMapEntries               : 0
LBDevInheritIPAddr                : 
LBMaglevMapEntries                : 0
LBMapEntries                      : 65536
LBRevNatEntries                   : 0
LBServiceMapEntries               : 0
LBSourceRangeMapEntries           : 0
LabelPrefixFile                   : 
Labels                            : []
LibDir                            : /var/lib/cilium
LoadBalancerDSRDispatch           : opt
LoadBalancerDSRL4Xlate            : frontend
LoadBalancerRSSv4
    IP                        : 
    Mask                      : <nil>
LoadBalancerRSSv4CIDR             : 
LoadBalancerRSSv6
    IP                        : 
    Mask                      : <nil>
LoadBalancerRSSv6CIDR             : 
LocalRouterIPv4                   : 169.254.23.0
LocalRouterIPv6                   : 
LogDriver                         : []
LogOpt
LogSystemLoadConfig               : false
Logstash                          : false
LoopbackIPv4                      : 169.254.42.1
MTU                               : 0
MaglevHashSeed                    : JLfvgnHc2kaSUFaI
MaglevTableSize                   : 16381
MaxControllerInterval             : 0
Monitor
    cpus                      : 8
    npages                    : 64
    pagesize                  : 4096
MonitorAggregation                : medium
MonitorAggregationFlags           : 255
MonitorAggregationInterval        : 5000000000
NATMapEntriesGlobal               : 258114
NeighMapEntriesGlobal             : 258114
NodeEncryptionOptOutLabels        : [map[]]
NodeEncryptionOptOutLabelsString  : node-role.kubernetes.io/control-plane
NodePortAcceleration              : disabled
NodePortAlg                       : random
NodePortBindProtection            : true
NodePortMax                       : 32767
NodePortMin                       : 30000
NodePortMode                      : snat
NodePortNat46X64                  : false
PolicyAuditMode                   : false
PolicyMapEntries                  : 16384
PolicyQueueSize                   : 100
PolicyTriggerInterval             : 1000000000
PreAllocateMaps                   : false
PrependIptablesChains             : true
ProcFs                            : /host/proc
PrometheusServeAddr               : 
ProxyConnectTimeout               : 2
ProxyGID                          : 1337
ProxyIdleTimeout                  : 60
ProxyMaxConnectionDuration        : 0
ProxyMaxRequestsPerConnection     : 0
ProxyPrometheusPort               : 0
ResetQueueMapping                 : true
RestoreState                      : true
RouteMetric                       : 0
RoutingMode                       : native
RunDir                            : /var/run/cilium
SRv6EncapMode                     : reduced
SidecarIstioProxyImage            : cilium/istio_proxy
SizeofCTElement                   : 94
SizeofNATElement                  : 94
SizeofNeighElement                : 24
SizeofSockRevElement              : 52
SockRevNatEntries                 : 129057
SocketPath                        : /var/run/cilium/cilium.sock
StateDir                          : /var/run/cilium/state
TCFilterPriority                  : 1
ToFQDNsEnableDNSCompression       : true
ToFQDNsIdleConnectionGracePeriod  : 0
ToFQDNsMaxDeferredConnectionDeletes: 10000
ToFQDNsMaxIPsPerHost              : 50
ToFQDNsMinTTL                     : 3600
ToFQDNsPreCache                   : 
ToFQDNsProxyPort                  : 0
TracePayloadlen                   : 128
Tunnel                            : 
TunnelPort                        : 8472
TunnelProtocol                    : vxlan
UseCiliumInternalIPForIPsec       : false
UseSingleClusterRoute             : false
VLANBPFBypass                     : []
Version                           : false
VtepCIDRs                         : <nil>
VtepCidrMask                      : 
VtepEndpoints                     : <nil>
VtepMACs                          : <nil>
XDPMode                           : disabled
k8s-configuration                 : 
k8s-endpoint                      : 
##### Read-write configurations #####
ConntrackAccounting               : Enabled
ConntrackLocal                    : Disabled
Debug                             : Disabled
DebugLB                           : Disabled
DropNotification                  : Enabled
MonitorAggregationLevel           : Medium
PolicyAuditMode                   : Disabled
PolicyTracing                     : Disabled
PolicyVerdictNotification         : Enabled
SourceIPVerification              : Enabled
TraceNotification                 : Enabled
PolicyEnforcement                 : default

Sysdump

Relevant log output

level=warning msg="Detected conflicting tunnel peer for prefix. This may cause connectivity issues for this address." cidr=169.254.23.0/32 conflictingResource=node//aks-nodepool1-30053602-vmss000000 conflictingTunnelPeer=10.10.0.4 resource=node//aks-nodepool1-30053602-vmss000001 subsys=ipcache tunnelPeer=10.10.0.5


### Anything else?

_No response_

### Code of Conduct

- [X] I agree to follow this project's Code of Conduct
tamilmani1989 commented 1 year ago

cc: @christarazi (author of PR https://github.com/cilium/cilium/pull/23208).

tamilmani1989 commented 1 year ago

May be not related to this PR, sorry. Since this PR adds log message its exposed.. should we consider skip logging this message if cilium agent running in native mode? I can open up a PR if that's the case

christarazi commented 1 year ago

Does it happen consistently or was it just a one-off?

christarazi commented 1 year ago

Is it possible for you to take a sysdump after the issue occurs and then upload it here?

tamilmani1989 commented 1 year ago

@christarazi Yes its consistent. Attached the sysdump

cilium-sysdump-20230718-125921.zip

tamilmani1989 commented 1 year ago

@christarazi did you get a chance to check on this issue?

christarazi commented 1 year ago

So after a brief investigation, I believe Cilium should still be operating correctly. When it discovers the conflict, it will keep the strongest "source" of the information in the datapath. However, you're asking about why this is happening.

The reason this is happening is because the local router IP is the same on both nodes. So when Cilium is notified of both nodes (via CiliumNode updates), it plumbs the IP addresses from each event into the ipcache. Since both nodes have the same router IP (169.254.23.0), Cilium discovers a conflict between two different events (2 nodes, 2 events) when it maps 169.254.23.0 to the node IPs, hence the log msg.

I assume based on the configuration passed to Cilium that configuring the local router IP is intended and intended to be the same on all nodes?

tamilmani1989 commented 1 year ago

Thanks for getting back on this issue.

I assume based on the configuration passed to Cilium that configuring the local router IP is intended and intended to be the same on all nodes?

Yes, that's correct.

christarazi commented 1 year ago

After thinking about it a bit more, I think it's safe to do this: https://github.com/cilium/cilium/pull/27331. Please let me know if this resolves your issue.

tamilmani1989 commented 1 year ago

@christarazi Thanks for taking care of this. I will let you know once I validate with your PR

tamilmani1989 commented 1 year ago

@christarazi validated with #27331 PR and didn't see warning message anymore

tamilmani1989 commented 1 year ago

A follow up question - any adverse impact using same local router ip 169.254.23.0 on different nodes in native routing mode?

christarazi commented 1 year ago

@tamilmani1989 No not that I'm aware of.