Failed to load and apply embedded addon (cni-canal?) with kubeone 1.4.4

kubermatic / kubeone

Kubermatic KubeOne automate cluster operations on all your cloud, on-prem, edge, and IoT environments.

Apache License 2.0

1.38k stars 233 forks source link

$ kubeone apply --manifest kubeone.yaml -t tf.json -y time="08:50:37 UTC" level=info msg="Determine hostname..." time="08:50:38 UTC" level=info msg="Determine operating system..." time="08:50:38 UTC" level=info msg="Running host probes..." time="08:50:40 UTC" level=info msg="Electing cluster leader..." time="08:50:40 UTC" level=info msg="Elected leader \"staging-control-plane-1\"..." time="08:50:41 UTC" level=info msg="Building Kubernetes clientset..." time="08:50:41 UTC" level=info msg="Running cluster probes..." time="08:50:41 UTC" level=info msg="Patching static pods..." time="08:50:41 UTC" level=info msg="Patching static pods..." The following actions will be taken: Run with --verbose flag for more information. ~ ensure embedded addons ~ ensure CNI ~ ensure custom addons ~ ensure credential ~ ensure external CCM time="08:50:41 UTC" level=info msg="Patching static pods..." time="08:50:42 UTC" level=info msg="Downloading kubeconfig..." time="08:50:42 UTC" level=info msg="Downloading PKI..." time="08:50:42 UTC" level=info msg="Creating local backup..." node=xxx.xxx.xxx.xxx time="08:50:42 UTC" level=info msg="Activating additional features..." time="08:50:42 UTC" level=info msg="Patching coreDNS with uninitialized toleration..." time="08:50:43 UTC" level=info msg="Applying addon metrics-server..." time="08:50:46 UTC" level=info msg="Applying addon cni-canal..." time="08:50:47 UTC" level=warning msg="Task failed, error was: failed to load and apply embedded addon: failed to apply addons: Process exited with status 1" time="08:50:55 UTC" level=warning msg="Retrying task..." time="08:50:56 UTC" level=info msg="Applying addon metrics-server..."

$ kubeone apply --manifest kubeone.yaml -t tf.json -y -v time="09:06:38 UTC" level=info msg="Determine hostname..." [xxx.xxx.xxx.xxx] + export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/sbin:/usr/local/bin:/opt/bin [xxx.xxx.xxx.xxx] + PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/sbin:/usr/local/bin:/opt/bin [xxx.xxx.xxx.xxx] ++ hostname -f [xxx.xxx.xxx.xxx] + fqdn=staging-control-plane-3 [xxx.xxx.xxx.xxx] staging-control-plane-3 [xxx.xxx.xxx.xxx] + '[' staging-control-plane-3 = localhost ']' [xxx.xxx.xxx.xxx] + echo staging-control-plane-3 time="09:06:39 UTC" level=debug msg="Hostname is detected: \"staging-control-plane-3\"" node=xxx.xxx.xxx.xxx [xxx.xxx.xxx.xxx] + export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/sbin:/usr/local/bin:/opt/bin [xxx.xxx.xxx.xxx] + PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/sbin:/usr/local/bin:/opt/bin [xxx.xxx.xxx.xxx] ++ hostname -f [xxx.xxx.xxx.xxx] staging-control-plane-1 [xxx.xxx.xxx.xxx] + fqdn=staging-control-plane-1 [xxx.xxx.xxx.xxx] + '[' staging-control-plane-1 = localhost ']' [xxx.xxx.xxx.xxx] + echo staging-control-plane-1 time="09:06:39 UTC" level=debug msg="Hostname is detected: \"staging-control-plane-1\"" node=xxx.xxx.xxx.xxx [xxx.xxx.xxx.xxx] + export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/sbin:/usr/local/bin:/opt/bin [xxx.xxx.xxx.xxx] + PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/sbin:/usr/local/bin:/opt/bin [xxx.xxx.xxx.xxx] ++ hostname -f [xxx.xxx.xxx.xxx] + fqdn=staging-control-plane-2 [xxx.xxx.xxx.xxx] staging-control-plane-2 [xxx.xxx.xxx.xxx] + '[' staging-control-plane-2 = localhost ']' [xxx.xxx.xxx.xxx] + echo staging-control-plane-2 time="09:06:39 UTC" level=debug msg="Hostname is detected: \"staging-control-plane-2\"" node=xxx.xxx.xxx.xxx time="09:06:39 UTC" level=info msg="Determine operating system..." time="09:06:39 UTC" level=debug msg="Operating system detected: \"ubuntu\"" node=xxx.xxx.xxx.xxx time="09:06:39 UTC" level=debug msg="Operating system detected: \"ubuntu\"" node=xxx.xxx.xxx.xxx time="09:06:39 UTC" level=debug msg="Operating system detected: \"ubuntu\"" node=xxx.xxx.xxx.xxx time="09:06:39 UTC" level=info msg="Running host probes..." time="09:06:41 UTC" level=info msg="Electing cluster leader..." time="09:06:41 UTC" level=info msg="Elected leader \"staging-control-plane-1\"..." time="09:06:42 UTC" level=info msg="Building Kubernetes clientset..." time="09:06:42 UTC" level=info msg="Running cluster probes..." Host: "staging-control-plane-1" Host initialized: yes containerd healthy: yes (1.4.13) Kubelet healthy: yes (1.23.7) containerd is installed: yes containerd is running: yes containerd is active: yes containerd is restarting: no kubelet is installed: yes kubelet is running: yes kubelet is active: yes kubelet is restarting: no Host: "staging-control-plane-2" Host initialized: yes containerd healthy: yes (1.4.13) Kubelet healthy: yes (1.23.7) containerd is installed: yes containerd is running: yes containerd is active: yes containerd is restarting: no kubelet is installed: yes kubelet is running: yes kubelet is active: yes kubelet is restarting: no Host: "staging-control-plane-3" Host initialized: yes containerd healthy: yes (1.4.13) docker healthy: yes (20.10.14) Kubelet healthy: yes (1.23.7) containerd is installed: yes containerd is running: yes containerd is active: yes containerd is restarting: no docker is installed: yes docker is running: yes docker is active: yes docker is restarting: no kubelet is installed: yes kubelet is running: yes kubelet is active: yes kubelet is restarting: no The following actions will be taken: ~ ensure embedded addons ~ ensure CNI ~ ensure custom addons ~ ensure credential ~ ensure external CCM time="09:06:43 UTC" level=info msg="Patching static pods..." time="09:06:43 UTC" level=info msg="Patching static pods..." time="09:06:43 UTC" level=info msg="Patching static pods..." time="09:06:43 UTC" level=info msg="Downloading kubeconfig..." time="09:06:43 UTC" level=info msg="Downloading PKI..." time="09:06:44 UTC" level=info msg="Creating local backup..." node=xxx.xxx.xxx.xxx time="09:06:44 UTC" level=info msg="Activating additional features..." time="09:06:44 UTC" level=info msg="Patching coreDNS with uninitialized toleration..." time="09:06:44 UTC" level=info msg="Applying addon metrics-server..." time="09:06:44 UTC" level=info msg="Parsing addons manifest 'metrics-server.yaml'\n" + sudo KUBECONFIG=/etc/kubernetes/admin.conf \ kubectl apply -f - --prune -l "kubeone.io/addon=metrics-server" clusterrole.rbac.authorization.k8s.io/system:aggregated-metrics-reader unchanged serviceaccount/metrics-server unchanged rolebinding.rbac.authorization.k8s.io/metrics-server-auth-reader unchanged clusterrolebinding.rbac.authorization.k8s.io/metrics-server:system:auth-delegator unchanged clusterrole.rbac.authorization.k8s.io/system:metrics-server unchanged clusterrolebinding.rbac.authorization.k8s.io/system:metrics-server unchanged apiservice.apiregistration.k8s.io/v1beta1.metrics.k8s.io unchanged secret/metrics-server-serving-cert configured deployment.apps/metrics-server configured service/metrics-server unchanged time="09:06:47 UTC" level=info msg="Applying addon cni-canal..." time="09:06:47 UTC" level=info msg="Skipping file \"README.md\" because it's not .yaml/.yml/.json file\n" time="09:06:47 UTC" level=info msg="Parsing addons manifest 'canal.yaml'\n" time="09:06:48 UTC" level=warning msg="Task failed, error was: failed to load and apply embedded addon: failed to apply addons: Process exited with status 1" + sudo KUBECONFIG=/etc/kubernetes/admin.conf \ kubectl apply -f - --prune -l "kubeone.io/addon=cni-canal" Warning: policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget Error from server (Invalid): error when applying patch: {"metadata":{"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"apiextensions.k8s.io/v1\",\"kind\":\"CustomResourceDefinition\",\"metadata\":{\"annotations\":{},\"labels\":{\"kubeone.io/addon\":\"cni-canal\"},\"name\":\"felixconfigurations.crd.projectcalico.org\"},\"spec\":{\"group\":\"crd.projectcalico.org\",\"names\":{\"kind\":\"FelixConfiguration\",\"listKind\":\"FelixConfigurationList\",\"plural\":\"felixconfigurations\",\"singular\":\"felixconfiguration\"},\"scope\":\"Cluster\",\"versions\":[{\"name\":\"v1\",\"schema\":{\"openAPIV3Schema\":{\"description\":\"Felix Configuration contains the configuration for Felix.\",\"properties\":{\"apiVersion\":{\"description\":\"APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources\",\"type\":\"string\"},\"kind\":{\"description\":\"Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds\",\"type\":\"string\"},\"metadata\":{\"type\":\"object\"},\"spec\":{\"description\":\"FelixConfigurationSpec contains the values of the Felix configuration.\",\"properties\":{\"allowIPIPPacketsFromWorkloads\":{\"description\":\"AllowIPIPPacketsFromWorkloads controls whether Felix will add a rule to drop IPIP encapsulated traffic from workloads [Default: false]\",\"type\":\"boolean\"},\"allowVXLANPacketsFromWorkloads\":{\"description\":\"AllowVXLANPacketsFromWorkloads controls whether Felix will add a rule to drop VXLAN encapsulated traffic from workloads [Default: false]\",\"type\":\"boolean\"},\"awsSrcDstCheck\":{\"description\":\"Set source-destination-check on AWS EC2 instances. Accepted value must be one of \\\"DoNothing\\\", \\\"Enable\\\" or \\\"Disable\\\". [Default: DoNothing]\",\"enum\":[\"DoNothing\",\"Enable\",\"Disable\"],\"type\":\"string\"},\"bpfConnectTimeLoadBalancingEnabled\":{\"description\":\"BPFConnectTimeLoadBalancingEnabled when in BPF mode, controls whether Felix installs the connection-time load balancer. The connect-time load balancer is required for the host to be able to reach Kubernetes services and it improves the performance of pod-to-service connections. The only reason to disable it is for debugging purposes. [Default: true]\",\"type\":\"boolean\"},\"bpfDataIfacePattern\":{\"description\":\"BPFDataIfacePattern is a regular expression that controls which interfaces Felix should attach BPF programs to in order to catch traffic to/from the network. This needs to match the interfaces that Calico workload traffic flows over as well as any interfaces that handle incoming traffic to nodeports and services from outside the cluster. It should not match the workload interfaces (usually named cali...).\",\"type\":\"string\"},\"bpfDisableUnprivileged\":{\"description\":\"BPFDisableUnprivileged, if enabled, Felix sets the kernel.unprivileged_bpf_disabled sysctl to disable unprivileged use of BPF. This ensures that unprivileged users cannot access Calico's BPF maps and cannot insert their own BPF programs to interfere with Calico's. [Default: true]\",\"type\":\"boolean\"},\"bpfEnabled\":{\"description\":\"BPFEnabled, if enabled Felix will use the BPF dataplane. [Default: false]\",\"type\":\"boolean\"},\"bpfExtToServiceConnmark\":{\"description\":\"BPFExtToServiceConnmark in BPF mode, control a 32bit mark that is set on connections from an external client to a local service. This mark allows us to control how packets of that connection are routed within the host and how is routing intepreted by RPF check. [Default: 0]\",\"type\":\"integer\"},\"bpfExternalServiceMode\":{\"description\":\"BPFExternalServiceMode in BPF mode, controls how connections from outside the cluster to services (node ports and cluster IPs) are forwarded to remote workloads. If set to \\\"Tunnel\\\" then both request and response traffic is tunneled to the remote node. If set to \\\"DSR\\\", the request traffic is tunneled but the response traffic is sent directly from the remote node. In \\\"DSR\\\" mode, the remote node appears to use the IP of the ingress node; this requires a permissive L2 network. [Default: Tunnel]\",\"type\":\"string\"},\"bpfKubeProxyEndpointSlicesEnabled\":{\"description\":\"BPFKubeProxyEndpointSlicesEnabled in BPF mode, controls whether Felix's embedded kube-proxy accepts EndpointSlices or not.\",\"type\":\"boolean\"},\"bpfKubeProxyIptablesCleanupEnabled\":{\"description\":\"BPFKubeProxyIptablesCleanupEnabled, if enabled in BPF mode, Felix will proactively clean up the upstream Kubernetes kube-proxy's iptables chains. Should only be enabled if kube-proxy is not running. [Default: true]\",\"type\":\"boolean\"},\"bpfKubeProxyMinSyncPeriod\":{\"description\":\"BPFKubeProxyMinSyncPeriod, in BPF mode, controls the minimum time between updates to the dataplane for Felix's embedded kube-proxy. Lower values give reduced set-up latency. Higher values reduce Felix CPU usage by batching up more work. [Default: 1s]\",\"type\":\"string\"},\"bpfLogLevel\":{\"description\":\"BPFLogLevel controls the log level of the BPF programs when in BPF dataplane mode. One of \\\"Off\\\", \\\"Info\\\", or \\\"Debug\\\". The logs are emitted to the BPF trace pipe, accessible with the command `tc exec bpf debug`. [Default: Off].\",\"type\":\"string\"},\"bpfPSNATPorts\":{\"anyOf\":[{\"type\":\"integer\"},{\"type\":\"string\"}],\"description\":\"BPFPSNATPorts sets the range from which we randomly pick a port if there is a source port collision. This should be within the ephemeral range as defined by RFC 6056 (1024â€“65535) and preferably outside the ephemeral ranges used by common operating systems. Linux uses 32768â€“60999, while others mostly use the IANA defined range 49152â€“65535. It is not necessarily a problem if this range overlaps with the operating systems. Both ends of the range are inclusive. [Default: 20000:29999]\",\"pattern\":\"^.*\",\"x-kubernetes-int-or-string\":true},\"chainInsertMode\":{\"description\":\"ChainInsertMode controls whether Felix hooks the kernel's top-level iptables chains by inserting a rule at the top of the chain or by appending a rule at the bottom. insert is the safe default since it prevents Calico's rules from being bypassed. If you switch to append mode, be sure that the other rules in the chains signal acceptance by falling through to the Calico rules, otherwise the Calico policy will be bypassed. [Default: insert]\",\"type\":\"string\"},\"dataplaneDriver\":{\"type\":\"string\"},\"debugDisableLogDropping\":{\"type\":\"boolean\"},\"debugMemoryProfilePath\":{\"type\":\"string\"},\"debugSimulateCalcGraphHangAfter\":{\"type\":\"string\"},\"debugSimulateDataplaneHangAfter\":{\"type\":\"string\"},\"defaultEndpointToHostAction\":{\"description\":\"DefaultEndpointToHostAction controls what happens to traffic that goes from a workload endpoint to the host itself (after the traffic hits the endpoint egress policy). By default Calico blocks traffic from workload endpoints to the host itself with an iptables \\\"DROP\\\" action. If you want to allow some or all traffic from endpoint to host, set this parameter to RETURN or ACCEPT. Use RETURN if you have your own rules in the iptables \\\"INPUT\\\" chain; Calico will insert its rules at the top of that chain, then \\\"RETURN\\\" packets to the \\\"INPUT\\\" chain once it has completed processing workload endpoint egress policy. Use ACCEPT to unconditionally accept packets from workloads after processing workload endpoint egress policy. [Default: Drop]\",\"type\":\"string\"},\"deviceRouteProtocol\":{\"description\":\"This defines the route protocol added to programmed device routes, by default this will be RTPROT_BOOT when left blank.\",\"type\":\"integer\"},\"deviceRouteSourceAddress\":{\"description\":\"This is the source address to use on programmed device routes. By default the source address is left blank, leaving the kernel to choose the source address used.\",\"type\":\"string\"},\"disableConntrackInvalidCheck\":{\"type\":\"boolean\"},\"endpointReportingDelay\":{\"type\":\"string\"},\"endpointReportingEnabled\":{\"type\":\"boolean\"},\"externalNodesList\":{\"description\":\"ExternalNodesCIDRList is a list of CIDR's of external-non-calico-nodes which may source tunnel traffic and have the tunneled traffic be accepted at calico nodes.\",\"items\":{\"type\":\"string\"},\"type\":\"array\"},\"failsafeInboundHostPorts\":{\"description\":\"FailsafeInboundHostPorts is a list of UDP/TCP ports and CIDRs that Felix will allow incoming traffic to host endpoints on irrespective of the security policy. This is useful to avoid accidentally cutting off a host with incorrect configuration. For back-compatibility, if the protocol is not specified, it defaults to \\\"tcp\\\". If a CIDR is not specified, it will allow traffic from all addresses. To disable all inbound host ports, use the value none. The default value allows ssh access and DHCP. [Default: tcp:22, udp:68, tcp:179, tcp:2379, tcp:2380, tcp:6443, tcp:6666, tcp:6667]\",\"items\":{\"description\":\"ProtoPort is combination of protocol, port, and CIDR. Protocol and port must be specified.\",\"properties\":{\"net\":{\"type\":\"string\"},\"port\":{\"type\":\"integer\"},\"protocol\":{\"type\":\"string\"}},\"required\":[\"port\",\"protocol\"],\"type\":\"object\"},\"type\":\"array\"},\"failsafeOutboundHostPorts\":{\"description\":\"FailsafeOutboundHostPorts is a list of UDP/TCP ports and CIDRs that Felix will allow outgoing traffic from host endpoints to irrespective of the security policy. This is useful to avoid accidentally cutting off a host with incorrect configuration. For back-compatibility, if the protocol is not specified, it defaults to \\\"tcp\\\". If a CIDR is not specified, it will allow traffic from all addresses. To disable all outbound host ports, use the value none. The default value opens etcd's standard ports to ensure that Felix does not get cut off from etcd as well as allowing DHCP and DNS. [Default: tcp:179, tcp:2379, tcp:2380, tcp:6443, tcp:6666, tcp:6667, udp:53, udp:67]\",\"items\":{\"description\":\"ProtoPort is combination of protocol, port, and CIDR. Protocol and port must be specified.\",\"properties\":{\"net\":{\"type\":\"string\"},\"port\":{\"type\":\"integer\"},\"protocol\":{\"type\":\"string\"}},\"required\":[\"port\",\"protocol\"],\"type\":\"object\"},\"type\":\"array\"},\"featureDetectOverride\":{\"description\":\"FeatureDetectOverride is used to override the feature detection. Values are specified in a comma separated list with no spaces, example; \\\"SNATFullyRandom=true,MASQFullyRandom=false,RestoreSupportsLock=\\\". \\\"true\\\" or \\\"false\\\" will force the feature, empty or omitted values are auto-detected.\",\"type\":\"string\"},\"floatingIPs\":{\"default\":\"Disabled\",\"description\":\"FloatingIPs configures whether or not Felix will program floating IP addresses.\",\"enum\":[\"Enabled\",\"Disabled\"],\"type\":\"string\"},\"genericXDPEnabled\":{\"description\":\"GenericXDPEnabled enables Generic XDP so network cards that don't support XDP offload or driver modes can use XDP. This is not recommended since it doesn't provide better performance than iptables. [Default: false]\",\"type\":\"boolean\"},\"healthEnabled\":{\"type\":\"boolean\"},\"healthHost\":{\"type\":\"string\"},\"healthPort\":{\"type\":\"integer\"},\"interfaceExclude\":{\"description\":\"InterfaceExclude is a comma-separated list of interfaces that Felix should exclude when monitoring for host endpoints. The default value ensures that Felix ignores Kubernetes' IPVS dummy interface, which is used internally by kube-proxy. If you want to exclude multiple interface names using a single value, the list supports regular expressions. For regular expressions you must wrap the value with '/'. For example having values '/^kube/,veth1' will exclude all interfaces that begin with 'kube' and also the interface 'veth1'. [Default: kube-ipvs0]\",\"type\":\"string\"},\"interfacePrefix\":{\"description\":\"InterfacePrefix is the interface name prefix that identifies workload endpoints and so distinguishes them from host endpoint interfaces. Note: in environments other than bare metal, the orchestrators configure this appropriately. For example our Kubernetes and Docker integrations set the 'cali' value, and our OpenStack integration sets the 'tap' value. [Default: cali]\",\"type\":\"string\"},\"interfaceRefreshInterval\":{\"description\":\"InterfaceRefreshInterval is the period at which Felix rescans local interfaces to verify their state. The rescan can be disabled by setting the interval to 0.\",\"type\":\"string\"},\"ipipEnabled\":{\"type\":\"boolean\"},\"ipipMTU\":{\"description\":\"IPIPMTU is the MTU to set on the tunnel device. See Configuring MTU [Default: 1440]\",\"type\":\"integer\"},\"ipsetsRefreshInterval\":{\"description\":\"IpsetsRefreshInterval is the period at which Felix re-checks all iptables state to ensure that no other process has accidentally broken Calico's rules. Set to 0 to disable iptables refresh. [Default: 90s]\",\"type\":\"string\"},\"iptablesBackend\":{\"description\":\"IptablesBackend specifies which backend of iptables will be used. The default is legacy.\",\"type\":\"string\"},\"iptablesFilterAllowAction\":{\"type\":\"string\"},\"iptablesLockFilePath\":{\"description\":\"IptablesLockFilePath is the location of the iptables lock file. You may need to change this if the lock file is not in its standard location (for example if you have mapped it into Felix's container at a different path). [Default: /run/xtables.lock]\",\"type\":\"string\"},\"iptablesLockProbeInterval\":{\"description\":\"IptablesLockProbeInterval is the time that Felix will wait between attempts to acquire the iptables lock if it is not available. Lower values make Felix more responsive when the lock is contended, but use more CPU. [Default: 50ms]\",\"type\":\"string\"},\"iptablesLockTimeout\":{\"description\":\"IptablesLockTimeout is the time that Felix will wait for the iptables lock, or 0, to disable. To use this feature, Felix must share the iptables lock file with all other processes that also take the lock. When running Felix inside a container, this requires the /run directory of the host to be mounted into the calico/node or calico/felix container. [Default: 0s disabled]\",\"type\":\"string\"},\"iptablesMangleAllowAction\":{\"type\":\"string\"},\"iptablesMarkMask\":{\"description\":\"IptablesMarkMask is the mask that Felix selects its IPTables Mark bits from. Should be a 32 bit hexadecimal number with at least 8 bits set, none of which clash with any other mark bits in use on the system. [Default: 0xff000000]\",\"format\":\"int32\",\"type\":\"integer\"},\"iptablesNATOutgoingInterfaceFilter\":{\"type\":\"string\"},\"iptablesPostWriteCheckInterval\":{\"description\":\"IptablesPostWriteCheckInterval is the period after Felix has done a write to the dataplane that it schedules an extra read back in order to check the write was not clobbered by another process. This should only occur if another application on the system doesn't respect the iptables lock. [Default: 1s]\",\"type\":\"string\"},\"iptablesRefreshInterval\":{\"description\":\"IptablesRefreshInterval is the period at which Felix re-checks the IP sets in the dataplane to ensure that no other process has accidentally broken Calico's rules. Set to 0 to disable IP sets refresh. Note: the default for this value is lower than the other refresh intervals as a workaround for a Linux kernel bug that was fixed in kernel version 4.11. If you are using v4.11 or greater you may want to set this to, a higher value to reduce Felix CPU usage. [Default: 10s]\",\"type\":\"string\"},\"ipv6Support\":{\"type\":\"boolean\"},\"kubeNodePortRanges\":{\"description\":\"KubeNodePortRanges holds list of port ranges used for service node ports. Only used if felix detects kube-proxy running in ipvs mode. Felix uses these ranges to separate host and workload traffic. [Default: 30000:32767].\",\"items\":{\"anyOf\":[{\"type\":\"integer\"},{\"type\":\"string\"}],\"pattern\":\"^.*\",\"x-kubernetes-int-or-string\":true},\"type\":\"array\"},\"logDebugFilenameRegex\":{\"description\":\"LogDebugFilenameRegex controls which source code files have their Debug log output included in the logs. Only logs from files with names that match the given regular expression are included. The filter only applies to Debug level logs.\",\"type\":\"string\"},\"logFilePath\":{\"description\":\"LogFilePath is the full path to the Felix log. Set to none to disable file logging. [Default: /var/log/calico/felix.log]\",\"type\":\"string\"},\"logPrefix\":{\"description\":\"LogPrefix is the log prefix that Felix uses when rendering LOG rules. [Default: calico-packet]\",\"type\":\"string\"},\"logSeverityFile\":{\"description\":\"LogSeverityFile is the log severity above which logs are sent to the log file. [Default: Info]\",\"type\":\"string\"},\"logSeverityScreen\":{\"description\":\"LogSeverityScreen is the log severity above which logs are sent to the stdout. [Default: Info]\",\"type\":\"string\"},\"logSeveritySys\":{\"description\":\"LogSeveritySys is the log severity above which logs are sent to the syslog. Set to None for no logging to syslog. [Default: Info]\",\"type\":\"string\"},\"maxIpsetSize\":{\"type\":\"integer\"},\"metadataAddr\":{\"description\":\"MetadataAddr is the IP address or domain name of the server that can answer VM queries for cloud-init metadata. In OpenStack, this corresponds to the machine running nova-api (or in Ubuntu, nova-api-metadata). A value of none (case insensitive) means that Felix should not set up any NAT rule for the metadata path. [Default: 127.0.0.1]\",\"type\":\"string\"},\"metadataPort\":{\"description\":\"MetadataPort is the port of the metadata server. This, combined with global.MetadataAddr (if not 'None'), is used to set up a NAT rule, from 169.254.169.254:80 to MetadataAddr:MetadataPort. In most cases this should not need to be changed [Default: 8775].\",\"type\":\"integer\"},\"mtuIfacePattern\":{\"description\":\"MTUIfacePattern is a regular expression that controls which interfaces Felix should scan in order to calculate the host's MTU. This should not match workload interfaces (usually named cali...).\",\"type\":\"string\"},\"natOutgoingAddress\":{\"description\":\"NATOutgoingAddress specifies an address to use when performing source NAT for traffic in a natOutgoing pool that is leaving the network. By default the address used is an address on the interface the traffic is leaving on (ie it uses the iptables MASQUERADE target)\",\"type\":\"string\"},\"natPortRange\":{\"anyOf\":[{\"type\":\"integer\"},{\"type\":\"string\"}],\"description\":\"NATPortRange specifies the range of ports that is used for port mapping when doing outgoing NAT. When unset the default behavior of the network stack is used.\",\"pattern\":\"^.*\",\"x-kubernetes-int-or-string\":true},\"netlinkTimeout\":{\"type\":\"string\"},\"openstackRegion\":{\"description\":\"OpenstackRegion is the name of the region that a particular Felix belongs to. In a multi-region Calico/OpenStack deployment, this must be configured somehow for each Felix (here in the datamodel, or in felix.cfg or the environment on each compute node), and must match the [calico] openstack_region value configured in neutron.conf on each node. [Default: Empty]\",\"type\":\"string\"},\"policySyncPathPrefix\":{\"description\":\"PolicySyncPathPrefix is used to by Felix to communicate policy changes to external services, like Application layer policy. [Default: Empty]\",\"type\":\"string\"},\"prometheusGoMetricsEnabled\":{\"description\":\"PrometheusGoMetricsEnabled disables Go runtime metrics collection, which the Prometheus client does by default, when set to false. This reduces the number of metrics reported, reducing Prometheus load. [Default: true]\",\"type\":\"boolean\"},\"prometheusMetricsEnabled\":{\"description\":\"PrometheusMetricsEnabled enables the Prometheus metrics server in Felix if set to true. [Default: false]\",\"type\":\"boolean\"},\"prometheusMetricsHost\":{\"description\":\"PrometheusMetricsHost is the host that the Prometheus metrics server should bind to. [Default: empty]\",\"type\":\"string\"},\"prometheusMetricsPort\":{\"description\":\"PrometheusMetricsPort is the TCP port that the Prometheus metrics server should bind to. [Default: 9091]\",\"type\":\"integer\"},\"prometheusProcessMetricsEnabled\":{\"description\":\"PrometheusProcessMetricsEnabled disables process metrics collection, which the Prometheus client does by default, when set to false. This reduces the number of metrics reported, reducing Prometheus load. [Default: true]\",\"type\":\"boolean\"},\"prometheusWireGuardMetricsEnabled\":{\"description\":\"PrometheusWireGuardMetricsEnabled disables wireguard metrics collection, which the Prometheus client does by default, when set to false. This reduces the number of metrics reported, reducing Prometheus load. [Default: true]\",\"type\":\"boolean\"},\"removeExternalRoutes\":{\"description\":\"Whether or not to remove device routes that have not been programmed by Felix. Disabling this will allow external applications to also add device routes. This is enabled by default which means we will remove externally added routes.\",\"type\":\"boolean\"},\"reportingInterval\":{\"description\":\"ReportingInterval is the interval at which Felix reports its status into the datastore or 0 to disable. Must be non-zero in OpenStack deployments. [Default: 30s]\",\"type\":\"string\"},\"reportingTTL\":{\"description\":\"ReportingTTL is the time-to-live setting for process-wide status reports. [Default: 90s]\",\"type\":\"string\"},\"routeRefreshInterval\":{\"description\":\"RouteRefreshInterval is the period at which Felix re-checks the routes in the dataplane to ensure that no other process has accidentally broken Calico's rules. Set to 0 to disable route refresh. [Default: 90s]\",\"type\":\"string\"},\"routeSource\":{\"description\":\"RouteSource configures where Felix gets its routing information. - WorkloadIPs: use workload endpoints to construct routes. - CalicoIPAM: the default - use IPAM data to construct routes.\",\"type\":\"string\"},\"routeTableRange\":{\"description\":\"Calico programs additional Linux route tables for various purposes. RouteTableRange specifies the indices of the route tables that Calico should use.\",\"properties\":{\"max\":{\"type\":\"integer\"},\"min\":{\"type\":\"integer\"}},\"required\":[\"max\",\"min\"],\"type\":\"object\"},\"serviceLoopPrevention\":{\"description\":\"When service IP advertisement is enabled, prevent routing loops to service IPs that are not in use, by dropping or rejecting packets that do not get DNAT'd by kube-proxy. Unless set to \\\"Disabled\\\", in which case such routing loops continue to be allowed. [Default: Drop]\",\"type\":\"string\"},\"sidecarAccelerationEnabled\":{\"description\":\"SidecarAccelerationEnabled enables experimental sidecar acceleration [Default: false]\",\"type\":\"boolean\"},\"usageReportingEnabled\":{\"description\":\"UsageReportingEnabled reports anonymous Calico version number and cluster size to projectcalico.org. Logs warnings returned by the usage server. For example, if a significant security vulnerability has been discovered in the version of Calico being used. [Default: true]\",\"type\":\"boolean\"},\"usageReportingInitialDelay\":{\"description\":\"UsageReportingInitialDelay controls the minimum delay before Felix makes a report. [Default: 300s]\",\"type\":\"string\"},\"usageReportingInterval\":{\"description\":\"UsageReportingInterval controls the interval at which Felix makes reports. [Default: 86400s]\",\"type\":\"string\"},\"useInternalDataplaneDriver\":{\"type\":\"boolean\"},\"vxlanEnabled\":{\"type\":\"boolean\"},\"vxlanMTU\":{\"description\":\"VXLANMTU is the MTU to set on the tunnel device. See Configuring MTU [Default: 1440]\",\"type\":\"integer\"},\"vxlanPort\":{\"type\":\"integer\"},\"vxlanVNI\":{\"type\":\"integer\"},\"wireguardEnabled\":{\"description\":\"WireguardEnabled controls whether Wireguard is enabled. [Default: false]\",\"type\":\"boolean\"},\"wireguardHostEncryptionEnabled\":{\"description\":\"WireguardHostEncryptionEnabled controls whether Wireguard host-to-host encryption is enabled. [Default: false]\",\"type\":\"boolean\"},\"wireguardInterfaceName\":{\"description\":\"WireguardInterfaceName specifies the name to use for the Wireguard interface. [Default: wg.calico]\",\"type\":\"string\"},\"wireguardListeningPort\":{\"description\":\"WireguardListeningPort controls the listening port used by Wireguard. [Default: 51820]\",\"type\":\"integer\"},\"wireguardMTU\":{\"description\":\"WireguardMTU controls the MTU on the Wireguard interface. See Configuring MTU [Default: 1420]\",\"type\":\"integer\"},\"wireguardRoutingRulePriority\":{\"description\":\"WireguardRoutingRulePriority controls the priority value to use for the Wireguard routing rule. [Default: 99]\",\"type\":\"integer\"},\"xdpEnabled\":{\"description\":\"XDPEnabled enables XDP acceleration for suitable untracked incoming deny rules. [Default: true]\",\"type\":\"boolean\"},\"xdpRefreshInterval\":{\"description\":\"XDPRefreshInterval is the period at which Felix re-checks all XDP state to ensure that no other process has accidentally broken Calico's BPF maps or attached programs. Set to 0 to disable XDP refresh. [Default: 90s]\",\"type\":\"string\"}},\"type\":\"object\"}},\"type\":\"object\"}},\"served\":true,\"storage\":true}]},\"status\":{\"acceptedNames\":{\"kind\":\"\",\"plural\":\"\"},\"conditions\":[],\"storedVersions\":[]}}\n"}},"spec":{"versions":[{"name":"v1","schema":{"openAPIV3Schema":{"description":"Felix Configuration contains the configuration for Felix.","properties":{"apiVersion":{"description":"APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources","type":"string"},"kind":{"description":"Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds","type":"string"},"metadata":{"type":"object"},"spec":{"description":"FelixConfigurationSpec contains the values of the Felix configuration.","properties":{"allowIPIPPacketsFromWorkloads":{"description":"AllowIPIPPacketsFromWorkloads controls whether Felix will add a rule to drop IPIP encapsulated traffic from workloads [Default: false]","type":"boolean"},"allowVXLANPacketsFromWorkloads":{"description":"AllowVXLANPacketsFromWorkloads controls whether Felix will add a rule to drop VXLAN encapsulated traffic from workloads [Default: false]","type":"boolean"},"awsSrcDstCheck":{"description":"Set source-destination-check on AWS EC2 instances. Accepted value must be one of \"DoNothing\", \"Enable\" or \"Disable\". [Default: DoNothing]","enum":["DoNothing","Enable","Disable"],"type":"string"},"bpfConnectTimeLoadBalancingEnabled":{"description":"BPFConnectTimeLoadBalancingEnabled when in BPF mode, controls whether Felix installs the connection-time load balancer. The connect-time load balancer is required for the host to be able to reach Kubernetes services and it improves the performance of pod-to-service connections. The only reason to disable it is for debugging purposes. [Default: true]","type":"boolean"},"bpfDataIfacePattern":{"description":"BPFDataIfacePattern is a regular expression that controls which interfaces Felix should attach BPF programs to in order to catch traffic to/from the network. This needs to match the interfaces that Calico workload traffic flows over as well as any interfaces that handle incoming traffic to nodeports and services from outside the cluster. It should not match the workload interfaces (usually named cali...).","type":"string"},"bpfDisableUnprivileged":{"description":"BPFDisableUnprivileged, if enabled, Felix sets the kernel.unprivileged_bpf_disabled sysctl to disable unprivileged use of BPF. This ensures that unprivileged users cannot access Calico's BPF maps and cannot insert their own BPF programs to interfere with Calico's. [Default: true]","type":"boolean"},"bpfEnabled":{"description":"BPFEnabled, if enabled Felix will use the BPF dataplane. [Default: false]","type":"boolean"},"bpfExtToServiceConnmark":{"description":"BPFExtToServiceConnmark in BPF mode, control a 32bit mark that is set on connections from an external client to a local service. This mark allows us to control how packets of that connection are routed within the host and how is routing intepreted by RPF check. [Default: 0]","type":"integer"},"bpfExternalServiceMode":{"description":"BPFExternalServiceMode in BPF mode, controls how connections from outside the cluster to services (node ports and cluster IPs) are forwarded to remote workloads. If set to \"Tunnel\" then both request and response traffic is tunneled to the remote node. If set to \"DSR\", the request traffic is tunneled but the response traffic is sent directly from the remote node. In \"DSR\" mode, the remote node appears to use the IP of the ingress node; this requires a permissive L2 network. [Default: Tunnel]","type":"string"},"bpfKubeProxyEndpointSlicesEnabled":{"description":"BPFKubeProxyEndpointSlicesEnabled in BPF mode, controls whether Felix's embedded kube-proxy accepts EndpointSlices or not.","type":"boolean"},"bpfKubeProxyIptablesCleanupEnabled":{"description":"BPFKubeProxyIptablesCleanupEnabled, if enabled in BPF mode, Felix will proactively clean up the upstream Kubernetes kube-proxy's iptables chains. Should only be enabled if kube-proxy is not running. [Default: true]","type":"boolean"},"bpfKubeProxyMinSyncPeriod":{"description":"BPFKubeProxyMinSyncPeriod, in BPF mode, controls the minimum time between updates to the dataplane for Felix's embedded kube-proxy. Lower values give reduced set-up latency. Higher values reduce Felix CPU usage by batching up more work. [Default: 1s]","type":"string"},"bpfLogLevel":{"description":"BPFLogLevel controls the log level of the BPF programs when in BPF dataplane mode. One of \"Off\", \"Info\", or \"Debug\". The logs are emitted to the BPF trace pipe, accessible with the command `tc exec bpf debug`. [Default: Off].","type":"string"},"bpfPSNATPorts":{"anyOf":[{"type":"integer"},{"type":"string"}],"description":"BPFPSNATPorts sets the range from which we randomly pick a port if there is a source port collision. This should be within the ephemeral range as defined by RFC 6056 (1024â€“65535) and preferably outside the ephemeral ranges used by common operating systems. Linux uses 32768â€“60999, while others mostly use the IANA defined range 49152â€“65535. It is not necessarily a problem if this range overlaps with the operating systems. Both ends of the range are inclusive. [Default: 20000:29999]","pattern":"^.*","x-kubernetes-int-or-string":true},"chainInsertMode":{"description":"ChainInsertMode controls whether Felix hooks the kernel's top-level iptables chains by inserting a rule at the top of the chain or by appending a rule at the bottom. insert is the safe default since it prevents Calico's rules from being bypassed. If you switch to append mode, be sure that the other rules in the chains signal acceptance by falling through to the Calico rules, otherwise the Calico policy will be bypassed. [Default: insert]","type":"string"},"dataplaneDriver":{"type":"string"},"debugDisableLogDropping":{"type":"boolean"},"debugMemoryProfilePath":{"type":"string"},"debugSimulateCalcGraphHangAfter":{"type":"string"},"debugSimulateDataplaneHangAfter":{"type":"string"},"defaultEndpointToHostAction":{"description":"DefaultEndpointToHostAction controls what happens to traffic that goes from a workload endpoint to the host itself (after the traffic hits the endpoint egress policy). By default Calico blocks traffic from workload endpoints to the host itself with an iptables \"DROP\" action. If you want to allow some or all traffic from endpoint to host, set this parameter to RETURN or ACCEPT. Use RETURN if you have your own rules in the iptables \"INPUT\" chain; Calico will insert its rules at the top of that chain, then \"RETURN\" packets to the \"INPUT\" chain once it has completed processing workload endpoint egress policy. Use ACCEPT to unconditionally accept packets from workloads after processing workload endpoint egress policy. [Default: Drop]","type":"string"},"deviceRouteProtocol":{"description":"This defines the route protocol added to programmed device routes, by default this will be RTPROT_BOOT when left blank.","type":"integer"},"deviceRouteSourceAddress":{"description":"This is the source address to use on programmed device routes. By default the source address is left blank, leaving the kernel to choose the source address used.","type":"string"},"disableConntrackInvalidCheck":{"type":"boolean"},"endpointReportingDelay":{"type":"string"},"endpointReportingEnabled":{"type":"boolean"},"externalNodesList":{"description":"ExternalNodesCIDRList is a list of CIDR's of external-non-calico-nodes which may source tunnel traffic and have the tunneled traffic be accepted at calico nodes.","items":{"type":"string"},"type":"array"},"failsafeInboundHostPorts":{"description":"FailsafeInboundHostPorts is a list of UDP/TCP ports and CIDRs that Felix will allow incoming traffic to host endpoints on irrespective of the security policy. This is useful to avoid accidentally cutting off a host with incorrect configuration. For back-compatibility, if the protocol is not specified, it defaults to \"tcp\". If a CIDR is not specified, it will allow traffic from all addresses. To disable all inbound host ports, use the value none. The default value allows ssh access and DHCP. [Default: tcp:22, udp:68, tcp:179, tcp:2379, tcp:2380, tcp:6443, tcp:6666, tcp:6667]","items":{"description":"ProtoPort is combination of protocol, port, and CIDR. Protocol and port must be specified.","properties":{"net":{"type":"string"},"port":{"type":"integer"},"protocol":{"type":"string"}},"required":["port","protocol"],"type":"object"},"type":"array"},"failsafeOutboundHostPorts":{"description":"FailsafeOutboundHostPorts is a list of UDP/TCP ports and CIDRs that Felix will allow outgoing traffic from host endpoints to irrespective of the security policy. This is useful to avoid accidentally cutting off a host with incorrect configuration. For back-compatibility, if the protocol is not specified, it defaults to \"tcp\". If a CIDR is not specified, it will allow traffic from all addresses. To disable all outbound host ports, use the value none. The default value opens etcd's standard ports to ensure that Felix does not get cut off from etcd as well as allowing DHCP and DNS. [Default: tcp:179, tcp:2379, tcp:2380, tcp:6443, tcp:6666, tcp:6667, udp:53, udp:67]","items":{"description":"ProtoPort is combination of protocol, port, and CIDR. Protocol and port must be specified.","properties":{"net":{"type":"string"},"port":{"type":"integer"},"protocol":{"type":"string"}},"required":["port","protocol"],"type":"object"},"type":"array"},"featureDetectOverride":{"description":"FeatureDetectOverride is used to override the feature detection. Values are specified in a comma separated list with no spaces, example; \"SNATFullyRandom=true,MASQFullyRandom=false,RestoreSupportsLock=\". \"true\" or \"false\" will force the feature, empty or omitted values are auto-detected.","type":"string"},"floatingIPs":{"default":"Disabled","description":"FloatingIPs configures whether or not Felix will program floating IP addresses.","enum":["Enabled","Disabled"],"type":"string"},"genericXDPEnabled":{"description":"GenericXDPEnabled enables Generic XDP so network cards that don't support XDP offload or driver modes can use XDP. This is not recommended since it doesn't provide better performance than iptables. [Default: false]","type":"boolean"},"healthEnabled":{"type":"boolean"},"healthHost":{"type":"string"},"healthPort":{"type":"integer"},"interfaceExclude":{"description":"InterfaceExclude is a comma-separated list of interfaces that Felix should exclude when monitoring for host endpoints. The default value ensures that Felix ignores Kubernetes' IPVS dummy interface, which is used internally by kube-proxy. If you want to exclude multiple interface names using a single value, the list supports regular expressions. For regular expressions you must wrap the value with '/'. For example having values '/^kube/,veth1' will exclude all interfaces that begin with 'kube' and also the interface 'veth1'. [Default: kube-ipvs0]","type":"string"},"interfacePrefix":{"description":"InterfacePrefix is the interface name prefix that identifies workload endpoints and so distinguishes them from host endpoint interfaces. Note: in environments other than bare metal, the orchestrators configure this appropriately. For example our Kubernetes and Docker integrations set the 'cali' value, and our OpenStack integration sets the 'tap' value. [Default: cali]","type":"string"},"interfaceRefreshInterval":{"description":"InterfaceRefreshInterval is the period at which Felix rescans local interfaces to verify their state. The rescan can be disabled by setting the interval to 0.","type":"string"},"ipipEnabled":{"type":"boolean"},"ipipMTU":{"description":"IPIPMTU is the MTU to set on the tunnel device. See Configuring MTU [Default: 1440]","type":"integer"},"ipsetsRefreshInterval":{"description":"IpsetsRefreshInterval is the period at which Felix re-checks all iptables state to ensure that no other process has accidentally broken Calico's rules. Set to 0 to disable iptables refresh. [Default: 90s]","type":"string"},"iptablesBackend":{"description":"IptablesBackend specifies which backend of iptables will be used. The default is legacy.","type":"string"},"iptablesFilterAllowAction":{"type":"string"},"iptablesLockFilePath":{"description":"IptablesLockFilePath is the location of the iptables lock file. You may need to change this if the lock file is not in its standard location (for example if you have mapped it into Felix's container at a different path). [Default: /run/xtables.lock]","type":"string"},"iptablesLockProbeInterval":{"description":"IptablesLockProbeInterval is the time that Felix will wait between attempts to acquire the iptables lock if it is not available. Lower values make Felix more responsive when the lock is contended, but use more CPU. [Default: 50ms]","type":"string"},"iptablesLockTimeout":{"description":"IptablesLockTimeout is the time that Felix will wait for the iptables lock, or 0, to disable. To use this feature, Felix must share the iptables lock file with all other processes that also take the lock. When running Felix inside a container, this requires the /run directory of the host to be mounted into the calico/node or calico/felix container. [Default: 0s disabled]","type":"string"},"iptablesMangleAllowAction":{"type":"string"},"iptablesMarkMask":{"description":"IptablesMarkMask is the mask that Felix selects its IPTables Mark bits from. Should be a 32 bit hexadecimal number with at least 8 bits set, none of which clash with any other mark bits in use on the system. [Default: 0xff000000]","format":"int32","type":"integer"},"iptablesNATOutgoingInterfaceFilter":{"type":"string"},"iptablesPostWriteCheckInterval":{"description":"IptablesPostWriteCheckInterval is the period after Felix has done a write to the dataplane that it schedules an extra read back in order to check the write was not clobbered by another process. This should only occur if another application on the system doesn't respect the iptables lock. [Default: 1s]","type":"string"},"iptablesRefreshInterval":{"description":"IptablesRefreshInterval is the period at which Felix re-checks the IP sets in the dataplane to ensure that no other process has accidentally broken Calico's rules. Set to 0 to disable IP sets refresh. Note: the default for this value is lower than the other refresh intervals as a workaround for a Linux kernel bug that was fixed in kernel version 4.11. If you are using v4.11 or greater you may want to set this to, a higher value to reduce Felix CPU usage. [Default: 10s]","type":"string"},"ipv6Support":{"type":"boolean"},"kubeNodePortRanges":{"description":"KubeNodePortRanges holds list of port ranges used for service node ports. Only used if felix detects kube-proxy running in ipvs mode. Felix uses these ranges to separate host and workload traffic. [Default: 30000:32767].","items":{"anyOf":[{"type":"integer"},{"type":"string"}],"pattern":"^.*","x-kubernetes-int-or-string":true},"type":"array"},"logDebugFilenameRegex":{"description":"LogDebugFilenameRegex controls which source code files have their Debug log output included in the logs. Only logs from files with names that match the given regular expression are included. The filter only applies to Debug level logs.","type":"string"},"logFilePath":{"description":"LogFilePath is the full path to the Felix log. Set to none to disable file logging. [Default: /var/log/calico/felix.log]","type":"string"},"logPrefix":{"description":"LogPrefix is the log prefix that Felix uses when rendering LOG rules. [Default: calico-packet]","type":"string"},"logSeverityFile":{"description":"LogSeverityFile is the log severity above which logs are sent to the log file. [Default: Info]","type":"string"},"logSeverityScreen":{"description":"LogSeverityScreen is the log severity above which logs are sent to the stdout. [Default: Info]","type":"string"},"logSeveritySys":{"description":"LogSeveritySys is the log severity above which logs are sent to the syslog. Set to None for no logging to syslog. [Default: Info]","type":"string"},"maxIpsetSize":{"type":"integer"},"metadataAddr":{"description":"MetadataAddr is the IP address or domain name of the server that can answer VM queries for cloud-init metadata. In OpenStack, this corresponds to the machine running nova-api (or in Ubuntu, nova-api-metadata). A value of none (case insensitive) means that Felix should not set up any NAT rule for the metadata path. [Default: 127.0.0.1]","type":"string"},"metadataPort":{"description":"MetadataPort is the port of the metadata server. This, combined with global.MetadataAddr (if not 'None'), is used to set up a NAT rule, from 169.254.169.254:80 to MetadataAddr:MetadataPort. In most cases this should not need to be changed [Default: 8775].","type":"integer"},"mtuIfacePattern":{"description":"MTUIfacePattern is a regular expression that controls which interfaces Felix should scan in order to calculate the host's MTU. This should not match workload interfaces (usually named cali...).","type":"string"},"natOutgoingAddress":{"description":"NATOutgoingAddress specifies an address to use when performing source NAT for traffic in a natOutgoing pool that is leaving the network. By default the address used is an address on the interface the traffic is leaving on (ie it uses the iptables MASQUERADE target)","type":"string"},"natPortRange":{"anyOf":[{"type":"integer"},{"type":"string"}],"description":"NATPortRange specifies the range of ports that is used for port mapping when doing outgoing NAT. When unset the default behavior of the network stack is used.","pattern":"^.*","x-kubernetes-int-or-string":true},"netlinkTimeout":{"type":"string"},"openstackRegion":{"description":"OpenstackRegion is the name of the region that a particular Felix belongs to. In a multi-region Calico/OpenStack deployment, this must be configured somehow for each Felix (here in the datamodel, or in felix.cfg or the environment on each compute node), and must match the [calico] openstack_region value configured in neutron.conf on each node. [Default: Empty]","type":"string"},"policySyncPathPrefix":{"description":"PolicySyncPathPrefix is used to by Felix to communicate policy changes to external services, like Application layer policy. [Default: Empty]","type":"string"},"prometheusGoMetricsEnabled":{"description":"PrometheusGoMetricsEnabled disables Go runtime metrics collection, which the Prometheus client does by default, when set to false. This reduces the number of metrics reported, reducing Prometheus load. [Default: true]","type":"boolean"},"prometheusMetricsEnabled":{"description":"PrometheusMetricsEnabled enables the Prometheus metrics server in Felix if set to true. [Default: false]","type":"boolean"},"prometheusMetricsHost":{"description":"PrometheusMetricsHost is the host that the Prometheus metrics server should bind to. [Default: empty]","type":"string"},"prometheusMetricsPort":{"description":"PrometheusMetricsPort is the TCP port that the Prometheus metrics server should bind to. [Default: 9091]","type":"integer"},"prometheusProcessMetricsEnabled":{"description":"PrometheusProcessMetricsEnabled disables process metrics collection, which the Prometheus client does by default, when set to false. This reduces the number of metrics reported, reducing Prometheus load. [Default: true]","type":"boolean"},"prometheusWireGuardMetricsEnabled":{"description":"PrometheusWireGuardMetricsEnabled disables wireguard metrics collection, which the Prometheus client does by default, when set to false. This reduces the number of metrics reported, reducing Prometheus load. [Default: true]","type":"boolean"},"removeExternalRoutes":{"description":"Whether or not to remove device routes that have not been programmed by Felix. Disabling this will allow external applications to also add device routes. This is enabled by default which means we will remove externally added routes.","type":"boolean"},"reportingInterval":{"description":"ReportingInterval is the interval at which Felix reports its status into the datastore or 0 to disable. Must be non-zero in OpenStack deployments. [Default: 30s]","type":"string"},"reportingTTL":{"description":"ReportingTTL is the time-to-live setting for process-wide status reports. [Default: 90s]","type":"string"},"routeRefreshInterval":{"description":"RouteRefreshInterval is the period at which Felix re-checks the routes in the dataplane to ensure that no other process has accidentally broken Calico's rules. Set to 0 to disable route refresh. [Default: 90s]","type":"string"},"routeSource":{"description":"RouteSource configures where Felix gets its routing information. - WorkloadIPs: use workload endpoints to construct routes. - CalicoIPAM: the default - use IPAM data to construct routes.","type":"string"},"routeTableRange":{"description":"Calico programs additional Linux route tables for various purposes. RouteTableRange specifies the indices of the route tables that Calico should use.","properties":{"max":{"type":"integer"},"min":{"type":"integer"}},"required":["max","min"],"type":"object"},"serviceLoopPrevention":{"description":"When service IP advertisement is enabled, prevent routing loops to service IPs that are not in use, by dropping or rejecting packets that do not get DNAT'd by kube-proxy. Unless set to \"Disabled\", in which case such routing loops continue to be allowed. [Default: Drop]","type":"string"},"sidecarAccelerationEnabled":{"description":"SidecarAccelerationEnabled enables experimental sidecar acceleration [Default: false]","type":"boolean"},"usageReportingEnabled":{"description":"UsageReportingEnabled reports anonymous Calico version number and cluster size to projectcalico.org. Logs warnings returned by the usage server. For example, if a significant security vulnerability has been discovered in the version of Calico being used. [Default: true]","type":"boolean"},"usageReportingInitialDelay":{"description":"UsageReportingInitialDelay controls the minimum delay before Felix makes a report. [Default: 300s]","type":"string"},"usageReportingInterval":{"description":"UsageReportingInterval controls the interval at which Felix makes reports. [Default: 86400s]","type":"string"},"useInternalDataplaneDriver":{"type":"boolean"},"vxlanEnabled":{"type":"boolean"},"vxlanMTU":{"description":"VXLANMTU is the MTU to set on the tunnel device. See Configuring MTU [Default: 1440]","type":"integer"},"vxlanPort":{"type":"integer"},"vxlanVNI":{"type":"integer"},"wireguardEnabled":{"description":"WireguardEnabled controls whether Wireguard is enabled. [Default: false]","type":"boolean"},"wireguardHostEncryptionEnabled":{"description":"WireguardHostEncryptionEnabled controls whether Wireguard host-to-host encryption is enabled. [Default: false]","type":"boolean"},"wireguardInterfaceName":{"description":"WireguardInterfaceName specifies the name to use for the Wireguard interface. [Default: wg.calico]","type":"string"},"wireguardListeningPort":{"description":"WireguardListeningPort controls the listening port used by Wireguard. [Default: 51820]","type":"integer"},"wireguardMTU":{"description":"WireguardMTU controls the MTU on the Wireguard interface. See Configuring MTU [Default: 1420]","type":"integer"},"wireguardRoutingRulePriority":{"description":"WireguardRoutingRulePriority controls the priority value to use for the Wireguard routing rule. [Default: 99]","type":"integer"},"xdpEnabled":{"description":"XDPEnabled enables XDP acceleration for suitable untracked incoming deny rules. [Default: true]","type":"boolean"},"xdpRefreshInterval":{"description":"XDPRefreshInterval is the period at which Felix re-checks all XDP state to ensure that no other process has accidentally broken Calico's BPF maps or attached programs. Set to 0 to disable XDP refresh. [Default: 90s]","type":"string"}},"type":"object"}},"type":"object"}},"served":true,"storage":true}]},"status":{"acceptedNames":{"kind":"","plural":""},"conditions":[],"storedVersions":[]}} to: Resource: "apiextensions.k8s.io/v1, Resource=customresourcedefinitions", GroupVersionKind: "apiextensions.k8s.io/v1, Kind=CustomResourceDefinition" Name: "felixconfigurations.crd.projectcalico.org", Namespace: "" for: "STDIN": CustomResourceDefinition.apiextensions.k8s.io "felixconfigurations.crd.projectcalico.org" is invalid: spec.preserveUnknownFields: Invalid value: true: must be false in order to use defaults in the schema Error from server (Invalid): error when applying patch: {"metadata":{"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"apiextensions.k8s.io/v1\",\"kind\":\"CustomResourceDefinition\",\"metadata\":{\"annotations\":{},\"labels\":{\"kubeone.io/addon\":\"cni-canal\"},\"name\":\"ipamblocks.crd.projectcalico.org\"},\"spec\":{\"group\":\"crd.projectcalico.org\",\"names\":{\"kind\":\"IPAMBlock\",\"listKind\":\"IPAMBlockList\",\"plural\":\"ipamblocks\",\"singular\":\"ipamblock\"},\"scope\":\"Cluster\",\"versions\":[{\"name\":\"v1\",\"schema\":{\"openAPIV3Schema\":{\"properties\":{\"apiVersion\":{\"description\":\"APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources\",\"type\":\"string\"},\"kind\":{\"description\":\"Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds\",\"type\":\"string\"},\"metadata\":{\"type\":\"object\"},\"spec\":{\"description\":\"IPAMBlockSpec contains the specification for an IPAMBlock resource.\",\"properties\":{\"affinity\":{\"description\":\"Affinity of the block, if this block has one. If set, it will be of the form \\\"host:\\u003chostname\\u003e\\\". If not set, this block is not affine to a host.\",\"type\":\"string\"},\"allocations\":{\"description\":\"Array of allocations in-use within this block. nil entries mean the allocation is free. For non-nil entries at index i, the index is the ordinal of the allocation within this block and the value is the index of the associated attributes in the Attributes array.\",\"items\":{\"nullable\":true,\"type\":\"integer\"},\"type\":\"array\"},\"attributes\":{\"description\":\"Attributes is an array of arbitrary metadata associated with allocations in the block. To find attributes for a given allocation, use the value of the allocation's entry in the Allocations array as the index of the element in this array.\",\"items\":{\"properties\":{\"handle_id\":{\"type\":\"string\"},\"secondary\":{\"additionalProperties\":{\"type\":\"string\"},\"type\":\"object\"}},\"type\":\"object\"},\"type\":\"array\"},\"cidr\":{\"description\":\"The block's CIDR.\",\"type\":\"string\"},\"deleted\":{\"description\":\"Deleted is an internal boolean used to workaround a limitation in the Kubernetes API whereby deletion will not return a conflict error if the block has been updated. It should not be set manually.\",\"type\":\"boolean\"},\"sequenceNumber\":{\"default\":0,\"description\":\"We store a sequence number that is updated each time the block is written. Each allocation will also store the sequence number of the block at the time of its creation. When releasing an IP, passing the sequence number associated with the allocation allows us to protect against a race condition and ensure the IP hasn't been released and re-allocated since the release request.\",\"format\":\"int64\",\"type\":\"integer\"},\"sequenceNumberForAllocation\":{\"additionalProperties\":{\"format\":\"int64\",\"type\":\"integer\"},\"description\":\"Map of allocated ordinal within the block to sequence number of the block at the time of allocation. Kubernetes does not allow numerical keys for maps, so the key is cast to a string.\",\"type\":\"object\"},\"strictAffinity\":{\"description\":\"StrictAffinity on the IPAMBlock is deprecated and no longer used by the code. Use IPAMConfig StrictAffinity instead.\",\"type\":\"boolean\"},\"unallocated\":{\"description\":\"Unallocated is an ordered list of allocations which are free in the block.\",\"items\":{\"type\":\"integer\"},\"type\":\"array\"}},\"required\":[\"allocations\",\"attributes\",\"cidr\",\"strictAffinity\",\"unallocated\"],\"type\":\"object\"}},\"type\":\"object\"}},\"served\":true,\"storage\":true}]},\"status\":{\"acceptedNames\":{\"kind\":\"\",\"plural\":\"\"},\"conditions\":[],\"storedVersions\":[]}}\n"}},"spec":{"versions":[{"name":"v1","schema":{"openAPIV3Schema":{"properties":{"apiVersion":{"description":"APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources","type":"string"},"kind":{"description":"Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds","type":"string"},"metadata":{"type":"object"},"spec":{"description":"IPAMBlockSpec contains the specification for an IPAMBlock resource.","properties":{"affinity":{"description":"Affinity of the block, if this block has one. If set, it will be of the form \"host:\u003chostname\u003e\". If not set, this block is not affine to a host.","type":"string"},"allocations":{"description":"Array of allocations in-use within this block. nil entries mean the allocation is free. For non-nil entries at index i, the index is the ordinal of the allocation within this block and the value is the index of the associated attributes in the Attributes array.","items":{"nullable":true,"type":"integer"},"type":"array"},"attributes":{"description":"Attributes is an array of arbitrary metadata associated with allocations in the block. To find attributes for a given allocation, use the value of the allocation's entry in the Allocations array as the index of the element in this array.","items":{"properties":{"handle_id":{"type":"string"},"secondary":{"additionalProperties":{"type":"string"},"type":"object"}},"type":"object"},"type":"array"},"cidr":{"description":"The block's CIDR.","type":"string"},"deleted":{"description":"Deleted is an internal boolean used to workaround a limitation in the Kubernetes API whereby deletion will not return a conflict error if the block has been updated. It should not be set manually.","type":"boolean"},"sequenceNumber":{"default":0,"description":"We store a sequence number that is updated each time the block is written. Each allocation will also store the sequence number of the block at the time of its creation. When releasing an IP, passing the sequence number associated with the allocation allows us to protect against a race condition and ensure the IP hasn't been released and re-allocated since the release request.","format":"int64","type":"integer"},"sequenceNumberForAllocation":{"additionalProperties":{"format":"int64","type":"integer"},"description":"Map of allocated ordinal within the block to sequence number of the block at the time of allocation. Kubernetes does not allow numerical keys for maps, so the key is cast to a string.","type":"object"},"strictAffinity":{"description":"StrictAffinity on the IPAMBlock is deprecated and no longer used by the code. Use IPAMConfig StrictAffinity instead.","type":"boolean"},"unallocated":{"description":"Unallocated is an ordered list of allocations which are free in the block.","items":{"type":"integer"},"type":"array"}},"required":["allocations","attributes","cidr","strictAffinity","unallocated"],"type":"object"}},"type":"object"}},"served":true,"storage":true}]},"status":{"acceptedNames":{"kind":"","plural":""},"conditions":[],"storedVersions":[]}} to: Resource: "apiextensions.k8s.io/v1, Resource=customresourcedefinitions", GroupVersionKind: "apiextensions.k8s.io/v1, Kind=CustomResourceDefinition" Name: "ipamblocks.crd.projectcalico.org", Namespace: "" for: "STDIN": CustomResourceDefinition.apiextensions.k8s.io "ipamblocks.crd.projectcalico.org" is invalid: spec.preserveUnknownFields: Invalid value: true: must be false in order to use defaults in the schema configmap/canal-config unchanged customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org configured customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org configured customresourcedefinition.apiextensions.k8s.io/blockaffinities.crd.projectcalico.org configured customresourcedefinition.apiextensions.k8s.io/caliconodestatuses.crd.projectcalico.org configured customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org configured customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org configured customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org configured customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org configured customresourcedefinition.apiextensions.k8s.io/ipamconfigs.crd.projectcalico.org configured customresourcedefinition.apiextensions.k8s.io/ipamhandles.crd.projectcalico.org configured customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org configured customresourcedefinition.apiextensions.k8s.io/ipreservations.crd.projectcalico.org configured customresourcedefinition.apiextensions.k8s.io/kubecontrollersconfigurations.crd.projectcalico.org configured customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org configured customresourcedefinition.apiextensions.k8s.io/networksets.crd.projectcalico.org configured clusterrole.rbac.authorization.k8s.io/calico-kube-controllers unchanged clusterrolebinding.rbac.authorization.k8s.io/calico-kube-controllers unchanged clusterrole.rbac.authorization.k8s.io/calico-node unchanged clusterrole.rbac.authorization.k8s.io/flannel unchanged clusterrolebinding.rbac.authorization.k8s.io/canal-flannel unchanged clusterrolebinding.rbac.authorization.k8s.io/canal-calico unchanged daemonset.apps/canal configured serviceaccount/canal unchanged deployment.apps/calico-kube-controllers unchanged serviceaccount/calico-kube-controllers unchanged poddisruptionbudget.policy/calico-kube-controllers unchanged time="09:06:53 UTC" level=warning msg="Retrying task..." time="09:06:54 UTC" level=info msg="Applying addon metrics-server..." time="09:06:54 UTC" level=info msg="Parsing addons manifest 'metrics-server.yaml'\n"

@kron4eg I just built from master and it still fails

$ kubeone version
{
  "kubeone": {
    "major": "1",
    "minor": "4",
    "gitVersion": "v1.4.0-103-g3ed4d8bd",
    "gitCommit": "3ed4d8bd749d11f088e30eaf5b0d15f7c6f9970a",
    "gitTreeState": "",
    "buildDate": "",
    "goVersion": "go1.18.3",
    "compiler": "gc",
    "platform": "linux/amd64"
  },
  "machine_controller": {
    "major": "1",
    "minor": "51",
    "gitVersion": "v1.51.0",
    "gitCommit": "",
    "gitTreeState": "",
    "buildDate": "",
    "goVersion": "",
    "compiler": "",
    "platform": "linux/amd64"
  }
}

$ kubeone apply --manifest kubeone.yaml -t tf.json -y -v
time="09:12:50 UTC" level=info msg="Determine hostname..."
[xxx.xxx.xxx.xxx] + export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/sbin:/usr/local/bin:/opt/bin
[xxx.xxx.xxx.xxx] + PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/sbin:/usr/local/bin:/opt/bin
[xxx.xxx.xxx.xxx] ++ hostname -f
[xxx.xxx.xxx.xxx] + fqdn=staging-control-plane-1
[xxx.xxx.xxx.xxx] + '[' staging-control-plane-1 = localhost ']'
[xxx.xxx.xxx.xxx] + echo staging-control-plane-1
time="09:12:54 UTC" level=debug msg="Hostname is detected: \"staging-control-plane-1\"" node=xxx.xxx.xxx.xxx
[xxx.xxx.xxx.xxx] staging-control-plane-1
[xxx.xxx.xxx.xxx] + export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/sbin:/usr/local/bin:/opt/bin
[xxx.xxx.xxx.xxx] + PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/sbin:/usr/local/bin:/opt/bin
[xxx.xxx.xxx.xxx] ++ hostname -f
[xxx.xxx.xxx.xxx] + fqdn=staging-control-plane-2
[xxx.xxx.xxx.xxx] + '[' staging-control-plane-2 = localhost ']'
[xxx.xxx.xxx.xxx] + echo staging-control-plane-2
time="09:12:54 UTC" level=debug msg="Hostname is detected: \"staging-control-plane-2\"" node=xxx.xxx.xxx.xxx
[xxx.xxx.xxx.xxx] staging-control-plane-2
[xxx.xxx.xxx.xxx] + export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/sbin:/usr/local/bin:/opt/bin
[xxx.xxx.xxx.xxx] + PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/sbin:/usr/local/bin:/opt/bin
[xxx.xxx.xxx.xxx] ++ hostname -f
[xxx.xxx.xxx.xxx] + fqdn=staging-control-plane-3
[xxx.xxx.xxx.xxx] + '[' staging-control-plane-3 = localhost ']'
[xxx.xxx.xxx.xxx] staging-control-plane-3
[xxx.xxx.xxx.xxx] + echo staging-control-plane-3
time="09:12:55 UTC" level=debug msg="Hostname is detected: \"staging-control-plane-3\"" node=xxx.xxx.xxx.xxx
time="09:12:55 UTC" level=info msg="Determine operating system..."
time="09:12:55 UTC" level=debug msg="Operating system detected: \"ubuntu\"" node=xxx.xxx.xxx.xxx
time="09:12:55 UTC" level=debug msg="Operating system detected: \"ubuntu\"" node=xxx.xxx.xxx.xxx
time="09:12:55 UTC" level=debug msg="Operating system detected: \"ubuntu\"" node=xxx.xxx.xxx.xxx
time="09:12:55 UTC" level=info msg="Running host probes..."
time="09:12:56 UTC" level=info msg="Electing cluster leader..."
time="09:12:57 UTC" level=info msg="Elected leader \"staging-control-plane-1\"..."
time="09:12:58 UTC" level=info msg="Building Kubernetes clientset..."
time="09:12:58 UTC" level=info msg="Running cluster probes..."
Host: "staging-control-plane-1"
    Host initialized: yes
    containerd healthy: yes (1.4.13)
    Kubelet healthy: yes (1.23.7)

    containerd is installed: yes
    containerd is running: yes
    containerd is active: yes
    containerd is restarting: no

    kubelet is installed: yes
    kubelet is running: yes
    kubelet is active: yes
    kubelet is restarting: no

Host: "staging-control-plane-2"
    Host initialized: yes
    containerd healthy: yes (1.4.13)
    Kubelet healthy: yes (1.23.7)

    containerd is installed: yes
    containerd is running: yes
    containerd is active: yes
    containerd is restarting: no

    kubelet is installed: yes
    kubelet is running: yes
    kubelet is active: yes
    kubelet is restarting: no

Host: "staging-control-plane-3"
    Host initialized: yes
    containerd healthy: yes (1.4.13)
    docker healthy: yes (20.10.14)
    Kubelet healthy: yes (1.23.7)

    containerd is installed: yes
    containerd is running: yes
    containerd is active: yes
    containerd is restarting: no

    docker is installed: yes
    docker is running: yes
    docker is active: yes
    docker is restarting: no

    kubelet is installed: yes
    kubelet is running: yes
    kubelet is active: yes
    kubelet is restarting: no

The following actions will be taken: 

    ~ ensure credential
    ~ ensure embedded addons
    ~ ensure CNI
    ~ ensure custom addons
    ~ ensure external CCM

time="09:12:58 UTC" level=info msg="Patching static pods..."
time="09:12:58 UTC" level=info msg="Patching static pods..."
time="09:12:58 UTC" level=info msg="Patching static pods..."
time="09:12:59 UTC" level=info msg="Downloading kubeconfig..."
time="09:12:59 UTC" level=info msg="Downloading PKI..."
time="09:12:59 UTC" level=info msg="Creating local backup..." node=xxx.xxx.xxx.xxx
time="09:12:59 UTC" level=info msg="Activating additional features..."
time="09:12:59 UTC" level=info msg="Patching coreDNS with uninitialized toleration..."
time="09:13:00 UTC" level=info msg="Creating machine-controller credentials secret..."
time="09:13:00 UTC" level=info msg="Creating CCM credentials secret..."
time="09:13:01 UTC" level=info msg="Applying addon metrics-server..."
time="09:13:01 UTC" level=info msg="Parsing addons manifest 'metrics-server.yaml'\n"
time="09:13:01 UTC" level=info msg="Parsing addons manifest 'poddisruptionbudget.yaml'\n"
+ sudo KUBECONFIG=/etc/kubernetes/admin.conf \
kubectl apply -f - --prune -l "kubeone.io/addon=metrics-server"

clusterrole.rbac.authorization.k8s.io/system:aggregated-metrics-reader unchanged
serviceaccount/metrics-server unchanged
rolebinding.rbac.authorization.k8s.io/metrics-server-auth-reader unchanged
clusterrolebinding.rbac.authorization.k8s.io/metrics-server:system:auth-delegator unchanged
clusterrole.rbac.authorization.k8s.io/system:metrics-server configured
clusterrolebinding.rbac.authorization.k8s.io/system:metrics-server unchanged
apiservice.apiregistration.k8s.io/v1beta1.metrics.k8s.io unchanged
secret/metrics-server-serving-cert configured
deployment.apps/metrics-server configured
service/metrics-server configured
poddisruptionbudget.policy/metrics-server created
time="09:13:03 UTC" level=info msg="Applying addon cni-canal..."
time="09:13:03 UTC" level=info msg="Skipping file \"README.md\" because it's not .yaml/.yml/.json file\n"
time="09:13:03 UTC" level=info msg="Parsing addons manifest 'canal.yaml'\n"
time="09:13:04 UTC" level=warning msg="Task failed, error was: runtime: running task on \"xxx.xxx.xxx.xxx\"\nssh: popen\nProcess exited with status 1"
+ sudo KUBECONFIG=/etc/kubernetes/admin.conf \
kubectl apply -f - --prune -l "kubeone.io/addon=cni-canal"

Warning: policy/v1beta1 PodDisruptionBudget is deprecated in v1.21+, unavailable in v1.25+; use policy/v1 PodDisruptionBudget
Error from server (Invalid): error when applying patch:
{"metadata":{"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"apiextensions.k8s.io/v1\",\"kind\":\"CustomResourceDefinition\",\"metadata\":{\"annotations\":{},\"labels\":{\"kubeone.io/addon\":\"cni-canal\"},\"name\":\"felixconfigurations.crd.projectcalico.org\"},\"spec\":{\"group\":\"crd.projectcalico.org\",\"names\":{\"kind\":\"FelixConfiguration\",\"listKind\":\"FelixConfigurationList\",\"plural\":\"felixconfigurations\",\"singular\":\"felixconfiguration\"},\"scope\":\"Cluster\",\"versions\":[{\"name\":\"v1\",\"schema\":{\"openAPIV3Schema\":{\"description\":\"Felix Configuration contains the configuration for Felix.\",\"properties\":{\"apiVersion\":{\"description\":\"APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources\",\"type\":\"string\"},\"kind\":{\"description\":\"Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds\",\"type\":\"string\"},\"metadata\":{\"type\":\"object\"},\"spec\":{\"description\":\"FelixConfigurationSpec contains the values of the Felix configuration.\",\"properties\":{\"allowIPIPPacketsFromWorkloads\":{\"description\":\"AllowIPIPPacketsFromWorkloads controls whether Felix will add a rule to drop IPIP encapsulated traffic from workloads [Default: false]\",\"type\":\"boolean\"},\"allowVXLANPacketsFromWorkloads\":{\"description\":\"AllowVXLANPacketsFromWorkloads controls whether Felix will add a rule to drop VXLAN encapsulated traffic from workloads [Default: false]\",\"type\":\"boolean\"},\"awsSrcDstCheck\":{\"description\":\"Set source-destination-check on AWS EC2 instances. Accepted value must be one of \\\"DoNothing\\\", \\\"Enable\\\" or \\\"Disable\\\". [Default: DoNothing]\",\"enum\":[\"DoNothing\",\"Enable\",\"Disable\"],\"type\":\"string\"},\"bpfConnectTimeLoadBalancingEnabled\":{\"description\":\"BPFConnectTimeLoadBalancingEnabled when in BPF mode, controls whether Felix installs the connection-time load balancer.  The connect-time load balancer is required for the host to be able to reach Kubernetes services and it improves the performance of pod-to-service connections.  The only reason to disable it is for debugging purposes.  [Default: true]\",\"type\":\"boolean\"},\"bpfDataIfacePattern\":{\"description\":\"BPFDataIfacePattern is a regular expression that controls which interfaces Felix should attach BPF programs to in order to catch traffic to/from the network.  This needs to match the interfaces that Calico workload traffic flows over as well as any interfaces that handle incoming traffic to nodeports and services from outside the cluster.  It should not match the workload interfaces (usually named cali...).\",\"type\":\"string\"},\"bpfDisableUnprivileged\":{\"description\":\"BPFDisableUnprivileged, if enabled, Felix sets the kernel.unprivileged_bpf_disabled sysctl to disable unprivileged use of BPF.  This ensures that unprivileged users cannot access Calico's BPF maps and cannot insert their own BPF programs to interfere with Calico's. [Default: true]\",\"type\":\"boolean\"},\"bpfEnabled\":{\"description\":\"BPFEnabled, if enabled Felix will use the BPF dataplane. [Default: false]\",\"type\":\"boolean\"},\"bpfExtToServiceConnmark\":{\"description\":\"BPFExtToServiceConnmark in BPF mode, control a 32bit mark that is set on connections from an external client to a local service. This mark allows us to control how packets of that connection are routed within the host and how is routing intepreted by RPF check. [Default: 0]\",\"type\":\"integer\"},\"bpfExternalServiceMode\":{\"description\":\"BPFExternalServiceMode in BPF mode, controls how connections from outside the cluster to services (node ports and cluster IPs) are forwarded to remote workloads.  If set to \\\"Tunnel\\\" then both request and response traffic is tunneled to the remote node.  If set to \\\"DSR\\\", the request traffic is tunneled but the response traffic is sent directly from the remote node.  In \\\"DSR\\\" mode, the remote node appears to use the IP of the ingress node; this requires a permissive L2 network.  [Default: Tunnel]\",\"type\":\"string\"},\"bpfKubeProxyEndpointSlicesEnabled\":{\"description\":\"BPFKubeProxyEndpointSlicesEnabled in BPF mode, controls whether Felix's embedded kube-proxy accepts EndpointSlices or not.\",\"type\":\"boolean\"},\"bpfKubeProxyIptablesCleanupEnabled\":{\"description\":\"BPFKubeProxyIptablesCleanupEnabled, if enabled in BPF mode, Felix will proactively clean up the upstream Kubernetes kube-proxy's iptables chains.  Should only be enabled if kube-proxy is not running.  [Default: true]\",\"type\":\"boolean\"},\"bpfKubeProxyMinSyncPeriod\":{\"description\":\"BPFKubeProxyMinSyncPeriod, in BPF mode, controls the minimum time between updates to the dataplane for Felix's embedded kube-proxy.  Lower values give reduced set-up latency.  Higher values reduce Felix CPU usage by batching up more work.  [Default: 1s]\",\"type\":\"string\"},\"bpfLogLevel\":{\"description\":\"BPFLogLevel controls the log level of the BPF programs when in BPF dataplane mode.  One of \\\"Off\\\", \\\"Info\\\", or \\\"Debug\\\".  The logs are emitted to the BPF trace pipe, accessible with the command `tc exec bpf debug`. [Default: Off].\",\"type\":\"string\"},\"bpfPSNATPorts\":{\"anyOf\":[{\"type\":\"integer\"},{\"type\":\"string\"}],\"description\":\"BPFPSNATPorts sets the range from which we randomly pick a port if there is a source port collision. This should be within the ephemeral range as defined by RFC 6056 (1024â€“65535) and preferably outside the  ephemeral ranges used by common operating systems. Linux uses 32768â€“60999, while others mostly use the IANA defined range 49152â€“65535. It is not necessarily a problem if this range overlaps with the operating systems. Both ends of the range are inclusive. [Default: 20000:29999]\",\"pattern\":\"^.*\",\"x-kubernetes-int-or-string\":true},\"chainInsertMode\":{\"description\":\"ChainInsertMode controls whether Felix hooks the kernel's top-level iptables chains by inserting a rule at the top of the chain or by appending a rule at the bottom. insert is the safe default since it prevents Calico's rules from being bypassed. If you switch to append mode, be sure that the other rules in the chains signal acceptance by falling through to the Calico rules, otherwise the Calico policy will be bypassed. [Default: insert]\",\"type\":\"string\"},\"dataplaneDriver\":{\"type\":\"string\"},\"debugDisableLogDropping\":{\"type\":\"boolean\"},\"debugMemoryProfilePath\":{\"type\":\"string\"},\"debugSimulateCalcGraphHangAfter\":{\"type\":\"string\"},\"debugSimulateDataplaneHangAfter\":{\"type\":\"string\"},\"defaultEndpointToHostAction\":{\"description\":\"DefaultEndpointToHostAction controls what happens to traffic that goes from a workload endpoint to the host itself (after the traffic hits the endpoint egress policy). By default Calico blocks traffic from workload endpoints to the host itself with an iptables \\\"DROP\\\" action. If you want to allow some or all traffic from endpoint to host, set this parameter to RETURN or ACCEPT. Use RETURN if you have your own rules in the iptables \\\"INPUT\\\" chain; Calico will insert its rules at the top of that chain, then \\\"RETURN\\\" packets to the \\\"INPUT\\\" chain once it has completed processing workload endpoint egress policy. Use ACCEPT to unconditionally accept packets from workloads after processing workload endpoint egress policy. [Default: Drop]\",\"type\":\"string\"},\"deviceRouteProtocol\":{\"description\":\"This defines the route protocol added to programmed device routes, by default this will be RTPROT_BOOT when left blank.\",\"type\":\"integer\"},\"deviceRouteSourceAddress\":{\"description\":\"This is the source address to use on programmed device routes. By default the source address is left blank, leaving the kernel to choose the source address used.\",\"type\":\"string\"},\"disableConntrackInvalidCheck\":{\"type\":\"boolean\"},\"endpointReportingDelay\":{\"type\":\"string\"},\"endpointReportingEnabled\":{\"type\":\"boolean\"},\"externalNodesList\":{\"description\":\"ExternalNodesCIDRList is a list of CIDR's of external-non-calico-nodes which may source tunnel traffic and have the tunneled traffic be accepted at calico nodes.\",\"items\":{\"type\":\"string\"},\"type\":\"array\"},\"failsafeInboundHostPorts\":{\"description\":\"FailsafeInboundHostPorts is a list of UDP/TCP ports and CIDRs that Felix will allow incoming traffic to host endpoints on irrespective of the security policy. This is useful to avoid accidentally cutting off a host with incorrect configuration. For back-compatibility, if the protocol is not specified, it defaults to \\\"tcp\\\". If a CIDR is not specified, it will allow traffic from all addresses. To disable all inbound host ports, use the value none. The default value allows ssh access and DHCP. [Default: tcp:22, udp:68, tcp:179, tcp:2379, tcp:2380, tcp:6443, tcp:6666, tcp:6667]\",\"items\":{\"description\":\"ProtoPort is combination of protocol, port, and CIDR. Protocol and port must be specified.\",\"properties\":{\"net\":{\"type\":\"string\"},\"port\":{\"type\":\"integer\"},\"protocol\":{\"type\":\"string\"}},\"required\":[\"port\",\"protocol\"],\"type\":\"object\"},\"type\":\"array\"},\"failsafeOutboundHostPorts\":{\"description\":\"FailsafeOutboundHostPorts is a list of UDP/TCP ports and CIDRs that Felix will allow outgoing traffic from host endpoints to irrespective of the security policy. This is useful to avoid accidentally cutting off a host with incorrect configuration. For back-compatibility, if the protocol is not specified, it defaults to \\\"tcp\\\". If a CIDR is not specified, it will allow traffic from all addresses. To disable all outbound host ports, use the value none. The default value opens etcd's standard ports to ensure that Felix does not get cut off from etcd as well as allowing DHCP and DNS. [Default: tcp:179, tcp:2379, tcp:2380, tcp:6443, tcp:6666, tcp:6667, udp:53, udp:67]\",\"items\":{\"description\":\"ProtoPort is combination of protocol, port, and CIDR. Protocol and port must be specified.\",\"properties\":{\"net\":{\"type\":\"string\"},\"port\":{\"type\":\"integer\"},\"protocol\":{\"type\":\"string\"}},\"required\":[\"port\",\"protocol\"],\"type\":\"object\"},\"type\":\"array\"},\"featureDetectOverride\":{\"description\":\"FeatureDetectOverride is used to override the feature detection. Values are specified in a comma separated list with no spaces, example; \\\"SNATFullyRandom=true,MASQFullyRandom=false,RestoreSupportsLock=\\\". \\\"true\\\" or \\\"false\\\" will force the feature, empty or omitted values are auto-detected.\",\"type\":\"string\"},\"floatingIPs\":{\"default\":\"Disabled\",\"description\":\"FloatingIPs configures whether or not Felix will program floating IP addresses.\",\"enum\":[\"Enabled\",\"Disabled\"],\"type\":\"string\"},\"genericXDPEnabled\":{\"description\":\"GenericXDPEnabled enables Generic XDP so network cards that don't support XDP offload or driver modes can use XDP. This is not recommended since it doesn't provide better performance than iptables. [Default: false]\",\"type\":\"boolean\"},\"healthEnabled\":{\"type\":\"boolean\"},\"healthHost\":{\"type\":\"string\"},\"healthPort\":{\"type\":\"integer\"},\"interfaceExclude\":{\"description\":\"InterfaceExclude is a comma-separated list of interfaces that Felix should exclude when monitoring for host endpoints. The default value ensures that Felix ignores Kubernetes' IPVS dummy interface, which is used internally by kube-proxy. If you want to exclude multiple interface names using a single value, the list supports regular expressions. For regular expressions you must wrap the value with '/'. For example having values '/^kube/,veth1' will exclude all interfaces that begin with 'kube' and also the interface 'veth1'. [Default: kube-ipvs0]\",\"type\":\"string\"},\"interfacePrefix\":{\"description\":\"InterfacePrefix is the interface name prefix that identifies workload endpoints and so distinguishes them from host endpoint interfaces. Note: in environments other than bare metal, the orchestrators configure this appropriately. For example our Kubernetes and Docker integrations set the 'cali' value, and our OpenStack integration sets the 'tap' value. [Default: cali]\",\"type\":\"string\"},\"interfaceRefreshInterval\":{\"description\":\"InterfaceRefreshInterval is the period at which Felix rescans local interfaces to verify their state. The rescan can be disabled by setting the interval to 0.\",\"type\":\"string\"},\"ipipEnabled\":{\"type\":\"boolean\"},\"ipipMTU\":{\"description\":\"IPIPMTU is the MTU to set on the tunnel device. See Configuring MTU [Default: 1440]\",\"type\":\"integer\"},\"ipsetsRefreshInterval\":{\"description\":\"IpsetsRefreshInterval is the period at which Felix re-checks all iptables state to ensure that no other process has accidentally broken Calico's rules. Set to 0 to disable iptables refresh. [Default: 90s]\",\"type\":\"string\"},\"iptablesBackend\":{\"description\":\"IptablesBackend specifies which backend of iptables will be used. The default is legacy.\",\"type\":\"string\"},\"iptablesFilterAllowAction\":{\"type\":\"string\"},\"iptablesLockFilePath\":{\"description\":\"IptablesLockFilePath is the location of the iptables lock file. You may need to change this if the lock file is not in its standard location (for example if you have mapped it into Felix's container at a different path). [Default: /run/xtables.lock]\",\"type\":\"string\"},\"iptablesLockProbeInterval\":{\"description\":\"IptablesLockProbeInterval is the time that Felix will wait between attempts to acquire the iptables lock if it is not available. Lower values make Felix more responsive when the lock is contended, but use more CPU. [Default: 50ms]\",\"type\":\"string\"},\"iptablesLockTimeout\":{\"description\":\"IptablesLockTimeout is the time that Felix will wait for the iptables lock, or 0, to disable. To use this feature, Felix must share the iptables lock file with all other processes that also take the lock. When running Felix inside a container, this requires the /run directory of the host to be mounted into the calico/node or calico/felix container. [Default: 0s disabled]\",\"type\":\"string\"},\"iptablesMangleAllowAction\":{\"type\":\"string\"},\"iptablesMarkMask\":{\"description\":\"IptablesMarkMask is the mask that Felix selects its IPTables Mark bits from. Should be a 32 bit hexadecimal number with at least 8 bits set, none of which clash with any other mark bits in use on the system. [Default: 0xff000000]\",\"format\":\"int32\",\"type\":\"integer\"},\"iptablesNATOutgoingInterfaceFilter\":{\"type\":\"string\"},\"iptablesPostWriteCheckInterval\":{\"description\":\"IptablesPostWriteCheckInterval is the period after Felix has done a write to the dataplane that it schedules an extra read back in order to check the write was not clobbered by another process. This should only occur if another application on the system doesn't respect the iptables lock. [Default: 1s]\",\"type\":\"string\"},\"iptablesRefreshInterval\":{\"description\":\"IptablesRefreshInterval is the period at which Felix re-checks the IP sets in the dataplane to ensure that no other process has accidentally broken Calico's rules. Set to 0 to disable IP sets refresh. Note: the default for this value is lower than the other refresh intervals as a workaround for a Linux kernel bug that was fixed in kernel version 4.11. If you are using v4.11 or greater you may want to set this to, a higher value to reduce Felix CPU usage. [Default: 10s]\",\"type\":\"string\"},\"ipv6Support\":{\"type\":\"boolean\"},\"kubeNodePortRanges\":{\"description\":\"KubeNodePortRanges holds list of port ranges used for service node ports. Only used if felix detects kube-proxy running in ipvs mode. Felix uses these ranges to separate host and workload traffic. [Default: 30000:32767].\",\"items\":{\"anyOf\":[{\"type\":\"integer\"},{\"type\":\"string\"}],\"pattern\":\"^.*\",\"x-kubernetes-int-or-string\":true},\"type\":\"array\"},\"logDebugFilenameRegex\":{\"description\":\"LogDebugFilenameRegex controls which source code files have their Debug log output included in the logs. Only logs from files with names that match the given regular expression are included.  The filter only applies to Debug level logs.\",\"type\":\"string\"},\"logFilePath\":{\"description\":\"LogFilePath is the full path to the Felix log. Set to none to disable file logging. [Default: /var/log/calico/felix.log]\",\"type\":\"string\"},\"logPrefix\":{\"description\":\"LogPrefix is the log prefix that Felix uses when rendering LOG rules. [Default: calico-packet]\",\"type\":\"string\"},\"logSeverityFile\":{\"description\":\"LogSeverityFile is the log severity above which logs are sent to the log file. [Default: Info]\",\"type\":\"string\"},\"logSeverityScreen\":{\"description\":\"LogSeverityScreen is the log severity above which logs are sent to the stdout. [Default: Info]\",\"type\":\"string\"},\"logSeveritySys\":{\"description\":\"LogSeveritySys is the log severity above which logs are sent to the syslog. Set to None for no logging to syslog. [Default: Info]\",\"type\":\"string\"},\"maxIpsetSize\":{\"type\":\"integer\"},\"metadataAddr\":{\"description\":\"MetadataAddr is the IP address or domain name of the server that can answer VM queries for cloud-init metadata. In OpenStack, this corresponds to the machine running nova-api (or in Ubuntu, nova-api-metadata). A value of none (case insensitive) means that Felix should not set up any NAT rule for the metadata path. [Default: 127.0.0.1]\",\"type\":\"string\"},\"metadataPort\":{\"description\":\"MetadataPort is the port of the metadata server. This, combined with global.MetadataAddr (if not 'None'), is used to set up a NAT rule, from 169.254.169.254:80 to MetadataAddr:MetadataPort. In most cases this should not need to be changed [Default: 8775].\",\"type\":\"integer\"},\"mtuIfacePattern\":{\"description\":\"MTUIfacePattern is a regular expression that controls which interfaces Felix should scan in order to calculate the host's MTU. This should not match workload interfaces (usually named cali...).\",\"type\":\"string\"},\"natOutgoingAddress\":{\"description\":\"NATOutgoingAddress specifies an address to use when performing source NAT for traffic in a natOutgoing pool that is leaving the network. By default the address used is an address on the interface the traffic is leaving on (ie it uses the iptables MASQUERADE target)\",\"type\":\"string\"},\"natPortRange\":{\"anyOf\":[{\"type\":\"integer\"},{\"type\":\"string\"}],\"description\":\"NATPortRange specifies the range of ports that is used for port mapping when doing outgoing NAT. When unset the default behavior of the network stack is used.\",\"pattern\":\"^.*\",\"x-kubernetes-int-or-string\":true},\"netlinkTimeout\":{\"type\":\"string\"},\"openstackRegion\":{\"description\":\"OpenstackRegion is the name of the region that a particular Felix belongs to. In a multi-region Calico/OpenStack deployment, this must be configured somehow for each Felix (here in the datamodel, or in felix.cfg or the environment on each compute node), and must match the [calico] openstack_region value configured in neutron.conf on each node. [Default: Empty]\",\"type\":\"string\"},\"policySyncPathPrefix\":{\"description\":\"PolicySyncPathPrefix is used to by Felix to communicate policy changes to external services, like Application layer policy. [Default: Empty]\",\"type\":\"string\"},\"prometheusGoMetricsEnabled\":{\"description\":\"PrometheusGoMetricsEnabled disables Go runtime metrics collection, which the Prometheus client does by default, when set to false. This reduces the number of metrics reported, reducing Prometheus load. [Default: true]\",\"type\":\"boolean\"},\"prometheusMetricsEnabled\":{\"description\":\"PrometheusMetricsEnabled enables the Prometheus metrics server in Felix if set to true. [Default: false]\",\"type\":\"boolean\"},\"prometheusMetricsHost\":{\"description\":\"PrometheusMetricsHost is the host that the Prometheus metrics server should bind to. [Default: empty]\",\"type\":\"string\"},\"prometheusMetricsPort\":{\"description\":\"PrometheusMetricsPort is the TCP port that the Prometheus metrics server should bind to. [Default: 9091]\",\"type\":\"integer\"},\"prometheusProcessMetricsEnabled\":{\"description\":\"PrometheusProcessMetricsEnabled disables process metrics collection, which the Prometheus client does by default, when set to false. This reduces the number of metrics reported, reducing Prometheus load. [Default: true]\",\"type\":\"boolean\"},\"prometheusWireGuardMetricsEnabled\":{\"description\":\"PrometheusWireGuardMetricsEnabled disables wireguard metrics collection, which the Prometheus client does by default, when set to false. This reduces the number of metrics reported, reducing Prometheus load. [Default: true]\",\"type\":\"boolean\"},\"removeExternalRoutes\":{\"description\":\"Whether or not to remove device routes that have not been programmed by Felix. Disabling this will allow external applications to also add device routes. This is enabled by default which means we will remove externally added routes.\",\"type\":\"boolean\"},\"reportingInterval\":{\"description\":\"ReportingInterval is the interval at which Felix reports its status into the datastore or 0 to disable. Must be non-zero in OpenStack deployments. [Default: 30s]\",\"type\":\"string\"},\"reportingTTL\":{\"description\":\"ReportingTTL is the time-to-live setting for process-wide status reports. [Default: 90s]\",\"type\":\"string\"},\"routeRefreshInterval\":{\"description\":\"RouteRefreshInterval is the period at which Felix re-checks the routes in the dataplane to ensure that no other process has accidentally broken Calico's rules. Set to 0 to disable route refresh. [Default: 90s]\",\"type\":\"string\"},\"routeSource\":{\"description\":\"RouteSource configures where Felix gets its routing information. - WorkloadIPs: use workload endpoints to construct routes. - CalicoIPAM: the default - use IPAM data to construct routes.\",\"type\":\"string\"},\"routeTableRange\":{\"description\":\"Calico programs additional Linux route tables for various purposes.  RouteTableRange specifies the indices of the route tables that Calico should use.\",\"properties\":{\"max\":{\"type\":\"integer\"},\"min\":{\"type\":\"integer\"}},\"required\":[\"max\",\"min\"],\"type\":\"object\"},\"serviceLoopPrevention\":{\"description\":\"When service IP advertisement is enabled, prevent routing loops to service IPs that are not in use, by dropping or rejecting packets that do not get DNAT'd by kube-proxy. Unless set to \\\"Disabled\\\", in which case such routing loops continue to be allowed. [Default: Drop]\",\"type\":\"string\"},\"sidecarAccelerationEnabled\":{\"description\":\"SidecarAccelerationEnabled enables experimental sidecar acceleration [Default: false]\",\"type\":\"boolean\"},\"usageReportingEnabled\":{\"description\":\"UsageReportingEnabled reports anonymous Calico version number and cluster size to projectcalico.org. Logs warnings returned by the usage server. For example, if a significant security vulnerability has been discovered in the version of Calico being used. [Default: true]\",\"type\":\"boolean\"},\"usageReportingInitialDelay\":{\"description\":\"UsageReportingInitialDelay controls the minimum delay before Felix makes a report. [Default: 300s]\",\"type\":\"string\"},\"usageReportingInterval\":{\"description\":\"UsageReportingInterval controls the interval at which Felix makes reports. [Default: 86400s]\",\"type\":\"string\"},\"useInternalDataplaneDriver\":{\"type\":\"boolean\"},\"vxlanEnabled\":{\"type\":\"boolean\"},\"vxlanMTU\":{\"description\":\"VXLANMTU is the MTU to set on the tunnel device. See Configuring MTU [Default: 1440]\",\"type\":\"integer\"},\"vxlanPort\":{\"type\":\"integer\"},\"vxlanVNI\":{\"type\":\"integer\"},\"wireguardEnabled\":{\"description\":\"WireguardEnabled controls whether Wireguard is enabled. [Default: false]\",\"type\":\"boolean\"},\"wireguardHostEncryptionEnabled\":{\"description\":\"WireguardHostEncryptionEnabled controls whether Wireguard host-to-host encryption is enabled. [Default: false]\",\"type\":\"boolean\"},\"wireguardInterfaceName\":{\"description\":\"WireguardInterfaceName specifies the name to use for the Wireguard interface. [Default: wg.calico]\",\"type\":\"string\"},\"wireguardListeningPort\":{\"description\":\"WireguardListeningPort controls the listening port used by Wireguard. [Default: 51820]\",\"type\":\"integer\"},\"wireguardMTU\":{\"description\":\"WireguardMTU controls the MTU on the Wireguard interface. See Configuring MTU [Default: 1420]\",\"type\":\"integer\"},\"wireguardRoutingRulePriority\":{\"description\":\"WireguardRoutingRulePriority controls the priority value to use for the Wireguard routing rule. [Default: 99]\",\"type\":\"integer\"},\"xdpEnabled\":{\"description\":\"XDPEnabled enables XDP acceleration for suitable untracked incoming deny rules. [Default: true]\",\"type\":\"boolean\"},\"xdpRefreshInterval\":{\"description\":\"XDPRefreshInterval is the period at which Felix re-checks all XDP state to ensure that no other process has accidentally broken Calico's BPF maps or attached programs. Set to 0 to disable XDP refresh. [Default: 90s]\",\"type\":\"string\"}},\"type\":\"object\"}},\"type\":\"object\"}},\"served\":true,\"storage\":true}]},\"status\":{\"acceptedNames\":{\"kind\":\"\",\"plural\":\"\"},\"conditions\":[],\"storedVersions\":[]}}\n"}},"spec":{"versions":[{"name":"v1","schema":{"openAPIV3Schema":{"description":"Felix Configuration contains the configuration for Felix.","properties":{"apiVersion":{"description":"APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources","type":"string"},"kind":{"description":"Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds","type":"string"},"metadata":{"type":"object"},"spec":{"description":"FelixConfigurationSpec contains the values of the Felix configuration.","properties":{"allowIPIPPacketsFromWorkloads":{"description":"AllowIPIPPacketsFromWorkloads controls whether Felix will add a rule to drop IPIP encapsulated traffic from workloads [Default: false]","type":"boolean"},"allowVXLANPacketsFromWorkloads":{"description":"AllowVXLANPacketsFromWorkloads controls whether Felix will add a rule to drop VXLAN encapsulated traffic from workloads [Default: false]","type":"boolean"},"awsSrcDstCheck":{"description":"Set source-destination-check on AWS EC2 instances. Accepted value must be one of \"DoNothing\", \"Enable\" or \"Disable\". [Default: DoNothing]","enum":["DoNothing","Enable","Disable"],"type":"string"},"bpfConnectTimeLoadBalancingEnabled":{"description":"BPFConnectTimeLoadBalancingEnabled when in BPF mode, controls whether Felix installs the connection-time load balancer.  The connect-time load balancer is required for the host to be able to reach Kubernetes services and it improves the performance of pod-to-service connections.  The only reason to disable it is for debugging purposes.  [Default: true]","type":"boolean"},"bpfDataIfacePattern":{"description":"BPFDataIfacePattern is a regular expression that controls which interfaces Felix should attach BPF programs to in order to catch traffic to/from the network.  This needs to match the interfaces that Calico workload traffic flows over as well as any interfaces that handle incoming traffic to nodeports and services from outside the cluster.  It should not match the workload interfaces (usually named cali...).","type":"string"},"bpfDisableUnprivileged":{"description":"BPFDisableUnprivileged, if enabled, Felix sets the kernel.unprivileged_bpf_disabled sysctl to disable unprivileged use of BPF.  This ensures that unprivileged users cannot access Calico's BPF maps and cannot insert their own BPF programs to interfere with Calico's. [Default: true]","type":"boolean"},"bpfEnabled":{"description":"BPFEnabled, if enabled Felix will use the BPF dataplane. [Default: false]","type":"boolean"},"bpfExtToServiceConnmark":{"description":"BPFExtToServiceConnmark in BPF mode, control a 32bit mark that is set on connections from an external client to a local service. This mark allows us to control how packets of that connection are routed within the host and how is routing intepreted by RPF check. [Default: 0]","type":"integer"},"bpfExternalServiceMode":{"description":"BPFExternalServiceMode in BPF mode, controls how connections from outside the cluster to services (node ports and cluster IPs) are forwarded to remote workloads.  If set to \"Tunnel\" then both request and response traffic is tunneled to the remote node.  If set to \"DSR\", the request traffic is tunneled but the response traffic is sent directly from the remote node.  In \"DSR\" mode, the remote node appears to use the IP of the ingress node; this requires a permissive L2 network.  [Default: Tunnel]","type":"string"},"bpfKubeProxyEndpointSlicesEnabled":{"description":"BPFKubeProxyEndpointSlicesEnabled in BPF mode, controls whether Felix's embedded kube-proxy accepts EndpointSlices or not.","type":"boolean"},"bpfKubeProxyIptablesCleanupEnabled":{"description":"BPFKubeProxyIptablesCleanupEnabled, if enabled in BPF mode, Felix will proactively clean up the upstream Kubernetes kube-proxy's iptables chains.  Should only be enabled if kube-proxy is not running.  [Default: true]","type":"boolean"},"bpfKubeProxyMinSyncPeriod":{"description":"BPFKubeProxyMinSyncPeriod, in BPF mode, controls the minimum time between updates to the dataplane for Felix's embedded kube-proxy.  Lower values give reduced set-up latency.  Higher values reduce Felix CPU usage by batching up more work.  [Default: 1s]","type":"string"},"bpfLogLevel":{"description":"BPFLogLevel controls the log level of the BPF programs when in BPF dataplane mode.  One of \"Off\", \"Info\", or \"Debug\".  The logs are emitted to the BPF trace pipe, accessible with the command `tc exec bpf debug`. [Default: Off].","type":"string"},"bpfPSNATPorts":{"anyOf":[{"type":"integer"},{"type":"string"}],"description":"BPFPSNATPorts sets the range from which we randomly pick a port if there is a source port collision. This should be within the ephemeral range as defined by RFC 6056 (1024â€“65535) and preferably outside the  ephemeral ranges used by common operating systems. Linux uses 32768â€“60999, while others mostly use the IANA defined range 49152â€“65535. It is not necessarily a problem if this range overlaps with the operating systems. Both ends of the range are inclusive. [Default: 20000:29999]","pattern":"^.*","x-kubernetes-int-or-string":true},"chainInsertMode":{"description":"ChainInsertMode controls whether Felix hooks the kernel's top-level iptables chains by inserting a rule at the top of the chain or by appending a rule at the bottom. insert is the safe default since it prevents Calico's rules from being bypassed. If you switch to append mode, be sure that the other rules in the chains signal acceptance by falling through to the Calico rules, otherwise the Calico policy will be bypassed. [Default: insert]","type":"string"},"dataplaneDriver":{"type":"string"},"debugDisableLogDropping":{"type":"boolean"},"debugMemoryProfilePath":{"type":"string"},"debugSimulateCalcGraphHangAfter":{"type":"string"},"debugSimulateDataplaneHangAfter":{"type":"string"},"defaultEndpointToHostAction":{"description":"DefaultEndpointToHostAction controls what happens to traffic that goes from a workload endpoint to the host itself (after the traffic hits the endpoint egress policy). By default Calico blocks traffic from workload endpoints to the host itself with an iptables \"DROP\" action. If you want to allow some or all traffic from endpoint to host, set this parameter to RETURN or ACCEPT. Use RETURN if you have your own rules in the iptables \"INPUT\" chain; Calico will insert its rules at the top of that chain, then \"RETURN\" packets to the \"INPUT\" chain once it has completed processing workload endpoint egress policy. Use ACCEPT to unconditionally accept packets from workloads after processing workload endpoint egress policy. [Default: Drop]","type":"string"},"deviceRouteProtocol":{"description":"This defines the route protocol added to programmed device routes, by default this will be RTPROT_BOOT when left blank.","type":"integer"},"deviceRouteSourceAddress":{"description":"This is the source address to use on programmed device routes. By default the source address is left blank, leaving the kernel to choose the source address used.","type":"string"},"disableConntrackInvalidCheck":{"type":"boolean"},"endpointReportingDelay":{"type":"string"},"endpointReportingEnabled":{"type":"boolean"},"externalNodesList":{"description":"ExternalNodesCIDRList is a list of CIDR's of external-non-calico-nodes which may source tunnel traffic and have the tunneled traffic be accepted at calico nodes.","items":{"type":"string"},"type":"array"},"failsafeInboundHostPorts":{"description":"FailsafeInboundHostPorts is a list of UDP/TCP ports and CIDRs that Felix will allow incoming traffic to host endpoints on irrespective of the security policy. This is useful to avoid accidentally cutting off a host with incorrect configuration. For back-compatibility, if the protocol is not specified, it defaults to \"tcp\". If a CIDR is not specified, it will allow traffic from all addresses. To disable all inbound host ports, use the value none. The default value allows ssh access and DHCP. [Default: tcp:22, udp:68, tcp:179, tcp:2379, tcp:2380, tcp:6443, tcp:6666, tcp:6667]","items":{"description":"ProtoPort is combination of protocol, port, and CIDR. Protocol and port must be specified.","properties":{"net":{"type":"string"},"port":{"type":"integer"},"protocol":{"type":"string"}},"required":["port","protocol"],"type":"object"},"type":"array"},"failsafeOutboundHostPorts":{"description":"FailsafeOutboundHostPorts is a list of UDP/TCP ports and CIDRs that Felix will allow outgoing traffic from host endpoints to irrespective of the security policy. This is useful to avoid accidentally cutting off a host with incorrect configuration. For back-compatibility, if the protocol is not specified, it defaults to \"tcp\". If a CIDR is not specified, it will allow traffic from all addresses. To disable all outbound host ports, use the value none. The default value opens etcd's standard ports to ensure that Felix does not get cut off from etcd as well as allowing DHCP and DNS. [Default: tcp:179, tcp:2379, tcp:2380, tcp:6443, tcp:6666, tcp:6667, udp:53, udp:67]","items":{"description":"ProtoPort is combination of protocol, port, and CIDR. Protocol and port must be specified.","properties":{"net":{"type":"string"},"port":{"type":"integer"},"protocol":{"type":"string"}},"required":["port","protocol"],"type":"object"},"type":"array"},"featureDetectOverride":{"description":"FeatureDetectOverride is used to override the feature detection. Values are specified in a comma separated list with no spaces, example; \"SNATFullyRandom=true,MASQFullyRandom=false,RestoreSupportsLock=\". \"true\" or \"false\" will force the feature, empty or omitted values are auto-detected.","type":"string"},"floatingIPs":{"default":"Disabled","description":"FloatingIPs configures whether or not Felix will program floating IP addresses.","enum":["Enabled","Disabled"],"type":"string"},"genericXDPEnabled":{"description":"GenericXDPEnabled enables Generic XDP so network cards that don't support XDP offload or driver modes can use XDP. This is not recommended since it doesn't provide better performance than iptables. [Default: false]","type":"boolean"},"healthEnabled":{"type":"boolean"},"healthHost":{"type":"string"},"healthPort":{"type":"integer"},"interfaceExclude":{"description":"InterfaceExclude is a comma-separated list of interfaces that Felix should exclude when monitoring for host endpoints. The default value ensures that Felix ignores Kubernetes' IPVS dummy interface, which is used internally by kube-proxy. If you want to exclude multiple interface names using a single value, the list supports regular expressions. For regular expressions you must wrap the value with '/'. For example having values '/^kube/,veth1' will exclude all interfaces that begin with 'kube' and also the interface 'veth1'. [Default: kube-ipvs0]","type":"string"},"interfacePrefix":{"description":"InterfacePrefix is the interface name prefix that identifies workload endpoints and so distinguishes them from host endpoint interfaces. Note: in environments other than bare metal, the orchestrators configure this appropriately. For example our Kubernetes and Docker integrations set the 'cali' value, and our OpenStack integration sets the 'tap' value. [Default: cali]","type":"string"},"interfaceRefreshInterval":{"description":"InterfaceRefreshInterval is the period at which Felix rescans local interfaces to verify their state. The rescan can be disabled by setting the interval to 0.","type":"string"},"ipipEnabled":{"type":"boolean"},"ipipMTU":{"description":"IPIPMTU is the MTU to set on the tunnel device. See Configuring MTU [Default: 1440]","type":"integer"},"ipsetsRefreshInterval":{"description":"IpsetsRefreshInterval is the period at which Felix re-checks all iptables state to ensure that no other process has accidentally broken Calico's rules. Set to 0 to disable iptables refresh. [Default: 90s]","type":"string"},"iptablesBackend":{"description":"IptablesBackend specifies which backend of iptables will be used. The default is legacy.","type":"string"},"iptablesFilterAllowAction":{"type":"string"},"iptablesLockFilePath":{"description":"IptablesLockFilePath is the location of the iptables lock file. You may need to change this if the lock file is not in its standard location (for example if you have mapped it into Felix's container at a different path). [Default: /run/xtables.lock]","type":"string"},"iptablesLockProbeInterval":{"description":"IptablesLockProbeInterval is the time that Felix will wait between attempts to acquire the iptables lock if it is not available. Lower values make Felix more responsive when the lock is contended, but use more CPU. [Default: 50ms]","type":"string"},"iptablesLockTimeout":{"description":"IptablesLockTimeout is the time that Felix will wait for the iptables lock, or 0, to disable. To use this feature, Felix must share the iptables lock file with all other processes that also take the lock. When running Felix inside a container, this requires the /run directory of the host to be mounted into the calico/node or calico/felix container. [Default: 0s disabled]","type":"string"},"iptablesMangleAllowAction":{"type":"string"},"iptablesMarkMask":{"description":"IptablesMarkMask is the mask that Felix selects its IPTables Mark bits from. Should be a 32 bit hexadecimal number with at least 8 bits set, none of which clash with any other mark bits in use on the system. [Default: 0xff000000]","format":"int32","type":"integer"},"iptablesNATOutgoingInterfaceFilter":{"type":"string"},"iptablesPostWriteCheckInterval":{"description":"IptablesPostWriteCheckInterval is the period after Felix has done a write to the dataplane that it schedules an extra read back in order to check the write was not clobbered by another process. This should only occur if another application on the system doesn't respect the iptables lock. [Default: 1s]","type":"string"},"iptablesRefreshInterval":{"description":"IptablesRefreshInterval is the period at which Felix re-checks the IP sets in the dataplane to ensure that no other process has accidentally broken Calico's rules. Set to 0 to disable IP sets refresh. Note: the default for this value is lower than the other refresh intervals as a workaround for a Linux kernel bug that was fixed in kernel version 4.11. If you are using v4.11 or greater you may want to set this to, a higher value to reduce Felix CPU usage. [Default: 10s]","type":"string"},"ipv6Support":{"type":"boolean"},"kubeNodePortRanges":{"description":"KubeNodePortRanges holds list of port ranges used for service node ports. Only used if felix detects kube-proxy running in ipvs mode. Felix uses these ranges to separate host and workload traffic. [Default: 30000:32767].","items":{"anyOf":[{"type":"integer"},{"type":"string"}],"pattern":"^.*","x-kubernetes-int-or-string":true},"type":"array"},"logDebugFilenameRegex":{"description":"LogDebugFilenameRegex controls which source code files have their Debug log output included in the logs. Only logs from files with names that match the given regular expression are included.  The filter only applies to Debug level logs.","type":"string"},"logFilePath":{"description":"LogFilePath is the full path to the Felix log. Set to none to disable file logging. [Default: /var/log/calico/felix.log]","type":"string"},"logPrefix":{"description":"LogPrefix is the log prefix that Felix uses when rendering LOG rules. [Default: calico-packet]","type":"string"},"logSeverityFile":{"description":"LogSeverityFile is the log severity above which logs are sent to the log file. [Default: Info]","type":"string"},"logSeverityScreen":{"description":"LogSeverityScreen is the log severity above which logs are sent to the stdout. [Default: Info]","type":"string"},"logSeveritySys":{"description":"LogSeveritySys is the log severity above which logs are sent to the syslog. Set to None for no logging to syslog. [Default: Info]","type":"string"},"maxIpsetSize":{"type":"integer"},"metadataAddr":{"description":"MetadataAddr is the IP address or domain name of the server that can answer VM queries for cloud-init metadata. In OpenStack, this corresponds to the machine running nova-api (or in Ubuntu, nova-api-metadata). A value of none (case insensitive) means that Felix should not set up any NAT rule for the metadata path. [Default: 127.0.0.1]","type":"string"},"metadataPort":{"description":"MetadataPort is the port of the metadata server. This, combined with global.MetadataAddr (if not 'None'), is used to set up a NAT rule, from 169.254.169.254:80 to MetadataAddr:MetadataPort. In most cases this should not need to be changed [Default: 8775].","type":"integer"},"mtuIfacePattern":{"description":"MTUIfacePattern is a regular expression that controls which interfaces Felix should scan in order to calculate the host's MTU. This should not match workload interfaces (usually named cali...).","type":"string"},"natOutgoingAddress":{"description":"NATOutgoingAddress specifies an address to use when performing source NAT for traffic in a natOutgoing pool that is leaving the network. By default the address used is an address on the interface the traffic is leaving on (ie it uses the iptables MASQUERADE target)","type":"string"},"natPortRange":{"anyOf":[{"type":"integer"},{"type":"string"}],"description":"NATPortRange specifies the range of ports that is used for port mapping when doing outgoing NAT. When unset the default behavior of the network stack is used.","pattern":"^.*","x-kubernetes-int-or-string":true},"netlinkTimeout":{"type":"string"},"openstackRegion":{"description":"OpenstackRegion is the name of the region that a particular Felix belongs to. In a multi-region Calico/OpenStack deployment, this must be configured somehow for each Felix (here in the datamodel, or in felix.cfg or the environment on each compute node), and must match the [calico] openstack_region value configured in neutron.conf on each node. [Default: Empty]","type":"string"},"policySyncPathPrefix":{"description":"PolicySyncPathPrefix is used to by Felix to communicate policy changes to external services, like Application layer policy. [Default: Empty]","type":"string"},"prometheusGoMetricsEnabled":{"description":"PrometheusGoMetricsEnabled disables Go runtime metrics collection, which the Prometheus client does by default, when set to false. This reduces the number of metrics reported, reducing Prometheus load. [Default: true]","type":"boolean"},"prometheusMetricsEnabled":{"description":"PrometheusMetricsEnabled enables the Prometheus metrics server in Felix if set to true. [Default: false]","type":"boolean"},"prometheusMetricsHost":{"description":"PrometheusMetricsHost is the host that the Prometheus metrics server should bind to. [Default: empty]","type":"string"},"prometheusMetricsPort":{"description":"PrometheusMetricsPort is the TCP port that the Prometheus metrics server should bind to. [Default: 9091]","type":"integer"},"prometheusProcessMetricsEnabled":{"description":"PrometheusProcessMetricsEnabled disables process metrics collection, which the Prometheus client does by default, when set to false. This reduces the number of metrics reported, reducing Prometheus load. [Default: true]","type":"boolean"},"prometheusWireGuardMetricsEnabled":{"description":"PrometheusWireGuardMetricsEnabled disables wireguard metrics collection, which the Prometheus client does by default, when set to false. This reduces the number of metrics reported, reducing Prometheus load. [Default: true]","type":"boolean"},"removeExternalRoutes":{"description":"Whether or not to remove device routes that have not been programmed by Felix. Disabling this will allow external applications to also add device routes. This is enabled by default which means we will remove externally added routes.","type":"boolean"},"reportingInterval":{"description":"ReportingInterval is the interval at which Felix reports its status into the datastore or 0 to disable. Must be non-zero in OpenStack deployments. [Default: 30s]","type":"string"},"reportingTTL":{"description":"ReportingTTL is the time-to-live setting for process-wide status reports. [Default: 90s]","type":"string"},"routeRefreshInterval":{"description":"RouteRefreshInterval is the period at which Felix re-checks the routes in the dataplane to ensure that no other process has accidentally broken Calico's rules. Set to 0 to disable route refresh. [Default: 90s]","type":"string"},"routeSource":{"description":"RouteSource configures where Felix gets its routing information. - WorkloadIPs: use workload endpoints to construct routes. - CalicoIPAM: the default - use IPAM data to construct routes.","type":"string"},"routeTableRange":{"description":"Calico programs additional Linux route tables for various purposes.  RouteTableRange specifies the indices of the route tables that Calico should use.","properties":{"max":{"type":"integer"},"min":{"type":"integer"}},"required":["max","min"],"type":"object"},"serviceLoopPrevention":{"description":"When service IP advertisement is enabled, prevent routing loops to service IPs that are not in use, by dropping or rejecting packets that do not get DNAT'd by kube-proxy. Unless set to \"Disabled\", in which case such routing loops continue to be allowed. [Default: Drop]","type":"string"},"sidecarAccelerationEnabled":{"description":"SidecarAccelerationEnabled enables experimental sidecar acceleration [Default: false]","type":"boolean"},"usageReportingEnabled":{"description":"UsageReportingEnabled reports anonymous Calico version number and cluster size to projectcalico.org. Logs warnings returned by the usage server. For example, if a significant security vulnerability has been discovered in the version of Calico being used. [Default: true]","type":"boolean"},"usageReportingInitialDelay":{"description":"UsageReportingInitialDelay controls the minimum delay before Felix makes a report. [Default: 300s]","type":"string"},"usageReportingInterval":{"description":"UsageReportingInterval controls the interval at which Felix makes reports. [Default: 86400s]","type":"string"},"useInternalDataplaneDriver":{"type":"boolean"},"vxlanEnabled":{"type":"boolean"},"vxlanMTU":{"description":"VXLANMTU is the MTU to set on the tunnel device. See Configuring MTU [Default: 1440]","type":"integer"},"vxlanPort":{"type":"integer"},"vxlanVNI":{"type":"integer"},"wireguardEnabled":{"description":"WireguardEnabled controls whether Wireguard is enabled. [Default: false]","type":"boolean"},"wireguardHostEncryptionEnabled":{"description":"WireguardHostEncryptionEnabled controls whether Wireguard host-to-host encryption is enabled. [Default: false]","type":"boolean"},"wireguardInterfaceName":{"description":"WireguardInterfaceName specifies the name to use for the Wireguard interface. [Default: wg.calico]","type":"string"},"wireguardListeningPort":{"description":"WireguardListeningPort controls the listening port used by Wireguard. [Default: 51820]","type":"integer"},"wireguardMTU":{"description":"WireguardMTU controls the MTU on the Wireguard interface. See Configuring MTU [Default: 1420]","type":"integer"},"wireguardRoutingRulePriority":{"description":"WireguardRoutingRulePriority controls the priority value to use for the Wireguard routing rule. [Default: 99]","type":"integer"},"xdpEnabled":{"description":"XDPEnabled enables XDP acceleration for suitable untracked incoming deny rules. [Default: true]","type":"boolean"},"xdpRefreshInterval":{"description":"XDPRefreshInterval is the period at which Felix re-checks all XDP state to ensure that no other process has accidentally broken Calico's BPF maps or attached programs. Set to 0 to disable XDP refresh. [Default: 90s]","type":"string"}},"type":"object"}},"type":"object"}},"served":true,"storage":true}]},"status":{"acceptedNames":{"kind":"","plural":""},"conditions":[],"storedVersions":[]}}
to:
Resource: "apiextensions.k8s.io/v1, Resource=customresourcedefinitions", GroupVersionKind: "apiextensions.k8s.io/v1, Kind=CustomResourceDefinition"
Name: "felixconfigurations.crd.projectcalico.org", Namespace: ""
for: "STDIN": CustomResourceDefinition.apiextensions.k8s.io "felixconfigurations.crd.projectcalico.org" is invalid: spec.preserveUnknownFields: Invalid value: true: must be false in order to use defaults in the schema
Error from server (Invalid): error when applying patch:
{"metadata":{"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"apiextensions.k8s.io/v1\",\"kind\":\"CustomResourceDefinition\",\"metadata\":{\"annotations\":{},\"labels\":{\"kubeone.io/addon\":\"cni-canal\"},\"name\":\"ipamblocks.crd.projectcalico.org\"},\"spec\":{\"group\":\"crd.projectcalico.org\",\"names\":{\"kind\":\"IPAMBlock\",\"listKind\":\"IPAMBlockList\",\"plural\":\"ipamblocks\",\"singular\":\"ipamblock\"},\"scope\":\"Cluster\",\"versions\":[{\"name\":\"v1\",\"schema\":{\"openAPIV3Schema\":{\"properties\":{\"apiVersion\":{\"description\":\"APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources\",\"type\":\"string\"},\"kind\":{\"description\":\"Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds\",\"type\":\"string\"},\"metadata\":{\"type\":\"object\"},\"spec\":{\"description\":\"IPAMBlockSpec contains the specification for an IPAMBlock resource.\",\"properties\":{\"affinity\":{\"description\":\"Affinity of the block, if this block has one. If set, it will be of the form \\\"host:\\u003chostname\\u003e\\\". If not set, this block is not affine to a host.\",\"type\":\"string\"},\"allocations\":{\"description\":\"Array of allocations in-use within this block. nil entries mean the allocation is free. For non-nil entries at index i, the index is the ordinal of the allocation within this block and the value is the index of the associated attributes in the Attributes array.\",\"items\":{\"nullable\":true,\"type\":\"integer\"},\"type\":\"array\"},\"attributes\":{\"description\":\"Attributes is an array of arbitrary metadata associated with allocations in the block. To find attributes for a given allocation, use the value of the allocation's entry in the Allocations array as the index of the element in this array.\",\"items\":{\"properties\":{\"handle_id\":{\"type\":\"string\"},\"secondary\":{\"additionalProperties\":{\"type\":\"string\"},\"type\":\"object\"}},\"type\":\"object\"},\"type\":\"array\"},\"cidr\":{\"description\":\"The block's CIDR.\",\"type\":\"string\"},\"deleted\":{\"description\":\"Deleted is an internal boolean used to workaround a limitation in the Kubernetes API whereby deletion will not return a conflict error if the block has been updated. It should not be set manually.\",\"type\":\"boolean\"},\"sequenceNumber\":{\"default\":0,\"description\":\"We store a sequence number that is updated each time the block is written. Each allocation will also store the sequence number of the block at the time of its creation. When releasing an IP, passing the sequence number associated with the allocation allows us to protect against a race condition and ensure the IP hasn't been released and re-allocated since the release request.\",\"format\":\"int64\",\"type\":\"integer\"},\"sequenceNumberForAllocation\":{\"additionalProperties\":{\"format\":\"int64\",\"type\":\"integer\"},\"description\":\"Map of allocated ordinal within the block to sequence number of the block at the time of allocation. Kubernetes does not allow numerical keys for maps, so the key is cast to a string.\",\"type\":\"object\"},\"strictAffinity\":{\"description\":\"StrictAffinity on the IPAMBlock is deprecated and no longer used by the code. Use IPAMConfig StrictAffinity instead.\",\"type\":\"boolean\"},\"unallocated\":{\"description\":\"Unallocated is an ordered list of allocations which are free in the block.\",\"items\":{\"type\":\"integer\"},\"type\":\"array\"}},\"required\":[\"allocations\",\"attributes\",\"cidr\",\"strictAffinity\",\"unallocated\"],\"type\":\"object\"}},\"type\":\"object\"}},\"served\":true,\"storage\":true}]},\"status\":{\"acceptedNames\":{\"kind\":\"\",\"plural\":\"\"},\"conditions\":[],\"storedVersions\":[]}}\n"}},"spec":{"versions":[{"name":"v1","schema":{"openAPIV3Schema":{"properties":{"apiVersion":{"description":"APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources","type":"string"},"kind":{"description":"Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds","type":"string"},"metadata":{"type":"object"},"spec":{"description":"IPAMBlockSpec contains the specification for an IPAMBlock resource.","properties":{"affinity":{"description":"Affinity of the block, if this block has one. If set, it will be of the form \"host:\u003chostname\u003e\". If not set, this block is not affine to a host.","type":"string"},"allocations":{"description":"Array of allocations in-use within this block. nil entries mean the allocation is free. For non-nil entries at index i, the index is the ordinal of the allocation within this block and the value is the index of the associated attributes in the Attributes array.","items":{"nullable":true,"type":"integer"},"type":"array"},"attributes":{"description":"Attributes is an array of arbitrary metadata associated with allocations in the block. To find attributes for a given allocation, use the value of the allocation's entry in the Allocations array as the index of the element in this array.","items":{"properties":{"handle_id":{"type":"string"},"secondary":{"additionalProperties":{"type":"string"},"type":"object"}},"type":"object"},"type":"array"},"cidr":{"description":"The block's CIDR.","type":"string"},"deleted":{"description":"Deleted is an internal boolean used to workaround a limitation in the Kubernetes API whereby deletion will not return a conflict error if the block has been updated. It should not be set manually.","type":"boolean"},"sequenceNumber":{"default":0,"description":"We store a sequence number that is updated each time the block is written. Each allocation will also store the sequence number of the block at the time of its creation. When releasing an IP, passing the sequence number associated with the allocation allows us to protect against a race condition and ensure the IP hasn't been released and re-allocated since the release request.","format":"int64","type":"integer"},"sequenceNumberForAllocation":{"additionalProperties":{"format":"int64","type":"integer"},"description":"Map of allocated ordinal within the block to sequence number of the block at the time of allocation. Kubernetes does not allow numerical keys for maps, so the key is cast to a string.","type":"object"},"strictAffinity":{"description":"StrictAffinity on the IPAMBlock is deprecated and no longer used by the code. Use IPAMConfig StrictAffinity instead.","type":"boolean"},"unallocated":{"description":"Unallocated is an ordered list of allocations which are free in the block.","items":{"type":"integer"},"type":"array"}},"required":["allocations","attributes","cidr","strictAffinity","unallocated"],"type":"object"}},"type":"object"}},"served":true,"storage":true}]},"status":{"acceptedNames":{"kind":"","plural":""},"conditions":[],"storedVersions":[]}}
to:
Resource: "apiextensions.k8s.io/v1, Resource=customresourcedefinitions", GroupVersionKind: "apiextensions.k8s.io/v1, Kind=CustomResourceDefinition"
Name: "ipamblocks.crd.projectcalico.org", Namespace: ""
for: "STDIN": CustomResourceDefinition.apiextensions.k8s.io "ipamblocks.crd.projectcalico.org" is invalid: spec.preserveUnknownFields: Invalid value: true: must be false in order to use defaults in the schema
configmap/canal-config unchanged
customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org configured
customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org configured
customresourcedefinition.apiextensions.k8s.io/blockaffinities.crd.projectcalico.org configured
customresourcedefinition.apiextensions.k8s.io/caliconodestatuses.crd.projectcalico.org configured
customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org configured
customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org configured
customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org configured
customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org configured
customresourcedefinition.apiextensions.k8s.io/ipamconfigs.crd.projectcalico.org configured
customresourcedefinition.apiextensions.k8s.io/ipamhandles.crd.projectcalico.org configured
customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org configured
customresourcedefinition.apiextensions.k8s.io/ipreservations.crd.projectcalico.org configured
customresourcedefinition.apiextensions.k8s.io/kubecontrollersconfigurations.crd.projectcalico.org configured
customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org configured
customresourcedefinition.apiextensions.k8s.io/networksets.crd.projectcalico.org configured
clusterrole.rbac.authorization.k8s.io/calico-kube-controllers unchanged
clusterrolebinding.rbac.authorization.k8s.io/calico-kube-controllers unchanged
clusterrole.rbac.authorization.k8s.io/calico-node unchanged
clusterrole.rbac.authorization.k8s.io/flannel unchanged
clusterrolebinding.rbac.authorization.k8s.io/canal-flannel unchanged
clusterrolebinding.rbac.authorization.k8s.io/canal-calico unchanged
daemonset.apps/canal configured
serviceaccount/canal unchanged
deployment.apps/calico-kube-controllers configured
serviceaccount/calico-kube-controllers unchanged
poddisruptionbudget.policy/calico-kube-controllers unchanged
time="09:13:09 UTC" level=warning msg="Retrying task..."
time="09:13:10 UTC" level=info msg="Applying addon metrics-server..."
time="09:13:10 UTC" level=info msg="Parsing addons manifest 'metrics-server.yaml'\n"
time="09:13:10 UTC" level=info msg="Parsing addons manifest 'poddisruptionbudget.yaml'\n"
+ sudo KUBECONFIG=/etc/kubernetes/admin.conf \
kubectl apply -f - --prune -l "kubeone.io/addon=metrics-server"

clusterrole.rbac.authorization.k8s.io/system:aggregated-metrics-reader unchanged
serviceaccount/metrics-server unchanged
rolebinding.rbac.authorization.k8s.io/metrics-server-auth-reader unchanged
clusterrolebinding.rbac.authorization.k8s.io/metrics-server:system:auth-delegator unchanged
clusterrole.rbac.authorization.k8s.io/system:metrics-server unchanged
clusterrolebinding.rbac.authorization.k8s.io/system:metrics-server unchanged
apiservice.apiregistration.k8s.io/v1beta1.metrics.k8s.io unchanged
secret/metrics-server-serving-cert configured
deployment.apps/metrics-server configured
service/metrics-server unchanged
poddisruptionbudget.policy/metrics-server configured

[... cut off ...]

$ kubectl get crd felixconfigurations.crd.projectcalico.org -oyaml apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: annotations: kubectl.kubernetes.io/last-applied-configuration: | {"apiVersion":"apiextensions.k8s.io/v1","kind":"CustomResourceDefinition","metadata":{"annotations":{},"labels":{"kubeone.io/addon":"cni-canal"},"name":"felixconfigurations.crd.projectcalico.org"},"spec":{"group":"crd.projectcalico.org","names":{"kind":"FelixConfiguration","listKind":"FelixConfigurationList","plural":"felixconfigurations","singular":"felixconfiguration"},"scope":"Cluster","versions":[{"name":"v1","schema":{"openAPIV3Schema":{"description":"Felix Configuration contains the configuration for Felix.","properties":{"apiVersion":{"description":"APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources","type":"string"},"kind":{"description":"Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds","type":"string"},"metadata":{"type":"object"},"spec":{"description":"FelixConfigurationSpec contains the values of the Felix configuration.","properties":{"allowIPIPPacketsFromWorkloads":{"description":"AllowIPIPPacketsFromWorkloads controls whether Felix will add a rule to drop IPIP encapsulated traffic from workloads [Default: false]","type":"boolean"},"allowVXLANPacketsFromWorkloads":{"description":"AllowVXLANPacketsFromWorkloads controls whether Felix will add a rule to drop VXLAN encapsulated traffic from workloads [Default: false]","type":"boolean"},"awsSrcDstCheck":{"description":"Set source-destination-check on AWS EC2 instances. Accepted value must be one of \"DoNothing\", \"Enable\" or \"Disable\". [Default: DoNothing]","enum":["DoNothing","Enable","Disable"],"type":"string"},"bpfConnectTimeLoadBalancingEnabled":{"description":"BPFConnectTimeLoadBalancingEnabled when in BPF mode, controls whether Felix installs the connection-time load balancer. The connect-time load balancer is required for the host to be able to reach Kubernetes services and it improves the performance of pod-to-service connections. The only reason to disable it is for debugging purposes. [Default: true]","type":"boolean"},"bpfDataIfacePattern":{"description":"BPFDataIfacePattern is a regular expression that controls which interfaces Felix should attach BPF programs to in order to catch traffic to/from the network. This needs to match the interfaces that Calico workload traffic flows over as well as any interfaces that handle incoming traffic to nodeports and services from outside the cluster. It should not match the workload interfaces (usually named cali...).","type":"string"},"bpfDisableUnprivileged":{"description":"BPFDisableUnprivileged, if enabled, Felix sets the kernel.unprivileged_bpf_disabled sysctl to disable unprivileged use of BPF. This ensures that unprivileged users cannot access Calico's BPF maps and cannot insert their own BPF programs to interfere with Calico's. [Default: true]","type":"boolean"},"bpfEnabled":{"description":"BPFEnabled, if enabled Felix will use the BPF dataplane. [Default: false]","type":"boolean"},"bpfExtToServiceConnmark":{"description":"BPFExtToServiceConnmark in BPF mode, control a 32bit mark that is set on connections from an external client to a local service. This mark allows us to control how packets of that connection are routed within the host and how is routing intepreted by RPF check. [Default: 0]","type":"integer"},"bpfExternalServiceMode":{"description":"BPFExternalServiceMode in BPF mode, controls how connections from outside the cluster to services (node ports and cluster IPs) are forwarded to remote workloads. If set to \"Tunnel\" then both request and response traffic is tunneled to the remote node. If set to \"DSR\", the request traffic is tunneled but the response traffic is sent directly from the remote node. In \"DSR\" mode, the remote node appears to use the IP of the ingress node; this requires a permissive L2 network. [Default: Tunnel]","type":"string"},"bpfKubeProxyEndpointSlicesEnabled":{"description":"BPFKubeProxyEndpointSlicesEnabled in BPF mode, controls whether Felix's embedded kube-proxy accepts EndpointSlices or not.","type":"boolean"},"bpfKubeProxyIptablesCleanupEnabled":{"description":"BPFKubeProxyIptablesCleanupEnabled, if enabled in BPF mode, Felix will proactively clean up the upstream Kubernetes kube-proxy's iptables chains. Should only be enabled if kube-proxy is not running. [Default: true]","type":"boolean"},"bpfKubeProxyMinSyncPeriod":{"description":"BPFKubeProxyMinSyncPeriod, in BPF mode, controls the minimum time between updates to the dataplane for Felix's embedded kube-proxy. Lower values give reduced set-up latency. Higher values reduce Felix CPU usage by batching up more work. [Default: 1s]","type":"string"},"bpfLogLevel":{"description":"BPFLogLevel controls the log level of the BPF programs when in BPF dataplane mode. One of \"Off\", \"Info\", or \"Debug\". The logs are emitted to the BPF trace pipe, accessible with the command `tc exec bpf debug`. [Default: Off].","type":"string"},"chainInsertMode":{"description":"ChainInsertMode controls whether Felix hooks the kernel's top-level iptables chains by inserting a rule at the top of the chain or by appending a rule at the bottom. insert is the safe default since it prevents Calico's rules from being bypassed. If you switch to append mode, be sure that the other rules in the chains signal acceptance by falling through to the Calico rules, otherwise the Calico policy will be bypassed. [Default: insert]","type":"string"},"dataplaneDriver":{"type":"string"},"debugDisableLogDropping":{"type":"boolean"},"debugMemoryProfilePath":{"type":"string"},"debugSimulateCalcGraphHangAfter":{"type":"string"},"debugSimulateDataplaneHangAfter":{"type":"string"},"defaultEndpointToHostAction":{"description":"DefaultEndpointToHostAction controls what happens to traffic that goes from a workload endpoint to the host itself (after the traffic hits the endpoint egress policy). By default Calico blocks traffic from workload endpoints to the host itself with an iptables \"DROP\" action. If you want to allow some or all traffic from endpoint to host, set this parameter to RETURN or ACCEPT. Use RETURN if you have your own rules in the iptables \"INPUT\" chain; Calico will insert its rules at the top of that chain, then \"RETURN\" packets to the \"INPUT\" chain once it has completed processing workload endpoint egress policy. Use ACCEPT to unconditionally accept packets from workloads after processing workload endpoint egress policy. [Default: Drop]","type":"string"},"deviceRouteProtocol":{"description":"This defines the route protocol added to programmed device routes, by default this will be RTPROT_BOOT when left blank.","type":"integer"},"deviceRouteSourceAddress":{"description":"This is the source address to use on programmed device routes. By default the source address is left blank, leaving the kernel to choose the source address used.","type":"string"},"disableConntrackInvalidCheck":{"type":"boolean"},"endpointReportingDelay":{"type":"string"},"endpointReportingEnabled":{"type":"boolean"},"externalNodesList":{"description":"ExternalNodesCIDRList is a list of CIDR's of external-non-calico-nodes which may source tunnel traffic and have the tunneled traffic be accepted at calico nodes.","items":{"type":"string"},"type":"array"},"failsafeInboundHostPorts":{"description":"FailsafeInboundHostPorts is a list of UDP/TCP ports and CIDRs that Felix will allow incoming traffic to host endpoints on irrespective of the security policy. This is useful to avoid accidentally cutting off a host with incorrect configuration. For back-compatibility, if the protocol is not specified, it defaults to \"tcp\". If a CIDR is not specified, it will allow traffic from all addresses. To disable all inbound host ports, use the value none. The default value allows ssh access and DHCP. [Default: tcp:22, udp:68, tcp:179, tcp:2379, tcp:2380, tcp:6443, tcp:6666, tcp:6667]","items":{"description":"ProtoPort is combination of protocol, port, and CIDR. Protocol and port must be specified.","properties":{"net":{"type":"string"},"port":{"type":"integer"},"protocol":{"type":"string"}},"required":["port","protocol"],"type":"object"},"type":"array"},"failsafeOutboundHostPorts":{"description":"FailsafeOutboundHostPorts is a list of UDP/TCP ports and CIDRs that Felix will allow outgoing traffic from host endpoints to irrespective of the security policy. This is useful to avoid accidentally cutting off a host with incorrect configuration. For back-compatibility, if the protocol is not specified, it defaults to \"tcp\". If a CIDR is not specified, it will allow traffic from all addresses. To disable all outbound host ports, use the value none. The default value opens etcd's standard ports to ensure that Felix does not get cut off from etcd as well as allowing DHCP and DNS. [Default: tcp:179, tcp:2379, tcp:2380, tcp:6443, tcp:6666, tcp:6667, udp:53, udp:67]","items":{"description":"ProtoPort is combination of protocol, port, and CIDR. Protocol and port must be specified.","properties":{"net":{"type":"string"},"port":{"type":"integer"},"protocol":{"type":"string"}},"required":["port","protocol"],"type":"object"},"type":"array"},"featureDetectOverride":{"description":"FeatureDetectOverride is used to override the feature detection. Values are specified in a comma separated list with no spaces, example; \"SNATFullyRandom=true,MASQFullyRandom=false,RestoreSupportsLock=\". \"true\" or \"false\" will force the feature, empty or omitted values are auto-detected.","type":"string"},"genericXDPEnabled":{"description":"GenericXDPEnabled enables Generic XDP so network cards that don't support XDP offload or driver modes can use XDP. This is not recommended since it doesn't provide better performance than iptables. [Default: false]","type":"boolean"},"healthEnabled":{"type":"boolean"},"healthHost":{"type":"string"},"healthPort":{"type":"integer"},"interfaceExclude":{"description":"InterfaceExclude is a comma-separated list of interfaces that Felix should exclude when monitoring for host endpoints. The default value ensures that Felix ignores Kubernetes' IPVS dummy interface, which is used internally by kube-proxy. If you want to exclude multiple interface names using a single value, the list supports regular expressions. For regular expressions you must wrap the value with '/'. For example having values '/^kube/,veth1' will exclude all interfaces that begin with 'kube' and also the interface 'veth1'. [Default: kube-ipvs0]","type":"string"},"interfacePrefix":{"description":"InterfacePrefix is the interface name prefix that identifies workload endpoints and so distinguishes them from host endpoint interfaces. Note: in environments other than bare metal, the orchestrators configure this appropriately. For example our Kubernetes and Docker integrations set the 'cali' value, and our OpenStack integration sets the 'tap' value. [Default: cali]","type":"string"},"interfaceRefreshInterval":{"description":"InterfaceRefreshInterval is the period at which Felix rescans local interfaces to verify their state. The rescan can be disabled by setting the interval to 0.","type":"string"},"ipipEnabled":{"type":"boolean"},"ipipMTU":{"description":"IPIPMTU is the MTU to set on the tunnel device. See Configuring MTU [Default: 1440]","type":"integer"},"ipsetsRefreshInterval":{"description":"IpsetsRefreshInterval is the period at which Felix re-checks all iptables state to ensure that no other process has accidentally broken Calico's rules. Set to 0 to disable iptables refresh. [Default: 90s]","type":"string"},"iptablesBackend":{"description":"IptablesBackend specifies which backend of iptables will be used. The default is legacy.","type":"string"},"iptablesFilterAllowAction":{"type":"string"},"iptablesLockFilePath":{"description":"IptablesLockFilePath is the location of the iptables lock file. You may need to change this if the lock file is not in its standard location (for example if you have mapped it into Felix's container at a different path). [Default: /run/xtables.lock]","type":"string"},"iptablesLockProbeInterval":{"description":"IptablesLockProbeInterval is the time that Felix will wait between attempts to acquire the iptables lock if it is not available. Lower values make Felix more responsive when the lock is contended, but use more CPU. [Default: 50ms]","type":"string"},"iptablesLockTimeout":{"description":"IptablesLockTimeout is the time that Felix will wait for the iptables lock, or 0, to disable. To use this feature, Felix must share the iptables lock file with all other processes that also take the lock. When running Felix inside a container, this requires the /run directory of the host to be mounted into the calico/node or calico/felix container. [Default: 0s disabled]","type":"string"},"iptablesMangleAllowAction":{"type":"string"},"iptablesMarkMask":{"description":"IptablesMarkMask is the mask that Felix selects its IPTables Mark bits from. Should be a 32 bit hexadecimal number with at least 8 bits set, none of which clash with any other mark bits in use on the system. [Default: 0xff000000]","format":"int32","type":"integer"},"iptablesNATOutgoingInterfaceFilter":{"type":"string"},"iptablesPostWriteCheckInterval":{"description":"IptablesPostWriteCheckInterval is the period after Felix has done a write to the dataplane that it schedules an extra read back in order to check the write was not clobbered by another process. This should only occur if another application on the system doesn't respect the iptables lock. [Default: 1s]","type":"string"},"iptablesRefreshInterval":{"description":"IptablesRefreshInterval is the period at which Felix re-checks the IP sets in the dataplane to ensure that no other process has accidentally broken Calico's rules. Set to 0 to disable IP sets refresh. Note: the default for this value is lower than the other refresh intervals as a workaround for a Linux kernel bug that was fixed in kernel version 4.11. If you are using v4.11 or greater you may want to set this to, a higher value to reduce Felix CPU usage. [Default: 10s]","type":"string"},"ipv6Support":{"type":"boolean"},"kubeNodePortRanges":{"description":"KubeNodePortRanges holds list of port ranges used for service node ports. Only used if felix detects kube-proxy running in ipvs mode. Felix uses these ranges to separate host and workload traffic. [Default: 30000:32767].","items":{"anyOf":[{"type":"integer"},{"type":"string"}],"pattern":"^.*","x-kubernetes-int-or-string":true},"type":"array"},"logFilePath":{"description":"LogFilePath is the full path to the Felix log. Set to none to disable file logging. [Default: /var/log/calico/felix.log]","type":"string"},"logPrefix":{"description":"LogPrefix is the log prefix that Felix uses when rendering LOG rules. [Default: calico-packet]","type":"string"},"logSeverityFile":{"description":"LogSeverityFile is the log severity above which logs are sent to the log file. [Default: Info]","type":"string"},"logSeverityScreen":{"description":"LogSeverityScreen is the log severity above which logs are sent to the stdout. [Default: Info]","type":"string"},"logSeveritySys":{"description":"LogSeveritySys is the log severity above which logs are sent to the syslog. Set to None for no logging to syslog. [Default: Info]","type":"string"},"maxIpsetSize":{"type":"integer"},"metadataAddr":{"description":"MetadataAddr is the IP address or domain name of the server that can answer VM queries for cloud-init metadata. In OpenStack, this corresponds to the machine running nova-api (or in Ubuntu, nova-api-metadata). A value of none (case insensitive) means that Felix should not set up any NAT rule for the metadata path. [Default: 127.0.0.1]","type":"string"},"metadataPort":{"description":"MetadataPort is the port of the metadata server. This, combined with global.MetadataAddr (if not 'None'), is used to set up a NAT rule, from 169.254.169.254:80 to MetadataAddr:MetadataPort. In most cases this should not need to be changed [Default: 8775].","type":"integer"},"mtuIfacePattern":{"description":"MTUIfacePattern is a regular expression that controls which interfaces Felix should scan in order to calculate the host's MTU. This should not match workload interfaces (usually named cali...).","type":"string"},"natOutgoingAddress":{"description":"NATOutgoingAddress specifies an address to use when performing source NAT for traffic in a natOutgoing pool that is leaving the network. By default the address used is an address on the interface the traffic is leaving on (ie it uses the iptables MASQUERADE target)","type":"string"},"natPortRange":{"anyOf":[{"type":"integer"},{"type":"string"}],"description":"NATPortRange specifies the range of ports that is used for port mapping when doing outgoing NAT. When unset the default behavior of the network stack is used.","pattern":"^.*","x-kubernetes-int-or-string":true},"netlinkTimeout":{"type":"string"},"openstackRegion":{"description":"OpenstackRegion is the name of the region that a particular Felix belongs to. In a multi-region Calico/OpenStack deployment, this must be configured somehow for each Felix (here in the datamodel, or in felix.cfg or the environment on each compute node), and must match the [calico] openstack_region value configured in neutron.conf on each node. [Default: Empty]","type":"string"},"policySyncPathPrefix":{"description":"PolicySyncPathPrefix is used to by Felix to communicate policy changes to external services, like Application layer policy. [Default: Empty]","type":"string"},"prometheusGoMetricsEnabled":{"description":"PrometheusGoMetricsEnabled disables Go runtime metrics collection, which the Prometheus client does by default, when set to false. This reduces the number of metrics reported, reducing Prometheus load. [Default: true]","type":"boolean"},"prometheusMetricsEnabled":{"description":"PrometheusMetricsEnabled enables the Prometheus metrics server in Felix if set to true. [Default: false]","type":"boolean"},"prometheusMetricsHost":{"description":"PrometheusMetricsHost is the host that the Prometheus metrics server should bind to. [Default: empty]","type":"string"},"prometheusMetricsPort":{"description":"PrometheusMetricsPort is the TCP port that the Prometheus metrics server should bind to. [Default: 9091]","type":"integer"},"prometheusProcessMetricsEnabled":{"description":"PrometheusProcessMetricsEnabled disables process metrics collection, which the Prometheus client does by default, when set to false. This reduces the number of metrics reported, reducing Prometheus load. [Default: true]","type":"boolean"},"prometheusWireGuardMetricsEnabled":{"description":"PrometheusWireGuardMetricsEnabled disables wireguard metrics collection, which the Prometheus client does by default, when set to false. This reduces the number of metrics reported, reducing Prometheus load. [Default: true]","type":"boolean"},"removeExternalRoutes":{"description":"Whether or not to remove device routes that have not been programmed by Felix. Disabling this will allow external applications to also add device routes. This is enabled by default which means we will remove externally added routes.","type":"boolean"},"reportingInterval":{"description":"ReportingInterval is the interval at which Felix reports its status into the datastore or 0 to disable. Must be non-zero in OpenStack deployments. [Default: 30s]","type":"string"},"reportingTTL":{"description":"ReportingTTL is the time-to-live setting for process-wide status reports. [Default: 90s]","type":"string"},"routeRefreshInterval":{"description":"RouteRefreshInterval is the period at which Felix re-checks the routes in the dataplane to ensure that no other process has accidentally broken Calico's rules. Set to 0 to disable route refresh. [Default: 90s]","type":"string"},"routeSource":{"description":"RouteSource configures where Felix gets its routing information. - WorkloadIPs: use workload endpoints to construct routes. - CalicoIPAM: the default - use IPAM data to construct routes.","type":"string"},"routeTableRange":{"description":"Calico programs additional Linux route tables for various purposes. RouteTableRange specifies the indices of the route tables that Calico should use.","properties":{"max":{"type":"integer"},"min":{"type":"integer"}},"required":["max","min"],"type":"object"},"serviceLoopPrevention":{"description":"When service IP advertisement is enabled, prevent routing loops to service IPs that are not in use, by dropping or rejecting packets that do not get DNAT'd by kube-proxy. Unless set to \"Disabled\", in which case such routing loops continue to be allowed. [Default: Drop]","type":"string"},"sidecarAccelerationEnabled":{"description":"SidecarAccelerationEnabled enables experimental sidecar acceleration [Default: false]","type":"boolean"},"usageReportingEnabled":{"description":"UsageReportingEnabled reports anonymous Calico version number and cluster size to projectcalico.org. Logs warnings returned by the usage server. For example, if a significant security vulnerability has been discovered in the version of Calico being used. [Default: true]","type":"boolean"},"usageReportingInitialDelay":{"description":"UsageReportingInitialDelay controls the minimum delay before Felix makes a report. [Default: 300s]","type":"string"},"usageReportingInterval":{"description":"UsageReportingInterval controls the interval at which Felix makes reports. [Default: 86400s]","type":"string"},"useInternalDataplaneDriver":{"type":"boolean"},"vxlanEnabled":{"type":"boolean"},"vxlanMTU":{"description":"VXLANMTU is the MTU to set on the tunnel device. See Configuring MTU [Default: 1440]","type":"integer"},"vxlanPort":{"type":"integer"},"vxlanVNI":{"type":"integer"},"wireguardEnabled":{"description":"WireguardEnabled controls whether Wireguard is enabled. [Default: false]","type":"boolean"},"wireguardHostEncryptionEnabled":{"description":"WireguardHostEncryptionEnabled controls whether Wireguard host-to-host encryption is enabled. [Default: false]","type":"boolean"},"wireguardInterfaceName":{"description":"WireguardInterfaceName specifies the name to use for the Wireguard interface. [Default: wg.calico]","type":"string"},"wireguardListeningPort":{"description":"WireguardListeningPort controls the listening port used by Wireguard. [Default: 51820]","type":"integer"},"wireguardMTU":{"description":"WireguardMTU controls the MTU on the Wireguard interface. See Configuring MTU [Default: 1420]","type":"integer"},"wireguardRoutingRulePriority":{"description":"WireguardRoutingRulePriority controls the priority value to use for the Wireguard routing rule. [Default: 99]","type":"integer"},"xdpEnabled":{"description":"XDPEnabled enables XDP acceleration for suitable untracked incoming deny rules. [Default: true]","type":"boolean"},"xdpRefreshInterval":{"description":"XDPRefreshInterval is the period at which Felix re-checks all XDP state to ensure that no other process has accidentally broken Calico's BPF maps or attached programs. Set to 0 to disable XDP refresh. [Default: 90s]","type":"string"}},"type":"object"}},"type":"object"}},"served":true,"storage":true}]},"status":{"acceptedNames":{"kind":"","plural":""},"conditions":[],"storedVersions":[]}} creationTimestamp: "2021-05-17T09:12:39Z" generation: 3 labels: kubeone.io/addon: cni-canal kubeone.io/component: canal name: felixconfigurations.crd.projectcalico.org resourceVersion: "97211797" uid: b0661472-deff-419a-824e-a4ee077e573c spec: conversion: strategy: None group: crd.projectcalico.org names: kind: FelixConfiguration listKind: FelixConfigurationList plural: felixconfigurations singular: felixconfiguration preserveUnknownFields: true scope: Cluster versions: - name: v1 schema: openAPIV3Schema: description: Felix Configuration contains the configuration for Felix. properties: apiVersion: description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources' type: string kind: description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds' type: string metadata: type: object spec: description: FelixConfigurationSpec contains the values of the Felix configuration. properties: allowIPIPPacketsFromWorkloads: description: 'AllowIPIPPacketsFromWorkloads controls whether Felix will add a rule to drop IPIP encapsulated traffic from workloads [Default: false]' type: boolean allowVXLANPacketsFromWorkloads: description: 'AllowVXLANPacketsFromWorkloads controls whether Felix will add a rule to drop VXLAN encapsulated traffic from workloads [Default: false]' type: boolean awsSrcDstCheck: description: 'Set source-destination-check on AWS EC2 instances. Accepted value must be one of "DoNothing", "Enable" or "Disable". [Default: DoNothing]' enum: - DoNothing - Enable - Disable type: string bpfConnectTimeLoadBalancingEnabled: description: 'BPFConnectTimeLoadBalancingEnabled when in BPF mode, controls whether Felix installs the connection-time load balancer. The connect-time load balancer is required for the host to be able to reach Kubernetes services and it improves the performance of pod-to-service connections. The only reason to disable it is for debugging purposes. [Default: true]' type: boolean bpfDataIfacePattern: description: BPFDataIfacePattern is a regular expression that controls which interfaces Felix should attach BPF programs to in order to catch traffic to/from the network. This needs to match the interfaces that Calico workload traffic flows over as well as any interfaces that handle incoming traffic to nodeports and services from outside the cluster. It should not match the workload interfaces (usually named cali...). type: string bpfDisableUnprivileged: description: 'BPFDisableUnprivileged, if enabled, Felix sets the kernel.unprivileged_bpf_disabled sysctl to disable unprivileged use of BPF. This ensures that unprivileged users cannot access Calico''s BPF maps and cannot insert their own BPF programs to interfere with Calico''s. [Default: true]' type: boolean bpfEnabled: description: 'BPFEnabled, if enabled Felix will use the BPF dataplane. [Default: false]' type: boolean bpfExtToServiceConnmark: description: 'BPFExtToServiceConnmark in BPF mode, control a 32bit mark that is set on connections from an external client to a local service. This mark allows us to control how packets of that connection are routed within the host and how is routing intepreted by RPF check. [Default: 0]' type: integer bpfExternalServiceMode: description: 'BPFExternalServiceMode in BPF mode, controls how connections from outside the cluster to services (node ports and cluster IPs) are forwarded to remote workloads. If set to "Tunnel" then both request and response traffic is tunneled to the remote node. If set to "DSR", the request traffic is tunneled but the response traffic is sent directly from the remote node. In "DSR" mode, the remote node appears to use the IP of the ingress node; this requires a permissive L2 network. [Default: Tunnel]' type: string bpfKubeProxyEndpointSlicesEnabled: description: BPFKubeProxyEndpointSlicesEnabled in BPF mode, controls whether Felix's embedded kube-proxy accepts EndpointSlices or not. type: boolean bpfKubeProxyIptablesCleanupEnabled: description: 'BPFKubeProxyIptablesCleanupEnabled, if enabled in BPF mode, Felix will proactively clean up the upstream Kubernetes kube-proxy''s iptables chains. Should only be enabled if kube-proxy is not running. [Default: true]' type: boolean bpfKubeProxyMinSyncPeriod: description: 'BPFKubeProxyMinSyncPeriod, in BPF mode, controls the minimum time between updates to the dataplane for Felix''s embedded kube-proxy. Lower values give reduced set-up latency. Higher values reduce Felix CPU usage by batching up more work. [Default: 1s]' type: string bpfLogLevel: description: 'BPFLogLevel controls the log level of the BPF programs when in BPF dataplane mode. One of "Off", "Info", or "Debug". The logs are emitted to the BPF trace pipe, accessible with the command `tc exec bpf debug`. [Default: Off].' type: string chainInsertMode: description: 'ChainInsertMode controls whether Felix hooks the kernel''s top-level iptables chains by inserting a rule at the top of the chain or by appending a rule at the bottom. insert is the safe default since it prevents Calico''s rules from being bypassed. If you switch to append mode, be sure that the other rules in the chains signal acceptance by falling through to the Calico rules, otherwise the Calico policy will be bypassed. [Default: insert]' type: string dataplaneDriver: type: string debugDisableLogDropping: type: boolean debugMemoryProfilePath: type: string debugSimulateCalcGraphHangAfter: type: string debugSimulateDataplaneHangAfter: type: string defaultEndpointToHostAction: description: 'DefaultEndpointToHostAction controls what happens to traffic that goes from a workload endpoint to the host itself (after the traffic hits the endpoint egress policy). By default Calico blocks traffic from workload endpoints to the host itself with an iptables "DROP" action. If you want to allow some or all traffic from endpoint to host, set this parameter to RETURN or ACCEPT. Use RETURN if you have your own rules in the iptables "INPUT" chain; Calico will insert its rules at the top of that chain, then "RETURN" packets to the "INPUT" chain once it has completed processing workload endpoint egress policy. Use ACCEPT to unconditionally accept packets from workloads after processing workload endpoint egress policy. [Default: Drop]' type: string deviceRouteProtocol: description: This defines the route protocol added to programmed device routes, by default this will be RTPROT_BOOT when left blank. type: integer deviceRouteSourceAddress: description: This is the source address to use on programmed device routes. By default the source address is left blank, leaving the kernel to choose the source address used. type: string disableConntrackInvalidCheck: type: boolean endpointReportingDelay: type: string endpointReportingEnabled: type: boolean externalNodesList: description: ExternalNodesCIDRList is a list of CIDR's of external-non-calico-nodes which may source tunnel traffic and have the tunneled traffic be accepted at calico nodes. items: type: string type: array failsafeInboundHostPorts: description: 'FailsafeInboundHostPorts is a list of UDP/TCP ports and CIDRs that Felix will allow incoming traffic to host endpoints on irrespective of the security policy. This is useful to avoid accidentally cutting off a host with incorrect configuration. For back-compatibility, if the protocol is not specified, it defaults to "tcp". If a CIDR is not specified, it will allow traffic from all addresses. To disable all inbound host ports, use the value none. The default value allows ssh access and DHCP. [Default: tcp:22, udp:68, tcp:179, tcp:2379, tcp:2380, tcp:6443, tcp:6666, tcp:6667]' items: description: ProtoPort is combination of protocol, port, and CIDR. Protocol and port must be specified. properties: net: type: string port: type: integer protocol: type: string required: - port - protocol type: object type: array failsafeOutboundHostPorts: description: 'FailsafeOutboundHostPorts is a list of UDP/TCP ports and CIDRs that Felix will allow outgoing traffic from host endpoints to irrespective of the security policy. This is useful to avoid accidentally cutting off a host with incorrect configuration. For back-compatibility, if the protocol is not specified, it defaults to "tcp". If a CIDR is not specified, it will allow traffic from all addresses. To disable all outbound host ports, use the value none. The default value opens etcd''s standard ports to ensure that Felix does not get cut off from etcd as well as allowing DHCP and DNS. [Default: tcp:179, tcp:2379, tcp:2380, tcp:6443, tcp:6666, tcp:6667, udp:53, udp:67]' items: description: ProtoPort is combination of protocol, port, and CIDR. Protocol and port must be specified. properties: net: type: string port: type: integer protocol: type: string required: - port - protocol type: object type: array featureDetectOverride: description: FeatureDetectOverride is used to override the feature detection. Values are specified in a comma separated list with no spaces, example; "SNATFullyRandom=true,MASQFullyRandom=false,RestoreSupportsLock=". "true" or "false" will force the feature, empty or omitted values are auto-detected. type: string genericXDPEnabled: description: 'GenericXDPEnabled enables Generic XDP so network cards that don''t support XDP offload or driver modes can use XDP. This is not recommended since it doesn''t provide better performance than iptables. [Default: false]' type: boolean healthEnabled: type: boolean healthHost: type: string healthPort: type: integer interfaceExclude: description: 'InterfaceExclude is a comma-separated list of interfaces that Felix should exclude when monitoring for host endpoints. The default value ensures that Felix ignores Kubernetes'' IPVS dummy interface, which is used internally by kube-proxy. If you want to exclude multiple interface names using a single value, the list supports regular expressions. For regular expressions you must wrap the value with ''/''. For example having values ''/^kube/,veth1'' will exclude all interfaces that begin with ''kube'' and also the interface ''veth1''. [Default: kube-ipvs0]' type: string interfacePrefix: description: 'InterfacePrefix is the interface name prefix that identifies workload endpoints and so distinguishes them from host endpoint interfaces. Note: in environments other than bare metal, the orchestrators configure this appropriately. For example our Kubernetes and Docker integrations set the ''cali'' value, and our OpenStack integration sets the ''tap'' value. [Default: cali]' type: string interfaceRefreshInterval: description: InterfaceRefreshInterval is the period at which Felix rescans local interfaces to verify their state. The rescan can be disabled by setting the interval to 0. type: string ipipEnabled: type: boolean ipipMTU: description: 'IPIPMTU is the MTU to set on the tunnel device. See Configuring MTU [Default: 1440]' type: integer ipsetsRefreshInterval: description: 'IpsetsRefreshInterval is the period at which Felix re-checks all iptables state to ensure that no other process has accidentally broken Calico''s rules. Set to 0 to disable iptables refresh. [Default: 90s]' type: string iptablesBackend: description: IptablesBackend specifies which backend of iptables will be used. The default is legacy. type: string iptablesFilterAllowAction: type: string iptablesLockFilePath: description: 'IptablesLockFilePath is the location of the iptables lock file. You may need to change this if the lock file is not in its standard location (for example if you have mapped it into Felix''s container at a different path). [Default: /run/xtables.lock]' type: string iptablesLockProbeInterval: description: 'IptablesLockProbeInterval is the time that Felix will wait between attempts to acquire the iptables lock if it is not available. Lower values make Felix more responsive when the lock is contended, but use more CPU. [Default: 50ms]' type: string iptablesLockTimeout: description: 'IptablesLockTimeout is the time that Felix will wait for the iptables lock, or 0, to disable. To use this feature, Felix must share the iptables lock file with all other processes that also take the lock. When running Felix inside a container, this requires the /run directory of the host to be mounted into the calico/node or calico/felix container. [Default: 0s disabled]' type: string iptablesMangleAllowAction: type: string iptablesMarkMask: description: 'IptablesMarkMask is the mask that Felix selects its IPTables Mark bits from. Should be a 32 bit hexadecimal number with at least 8 bits set, none of which clash with any other mark bits in use on the system. [Default: 0xff000000]' format: int32 type: integer iptablesNATOutgoingInterfaceFilter: type: string iptablesPostWriteCheckInterval: description: 'IptablesPostWriteCheckInterval is the period after Felix has done a write to the dataplane that it schedules an extra read back in order to check the write was not clobbered by another process. This should only occur if another application on the system doesn''t respect the iptables lock. [Default: 1s]' type: string iptablesRefreshInterval: description: 'IptablesRefreshInterval is the period at which Felix re-checks the IP sets in the dataplane to ensure that no other process has accidentally broken Calico''s rules. Set to 0 to disable IP sets refresh. Note: the default for this value is lower than the other refresh intervals as a workaround for a Linux kernel bug that was fixed in kernel version 4.11. If you are using v4.11 or greater you may want to set this to, a higher value to reduce Felix CPU usage. [Default: 10s]' type: string ipv6Support: type: boolean kubeNodePortRanges: description: 'KubeNodePortRanges holds list of port ranges used for service node ports. Only used if felix detects kube-proxy running in ipvs mode. Felix uses these ranges to separate host and workload traffic. [Default: 30000:32767].' items: anyOf: - type: integer - type: string pattern: ^.* x-kubernetes-int-or-string: true type: array logFilePath: description: 'LogFilePath is the full path to the Felix log. Set to none to disable file logging. [Default: /var/log/calico/felix.log]' type: string logPrefix: description: 'LogPrefix is the log prefix that Felix uses when rendering LOG rules. [Default: calico-packet]' type: string logSeverityFile: description: 'LogSeverityFile is the log severity above which logs are sent to the log file. [Default: Info]' type: string logSeverityScreen: description: 'LogSeverityScreen is the log severity above which logs are sent to the stdout. [Default: Info]' type: string logSeveritySys: description: 'LogSeveritySys is the log severity above which logs are sent to the syslog. Set to None for no logging to syslog. [Default: Info]' type: string maxIpsetSize: type: integer metadataAddr: description: 'MetadataAddr is the IP address or domain name of the server that can answer VM queries for cloud-init metadata. In OpenStack, this corresponds to the machine running nova-api (or in Ubuntu, nova-api-metadata). A value of none (case insensitive) means that Felix should not set up any NAT rule for the metadata path. [Default: 127.0.0.1]' type: string metadataPort: description: 'MetadataPort is the port of the metadata server. This, combined with global.MetadataAddr (if not ''None''), is used to set up a NAT rule, from 169.254.169.254:80 to MetadataAddr:MetadataPort. In most cases this should not need to be changed [Default: 8775].' type: integer mtuIfacePattern: description: MTUIfacePattern is a regular expression that controls which interfaces Felix should scan in order to calculate the host's MTU. This should not match workload interfaces (usually named cali...). type: string natOutgoingAddress: description: NATOutgoingAddress specifies an address to use when performing source NAT for traffic in a natOutgoing pool that is leaving the network. By default the address used is an address on the interface the traffic is leaving on (ie it uses the iptables MASQUERADE target) type: string natPortRange: anyOf: - type: integer - type: string description: NATPortRange specifies the range of ports that is used for port mapping when doing outgoing NAT. When unset the default behavior of the network stack is used. pattern: ^.* x-kubernetes-int-or-string: true netlinkTimeout: type: string openstackRegion: description: 'OpenstackRegion is the name of the region that a particular Felix belongs to. In a multi-region Calico/OpenStack deployment, this must be configured somehow for each Felix (here in the datamodel, or in felix.cfg or the environment on each compute node), and must match the [calico] openstack_region value configured in neutron.conf on each node. [Default: Empty]' type: string policySyncPathPrefix: description: 'PolicySyncPathPrefix is used to by Felix to communicate policy changes to external services, like Application layer policy. [Default: Empty]' type: string prometheusGoMetricsEnabled: description: 'PrometheusGoMetricsEnabled disables Go runtime metrics collection, which the Prometheus client does by default, when set to false. This reduces the number of metrics reported, reducing Prometheus load. [Default: true]' type: boolean prometheusMetricsEnabled: description: 'PrometheusMetricsEnabled enables the Prometheus metrics server in Felix if set to true. [Default: false]' type: boolean prometheusMetricsHost: description: 'PrometheusMetricsHost is the host that the Prometheus metrics server should bind to. [Default: empty]' type: string prometheusMetricsPort: description: 'PrometheusMetricsPort is the TCP port that the Prometheus metrics server should bind to. [Default: 9091]' type: integer prometheusProcessMetricsEnabled: description: 'PrometheusProcessMetricsEnabled disables process metrics collection, which the Prometheus client does by default, when set to false. This reduces the number of metrics reported, reducing Prometheus load. [Default: true]' type: boolean prometheusWireGuardMetricsEnabled: description: 'PrometheusWireGuardMetricsEnabled disables wireguard metrics collection, which the Prometheus client does by default, when set to false. This reduces the number of metrics reported, reducing Prometheus load. [Default: true]' type: boolean removeExternalRoutes: description: Whether or not to remove device routes that have not been programmed by Felix. Disabling this will allow external applications to also add device routes. This is enabled by default which means we will remove externally added routes. type: boolean reportingInterval: description: 'ReportingInterval is the interval at which Felix reports its status into the datastore or 0 to disable. Must be non-zero in OpenStack deployments. [Default: 30s]' type: string reportingTTL: description: 'ReportingTTL is the time-to-live setting for process-wide status reports. [Default: 90s]' type: string routeRefreshInterval: description: 'RouteRefreshInterval is the period at which Felix re-checks the routes in the dataplane to ensure that no other process has accidentally broken Calico''s rules. Set to 0 to disable route refresh. [Default: 90s]' type: string routeSource: description: 'RouteSource configures where Felix gets its routing information. - WorkloadIPs: use workload endpoints to construct routes. - CalicoIPAM: the default - use IPAM data to construct routes.' type: string routeTableRange: description: Calico programs additional Linux route tables for various purposes. RouteTableRange specifies the indices of the route tables that Calico should use. properties: max: type: integer min: type: integer required: - max - min type: object serviceLoopPrevention: description: 'When service IP advertisement is enabled, prevent routing loops to service IPs that are not in use, by dropping or rejecting packets that do not get DNAT''d by kube-proxy. Unless set to "Disabled", in which case such routing loops continue to be allowed. [Default: Drop]' type: string sidecarAccelerationEnabled: description: 'SidecarAccelerationEnabled enables experimental sidecar acceleration [Default: false]' type: boolean usageReportingEnabled: description: 'UsageReportingEnabled reports anonymous Calico version number and cluster size to projectcalico.org. Logs warnings returned by the usage server. For example, if a significant security vulnerability has been discovered in the version of Calico being used. [Default: true]' type: boolean usageReportingInitialDelay: description: 'UsageReportingInitialDelay controls the minimum delay before Felix makes a report. [Default: 300s]' type: string usageReportingInterval: description: 'UsageReportingInterval controls the interval at which Felix makes reports. [Default: 86400s]' type: string useInternalDataplaneDriver: type: boolean vxlanEnabled: type: boolean vxlanMTU: description: 'VXLANMTU is the MTU to set on the tunnel device. See Configuring MTU [Default: 1440]' type: integer vxlanPort: type: integer vxlanVNI: type: integer wireguardEnabled: description: 'WireguardEnabled controls whether Wireguard is enabled. [Default: false]' type: boolean wireguardHostEncryptionEnabled: description: 'WireguardHostEncryptionEnabled controls whether Wireguard host-to-host encryption is enabled. [Default: false]' type: boolean wireguardInterfaceName: description: 'WireguardInterfaceName specifies the name to use for the Wireguard interface. [Default: wg.calico]' type: string wireguardListeningPort: description: 'WireguardListeningPort controls the listening port used by Wireguard. [Default: 51820]' type: integer wireguardMTU: description: 'WireguardMTU controls the MTU on the Wireguard interface. See Configuring MTU [Default: 1420]' type: integer wireguardRoutingRulePriority: description: 'WireguardRoutingRulePriority controls the priority value to use for the Wireguard routing rule. [Default: 99]' type: integer xdpEnabled: description: 'XDPEnabled enables XDP acceleration for suitable untracked incoming deny rules. [Default: true]' type: boolean xdpRefreshInterval: description: 'XDPRefreshInterval is the period at which Felix re-checks all XDP state to ensure that no other process has accidentally broken Calico''s BPF maps or attached programs. Set to 0 to disable XDP refresh. [Default: 90s]' type: string type: object type: object served: true storage: true status: acceptedNames: kind: FelixConfiguration listKind: FelixConfigurationList plural: felixconfigurations singular: felixconfiguration conditions: - lastTransitionTime: "2021-05-17T09:12:39Z" message: 'spec.preserveUnknownFields: Invalid value: true: must be false' reason: Violations status: "True" type: NonStructuralSchema - lastTransitionTime: "2021-05-17T09:12:39Z" message: no conflicts found reason: NoConflicts status: "True" type: NamesAccepted - lastTransitionTime: "2021-05-17T09:12:39Z" message: the initial names have been accepted reason: InitialNamesAccepted status: "True" type: Established storedVersions: - v1

$ kubectl get crd ipamblocks.crd.projectcalico.org -oyaml apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: annotations: kubectl.kubernetes.io/last-applied-configuration: | {"apiVersion":"apiextensions.k8s.io/v1","kind":"CustomResourceDefinition","metadata":{"annotations":{},"labels":{"kubeone.io/addon":"cni-canal"},"name":"ipamblocks.crd.projectcalico.org"},"spec":{"group":"crd.projectcalico.org","names":{"kind":"IPAMBlock","listKind":"IPAMBlockList","plural":"ipamblocks","singular":"ipamblock"},"scope":"Cluster","versions":[{"name":"v1","schema":{"openAPIV3Schema":{"properties":{"apiVersion":{"description":"APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources","type":"string"},"kind":{"description":"Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds","type":"string"},"metadata":{"type":"object"},"spec":{"description":"IPAMBlockSpec contains the specification for an IPAMBlock resource.","properties":{"affinity":{"type":"string"},"allocations":{"items":{"nullable":true,"type":"integer"},"type":"array"},"attributes":{"items":{"properties":{"handle_id":{"type":"string"},"secondary":{"additionalProperties":{"type":"string"},"type":"object"}},"type":"object"},"type":"array"},"cidr":{"type":"string"},"deleted":{"type":"boolean"},"strictAffinity":{"type":"boolean"},"unallocated":{"items":{"type":"integer"},"type":"array"}},"required":["allocations","attributes","cidr","strictAffinity","unallocated"],"type":"object"}},"type":"object"}},"served":true,"storage":true}]},"status":{"acceptedNames":{"kind":"","plural":""},"conditions":[],"storedVersions":[]}} creationTimestamp: "2021-05-17T09:12:40Z" generation: 2 labels: kubeone.io/addon: cni-canal kubeone.io/component: canal name: ipamblocks.crd.projectcalico.org resourceVersion: "97102086" uid: f8a5a261-dbdd-4036-b96b-fccd34f5851c spec: conversion: strategy: None group: crd.projectcalico.org names: kind: IPAMBlock listKind: IPAMBlockList plural: ipamblocks singular: ipamblock preserveUnknownFields: true scope: Cluster versions: - name: v1 schema: openAPIV3Schema: properties: apiVersion: description: 'APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources' type: string kind: description: 'Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds' type: string metadata: type: object spec: description: IPAMBlockSpec contains the specification for an IPAMBlock resource. properties: affinity: type: string allocations: items: nullable: true type: integer type: array attributes: items: properties: handle_id: type: string secondary: additionalProperties: type: string type: object type: object type: array cidr: type: string deleted: type: boolean strictAffinity: type: boolean unallocated: items: type: integer type: array required: - allocations - attributes - cidr - strictAffinity - unallocated type: object type: object served: true storage: true status: acceptedNames: kind: IPAMBlock listKind: IPAMBlockList plural: ipamblocks singular: ipamblock conditions: - lastTransitionTime: "2021-05-17T09:12:40Z" message: no conflicts found reason: NoConflicts status: "True" type: NamesAccepted - lastTransitionTime: "2021-05-17T09:12:40Z" message: the initial names have been accepted reason: InitialNamesAccepted status: "True" type: Established - lastTransitionTime: "2021-05-17T09:12:40Z" message: 'spec.preserveUnknownFields: Invalid value: true: must be false' reason: Violations status: "True" type: NonStructuralSchema storedVersions: - v1

kubermatic / kubeone

Failed to load and apply embedded addon (cni-canal?) with kubeone 1.4.4 #2099

What happened?

Expected behavior

How to reproduce the issue?

What KubeOne version are you using?

Provide your KubeOneCluster manifest here (if applicable)

What cloud provider are you running on?

What operating system are you running in your cluster?

Additional information