kube-apiserver: unable to sync kubernetes service: Endpoints "kubernetes" is invalid, may not be in the loopback range

[typokign]

RKE version: v1.0.4

Docker version: (docker version,docker info preferred)

Client:
 Debug Mode: false

Server:
 Containers: 18
  Running: 7
  Paused: 0
  Stopped: 11
 Images: 4
 Server Version: 19.03.8
 Storage Driver: overlay2
  Backing Filesystem: <unknown>
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 7ad184331fa3e55e52b890ea95e65ba581ae3429
 runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd
 init version: fec3683
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 4.19.0-6-amd64
 Operating System: Debian GNU/Linux 10 (buster)
 OSType: linux
 Architecture: x86_64
 CPUs: 2
 Total Memory: 7.606GiB
 Name: anschluss
 ID: FMQD:7KLS:6ZKY:UIWM:4S6D:YURK:EVJ6:QI66:SUYL:6QI5:R7ES:GA6Q
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No swap limit support

Operating system and kernel: (cat /etc/os-release, uname -r preferred)

PRETTY_NAME="Debian GNU/Linux 10 (buster)"
NAME="Debian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

4.19.0-6-amd64

Type/provider of hosts: (VirtualBox/Bare-metal/AWS/GCE/DO) Hetzner Cloud VPS

cluster.yml file:

nodes:
  - user: kube
    role:
      - controlplane
      - etcd
      - worker

# Set the name of the Kubernetes cluster
cluster_name: cluster

ignore_docker_version: true

# The Kubernetes version used. The default versions of Kubernetes
# are tied to specific versions of the system images.
#
# For RKE v0.2.x and below, the map of Kubernetes versions and their system images is
# located here:
# https://github.com/rancher/types/blob/release/v2.2/apis/management.cattle.io/v3/k8s_defaults.go
#
# For RKE v0.3.0 and above, the map of Kubernetes versions and their system images is
# located here:
# https://github.com/rancher/kontainer-driver-metadata/blob/master/rke/k8s_rke_system_images.go
#
# In case the kubernetes_version and kubernetes image in
# system_images are defined, the system_images configuration
# will take precedence over kubernetes_version.
kubernetes_version: 'v1.17.2-rancher1-2'

services:
  kube-api:
    # IP range for any services created on Kubernetes
    # This must match the service_cluster_ip_range in kube-controller
    service_cluster_ip_range: 10.43.0.0/16
    # Expose a different port range for NodePort services
    service_node_port_range: 1025-65535
    pod_security_policy: false
    # Add additional arguments to the kubernetes API server
    # This WILL OVERRIDE any existing defaults
    extra_args:
      # Enable audit log to stdout
      audit-log-path: "-"
    # Encrypt secrets at rest
    secrets_encryption_config:
      enabled: true
  # Note for Rancher 2 users: If you are configuring Cluster Options
  # using a Config File when creating Rancher Launched Kubernetes,
  # the names of services should contain underscores only:
  # `kube_controller`. This only applies to Rancher v2.0.5 and v2.0.6.
  kube-controller:
    # CIDR pool used to assign IP addresses to pods in the cluster
    cluster_cidr: 10.42.0.0/16
    # IP range for any services created on Kubernetes
    # This must match the service_cluster_ip_range in kube-api
    service_cluster_ip_range: 10.43.0.0/16
  kubelet:
    # Base domain for the cluster
    cluster_domain: cluster.local
    # IP address for the DNS service endpoint
    cluster_dns_server: 10.43.0.10
    # Fail if swap is on
    fail_swap_on: true
    # Set max pods to 250 instead of default 110
    extra_args:
      max-pods: 1000
    # Optionally define additional volume binds to a service

# Kubernetes Authorization mode
# Use `mode: rbac` to enable RBAC
# Use `mode: none` to disable authorization
authorization:
  mode: rbac

network:
  plugin: canal

# Add-ons are deployed using kubernetes jobs. RKE will give
# up on trying to get the job status after this timeout in seconds..
addon_job_timeout: 30

# Currently only nginx ingress provider is supported.
# To disable ingress controller, set `provider: none`
# `node_selector` controls ingress placement and is optional
ingress:
  provider: nginx
  node_selector:
    app: ingress

Steps to Reproduce:

With this cluster.yml, set DOCKER_API_VERSION=1.40 and run rke up --local --config /etc/cluster.yml.

Results:

The node is bootstrapped and RKE logs seem healthy, up until Executing deploy job rke-network-plugin.

Running kubectl -n kube-system get pods, there are several of these deploy jobs failing, with the logs: The connection to the server 10.43.0.1:443 was refused - did you specify the right host or port?

This IP should be the IP of the kubernetes service, but this service does not exist in kube-system, though it does exist in default.

Checking the logs of the kube-apiserver I found:

W0323 19:57:24.563941       1 lease.go:224] Resetting endpoints for master service "kubernetes" to [127.0.0.1]
E0323 19:57:24.565445       1 controller.go:222] unable to sync kubernetes service: Endpoints "kubernetes" is invalid: subsets[0].addresses[0].ip: Invalid value: "127.0.0.1": may not be in the loopback range (127.0.0.0/8)

[typokign]

This is happening with both Flannel and Canal network plugins.

Node conditions are fine:

kubectl get nodes -o go-template='{{range .items}}{{$node := .}}{{range .status.conditions}}{{$node.metadata.name}}{{": "}}{{.type}}{{":"}}{{.status}}{{"\n"}}{{end}}{{end}}'

localhost: MemoryPressure:False
localhost: DiskPressure:False
localhost: PIDPressure:False
localhost: Ready:False

[stale[bot]]

This issue/PR has been automatically marked as stale because it has not had activity (commit/comment/label) for 60 days. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.

[nvanheuverzwijn]

It seems that you cannot start kube-api-server on 127.0.0.0/8 ip address. You need to assign a private ip (192.168.0.2 for example) to your host.