Memory leak kube-apiserver 1.8.1

ese commented 7 years ago

/kind bug /sig api-machinery @kubernetes/sig-apimachinery-bugs

What happened: Kube-api-server is consuming more and more memory until end with all available resources

What you expected to happen: kube-api-server consuming more or less the same amount of memory since there is no change in the api requests or cluster load

How to reproduce it (as minimally and precisely as possible): Run a cluster with kops in aws (manifest provided at the end of the issue)*:

Environment:

Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.1", GitCommit:"f38e43b221d08850172a9a4ea785a86a3ffa3b3a", GitTreeState:"clean", BuildDate:"2017-10-11T23:27:35Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.1", GitCommit:"f38e43b221d08850172a9a4ea785a86a3ffa3b3a", GitTreeState:"clean", BuildDate:"2017-10-11T23:16:41Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}

Cloud provider or hardware configuration**: AWS, running api-server in masters (m4.large) a cluster of 6 nodes (m4.2xlarge) with very low load of pods
OS (e.g. from /etc/os-release): kope.io/k8s-1.7-debian-jessie-amd64-hvm-ebs-2017-07-28
Kernel (e.g. uname -a): Linux ip-172-20-11-106 4.4.78-k8s #1 SMP Fri Jul 28 01:28:39 UTC 2017 x86_64 GNU/Linux
Install tools: kops 1.8.0-alpha1
Others:

kops manifest

apiVersion: kops/v1alpha2
kind: Cluster
metadata:
  creationTimestamp: 2017-10-16T09:11:48Z
  name: [REDACTED]
spec:
  additionalPolicies:
    node: '[{"Action":["sts:AssumeRole"],"Effect":"Allow","Resource":"*"}]'
  api:
    loadBalancer:
      type: Public
  authorization:
    rbac: {}
  channel: stable
  cloudLabels:
    environment: testing
    owner: unknown
  cloudProvider: aws
  configBase: [REDACTED]
  dnsZone: [REDACTED]
  etcdClusters:
  - enableEtcdTLS: true
    etcdMembers:
    - instanceGroup: master-eu-west-1a
      name: a
    - instanceGroup: master-eu-west-1b
      name: b
    - instanceGroup: master-eu-west-1c
      name: c
    name: main
    version: 3.1.10
  - enableEtcdTLS: true
    etcdMembers:
    - instanceGroup: master-eu-west-1a
      name: a
    - instanceGroup: master-eu-west-1b
      name: b
    - instanceGroup: master-eu-west-1c
      name: c
    name: events
    version: 3.1.10
  iam:
    legacy: false
  kubeAPIServer:
    auditLogMaxAge: 10
    auditLogMaxBackups: 1
    auditLogMaxSize: 100
    auditLogPath: /var/log/kube-apiserver-audit.log
  kubelet:
    featureGates:
      ExperimentalCriticalPodAnnotation: "true"
  kubernetesApiAccess:
  - 0.0.0.0/0
  kubernetesVersion: 1.8.1
  masterInternalName: [REDACTED]
  masterPublicName: [REDACTED]
  networkCIDR: 172.20.0.0/16
  networking:
    canal: {}
  nonMasqueradeCIDR: 100.64.0.0/10
  sshAccess:
  - 0.0.0.0/0
  subnets:
  - cidr: 172.20.32.0/19
    name: eu-west-1a
    type: Private
    zone: eu-west-1a
  - cidr: 172.20.64.0/19
    name: eu-west-1b
    type: Private
    zone: eu-west-1b
  - cidr: 172.20.96.0/19
    name: eu-west-1c
    type: Private
    zone: eu-west-1c
  - cidr: 172.20.0.0/22
    name: utility-eu-west-1a
    type: Utility
    zone: eu-west-1a
  - cidr: 172.20.4.0/22
    name: utility-eu-west-1b
    type: Utility
    zone: eu-west-1b
  - cidr: 172.20.8.0/22
    name: utility-eu-west-1c
    type: Utility
    zone: eu-west-1c
  topology:
    bastion:
      bastionPublicName: [REDACTED]
    dns:
      type: Public
    masters: private
    nodes: private