Persistent Volume size exceeds instance capability and releasing Jenkins fails.

[marioparris-qless]

https://github.com/GoogleCloudPlatform/continuous-deployment-on-kubernetes/blob/c67a9f9ca7d6959118de3f1d25a273a6c045d493/jenkins/values.yaml#L17

I think this is a bug. In the tutorial, we create 2 n1-standard-2 instances. These have a maximum persistent capability of 64GB, yet the values.yaml describes 100GB. See https://cloud.google.com/compute/docs/machine-types

When I've tried to release using the 100GB, my pod never starts and the persistent volume repeatedly fails to schedule. When I try lower values like 25GB and 8GB (the default) it succeeds.

(Admittedly, I'm new to GKE, so my understanding may be wrong)

[viglesiasce]

Hey @marioparris-qless!

Those volumes are provisioned on the fly and connected to the Jenkins master via GCE persistent disk.

Can you show kubectl describe pod <jenkins-pod-name>?

[viglesiasce]

It could also be a quota issue that youve hit in your project.

[marioparris-qless]

😥 You're probably right about the quota. I'm on the Free Trial, so perhaps I'm hitting a quota.

I've just tried it with 100GB and it works.

Name:           invincible-bobcat-jenkins-7b9cfb6669-sdxz5
Namespace:      default
Node:           gke-jenkins-default-pool-56453cac-n8vl/10.142.0.3
Start Time:     Fri, 24 Aug 2018 19:25:42 -0400
Labels:         app=invincible-bobcat-jenkins
                chart=jenkins-0.16.20
                component=invincible-bobcat-jenkins-master
                heritage=Tiller
                pod-template-hash=3657962225
                release=invincible-bobcat
Annotations:    checksum/config=fb934e337c5c274243e1d7be271ce6328ac37fee29477cb934594ab92ebd226e
Status:         Running
IP:             10.28.1.10
Controlled By:  ReplicaSet/invincible-bobcat-jenkins-7b9cfb6669
Init Containers:
  copy-default-config:
    Container ID:  docker://823e8a489d72b008e440064ab45fa27cf149f94c1826102273185f788784ee7c
    Image:         jenkins/jenkins:lts
    Image ID:      docker-pullable://jenkins/jenkins@sha256:8d4a5ad1ed1b5c4b5c5a73ba5341eaa18953afabac2703a77921ae1abd5cd822
    Port:          <none>
    Host Port:     <none>
    Command:
      sh
      /var/jenkins_config/apply_config.sh
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Fri, 24 Aug 2018 19:25:57 -0400
      Finished:     Fri, 24 Aug 2018 19:26:03 -0400
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     2
      memory:  2Gi
    Requests:
      cpu:        50m
      memory:     256Mi
    Environment:  <none>
    Mounts:
      /usr/share/jenkins/ref/secrets/ from secrets-dir (rw)
      /var/jenkins_config from jenkins-config (rw)
      /var/jenkins_home from jenkins-home (rw)
      /var/jenkins_plugins from plugin-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from invincible-bobcat-jenkins-token-lj28j (ro)
Containers:
  invincible-bobcat-jenkins:
    Container ID:  docker://97f6c1764295074fb9a8302d673c96557b3058825662a75f1dd1e391d2b94b7e
    Image:         jenkins/jenkins:lts
    Image ID:      docker-pullable://jenkins/jenkins@sha256:8d4a5ad1ed1b5c4b5c5a73ba5341eaa18953afabac2703a77921ae1abd5cd822
    Ports:         8080/TCP, 50000/TCP
    Host Ports:    0/TCP, 0/TCP
    Args:
      --argumentsRealm.passwd.$(ADMIN_USER)=$(ADMIN_PASSWORD)
      --argumentsRealm.roles.$(ADMIN_USER)=admin
    State:          Running
      Started:      Fri, 24 Aug 2018 19:26:03 -0400
    Ready:          True
    Restart Count:  0
    Requests:
      cpu:      1
      memory:   3500Mi
    Liveness:   http-get http://:http/login delay=90s timeout=5s period=10s #success=1 #failure=12
    Readiness:  http-get http://:http/login delay=60s timeout=1s period=10s #success=1 #failure=3
    Environment:
      JAVA_TOOL_OPTIONS:  -Xms3500m -Xmx3500m
      JENKINS_OPTS:       
      ADMIN_PASSWORD:     <set to the key 'jenkins-admin-password' in secret 'invincible-bobcat-jenkins'>  Optional: false
      ADMIN_USER:         <set to the key 'jenkins-admin-user' in secret 'invincible-bobcat-jenkins'>      Optional: false
    Mounts:
      /usr/share/jenkins/ref/plugins/ from plugin-dir (rw)
      /usr/share/jenkins/ref/secrets/ from secrets-dir (rw)
      /var/jenkins_config from jenkins-config (ro)
      /var/jenkins_home from jenkins-home (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from invincible-bobcat-jenkins-token-lj28j (ro)
Conditions:
  Type           Status
  Initialized    True 
  Ready          True 
  PodScheduled   True 
Volumes:
  jenkins-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      invincible-bobcat-jenkins
    Optional:  false
  plugin-dir:
    Type:    EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:  
  secrets-dir:
    Type:    EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:  
  jenkins-home:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  invincible-bobcat-jenkins
    ReadOnly:   false
  invincible-bobcat-jenkins-token-lj28j:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  invincible-bobcat-jenkins-token-lj28j
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason                 Age              From                                             Message
  ----     ------                 ----             ----                                             -------
  Warning  FailedScheduling       2m (x4 over 2m)  default-scheduler                                PersistentVolumeClaim is not bound: "invincible-bobcat-jenkins" (repeated 2 times)
  Normal   Scheduled              2m               default-scheduler                                Successfully assigned invincible-bobcat-jenkins-7b9cfb6669-sdxz5 to gke-jenkins-default-pool-56453cac-n8vl
  Normal   SuccessfulMountVolume  2m               kubelet, gke-jenkins-default-pool-56453cac-n8vl  MountVolume.SetUp succeeded for volume "secrets-dir"
  Normal   SuccessfulMountVolume  2m               kubelet, gke-jenkins-default-pool-56453cac-n8vl  MountVolume.SetUp succeeded for volume "plugin-dir"
  Normal   SuccessfulMountVolume  2m               kubelet, gke-jenkins-default-pool-56453cac-n8vl  MountVolume.SetUp succeeded for volume "jenkins-config"
  Normal   SuccessfulMountVolume  2m               kubelet, gke-jenkins-default-pool-56453cac-n8vl  MountVolume.SetUp succeeded for volume "invincible-bobcat-jenkins-token-lj28j"
  Normal   SuccessfulMountVolume  2m               kubelet, gke-jenkins-default-pool-56453cac-n8vl  MountVolume.SetUp succeeded for volume "pvc-0050b4a4-a7f5-11e8-9525-42010af0004c"
  Normal   Pulling                2m               kubelet, gke-jenkins-default-pool-56453cac-n8vl  pulling image "jenkins/jenkins:lts"
  Normal   Pulled                 2m               kubelet, gke-jenkins-default-pool-56453cac-n8vl  Successfully pulled image "jenkins/jenkins:lts"
  Normal   Created                2m               kubelet, gke-jenkins-default-pool-56453cac-n8vl  Created container
  Normal   Started                2m               kubelet, gke-jenkins-default-pool-56453cac-n8vl  Started container
  Normal   Pulling                2m               kubelet, gke-jenkins-default-pool-56453cac-n8vl  pulling image "jenkins/jenkins:lts"
  Normal   Pulled                 2m               kubelet, gke-jenkins-default-pool-56453cac-n8vl  Successfully pulled image "jenkins/jenkins:lts"
  Normal   Created                2m               kubelet, gke-jenkins-default-pool-56453cac-n8vl  Created container
  Normal   Started                2m               kubelet, gke-jenkins-default-pool-56453cac-n8vl  Started container

I'm puzzled. Earlier today I had a few attempts go into status CrashLoopBackOff and that's when I tried using lower values like 25GB and the default 8GB that worked.

I suppose that the scheduler warning isn't as alarming as I thought it was.

Warning  FailedScheduling       2m (x4 over 2m)  default-scheduler                                PersistentVolumeClaim is not bound: "invincible-bobcat-jenkins" (repeated 2 times)