search

fabric8io / fabric8-platform

Generates the distribution of the fabric8 microservices platform
Apache License 2.0
104 stars 66 forks source link

latest release on minishift results in failed DCs in the jenkins namespace #1102

Open jstrachan opened 6 years ago

jstrachan commented 6 years ago

I get this:

$ oc get dc
NAME                 REVISION   DESIRED   CURRENT   TRIGGERED BY
content-repository   1          1         0         config
jenkins              1          1         0         config

$oc describe dc jenkins
Name:       jenkins
Namespace:  developer-jenkins
Created:    About a minute ago
Labels:     app=jenkins
        group=io.fabric8.tenant.apps
        provider=fabric8
        version=4.0.37
Annotations:    configmap.fabric8.io/update-on-change=jenkins
        fabric8.io/git-branch=release-v4.0.37
        fabric8.io/git-commit=406a6dbfcc5da82114f2bdd4fafbd5b8dfe496d4
        fabric8.io/iconUrl=https://cdn.rawgit.com/fabric8-services/tenant-jenkins/tenant-jenkins-apps/jenkins/master/apps/jenkins/src/main/fabric8/icon.svg
        fabric8.io/metrics-path=dashboard/file/kubernetes-pods.json/?var-project=jenkins&var-version=4.0.37
        fabric8.io/scm-con-url=scm:git:git@github.com:fabric8-services/tenant-jenkins.git/tenant-jenkins-apps/jenkins
        fabric8.io/scm-devcon-url=scm:git:git@github.com:fabric8-services/tenant-jenkins.git/tenant-jenkins-apps/jenkins
        fabric8.io/scm-tag=fabric8-services/tenant-jenkins-1.0.0
        fabric8.io/scm-url=http://github.com/fabric8-services/tenant-jenkins/tenant-jenkins-apps/jenkins
        fabric8.io/target-platform=openshift
        kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"v1","kind":"DeploymentConfig","metadata":{"annotations":{"configmap.fabric8.io/update-on-change":"jenkins","fabric8.io/git-branch":"release-v4.0.37","fabric8.io/git-commit":"406a6dbfcc5da82114f2bdd4fafbd5b8dfe496d4","fabric8.io/iconUrl":"https://cdn.rawgit.com/fabric8-services/tenant-jenkins/tenant-jenkins-apps/jenkins/master/apps/jenkins/src/main/fabric8/icon.svg","fabric8.io/metrics-path":"dashboard/file/kubernetes-pods.json/?var-project=jenkins\u0026var-version=4.0.37","fabric8.io/scm-con-url":"scm:git:git@github.com:fabric8-services/tenant-jenkins.git/tenant-jenkins-apps/jenkins","fabric8.io/scm-devcon-url":"scm:git:git@github.com:fabric8-services/tenant-jenkins.git/tenant-jenkins-apps/jenkins","fabric8.io/scm-tag":"fabric8-services/tenant-jenkins-1.0.0","fabric8.io/scm-url":"http://github.com/fabric8-services/tenant-jenkins/tenant-jenkins-apps/jenkins","fabric8.io/target-platform":"openshift","maven.fabric8.io/source-url":"jar:file:/home/jenkins/workspace/bric8-tenant-jenkins_master-AGRITQKDOGHX6ZQLEFCNJVZP7VWPA5P7B6D2RTTKZ2KN4XI2LSCA/apps/jenkins/target/jenkins-4.0.37.jar!/META-INF/fabric8/openshift.yml"},"labels":{"app":"jenkins","group":"io.fabric8.tenant.apps","provider":"fabric8","version":"4.0.37"},"name":"jenkins","namespace":"developer-jenkins"},"spec":{"replicas":1,"revisionHistoryLimit":2,"selector":{"app":"jenkins","group":"io.fabric8.tenant.apps","provider":"fabric8"},"strategy":{"recreateParams":{"timeoutSeconds":10000},"type":"Recreate"},"template":{"metadata":{"annotations":{"configmap.fabric8.io/update-on-change":"jenkins","fabric8.io/git-branch":"release-v4.0.37","fabric8.io/git-commit":"406a6dbfcc5da82114f2bdd4fafbd5b8dfe496d4","fabric8.io/iconUrl":"https://cdn.rawgit.com/fabric8-services/tenant-jenkins/tenant-jenkins-apps/jenkins/master/apps/jenkins/src/main/fabric8/icon.svg","fabric8.io/metrics-path":"dashboard/file/kubernetes-pods.json/?var-project=jenkins\u0026var-version=4.0.37","fabric8.io/scm-con-url":"scm:git:git@github.com:fabric8-services/tenant-jenkins.git/tenant-jenkins-apps/jenkins","fabric8.io/scm-devcon-url":"scm:git:git@github.com:fabric8-services/tenant-jenkins.git/tenant-jenkins-apps/jenkins","fabric8.io/scm-tag":"fabric8-services/tenant-jenkins-1.0.0","fabric8.io/scm-url":"http://github.com/fabric8-services/tenant-jenkins/tenant-jenkins-apps/jenkins","fabric8.io/target-platform":"openshift"},"labels":{"app":"jenkins","group":"io.fabric8.tenant.apps","provider":"fabric8","version":"4.0.37"}},"spec":{"containers":[{"env":[{"name":"PROJECT_NAMESPACE","value":"developer"},{"name":"KUBERNETES_NAMESPACE","valueFrom":{"fieldRef":{"fieldPath":"metadata.namespace"}}},{"name":"GIT_COMMITTER_EMAIL","value":"fabric8@googlegroups.com"},{"name":"GIT_COMMITTER_NAME","value":"fabric8"},{"name":"OPENSHIFT_ENABLE_OAUTH","value":"true"},{"name":"OPENSHIFT_ENABLE_REDIRECT_PROMPT","value":"true"},{"name":"KUBERNETES_TRUST_CERTIFICATES","value":"true"},{"name":"KUBERNETES_MASTER","value":"https://kubernetes.default:443"},{"name":"JAVA_GC_OPTS","value":"-XX:+UseParallelGC -XX:MinHeapFreeRatio=5 -XX:MaxHeapFreeRatio=10 -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90"},{"name":"JAVA_OPTS","value":"-XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -Dsun.zip.disableMemoryMapping=true"},{"name":"OPENSHIFT_JENKINS_JVM_ARCH","value":"i686"},{"name":"CONTAINER_INITIAL_PERCENT","value":"0.07"},{"name":"CONTAINTER_INITIAL_PERCENT","value":"something-non-empty"},{"name":"JENKINS_OPTS","value":"-Dgroovy.use.classvalue=true"},{"name":"JAVA_HOME","value":"/etc/alternatives/java_sdk"},{"name":"MAVEN_OPTS","value":"-Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransferListener=warn"},{"name":"RECOMMENDER_API_TOKEN","valueFrom":{"secretKeyRef":{"key":"token","name":"jenkins-recommender-api-token"}}}],"image":"fabric8/jenkins-openshift:vb6fc097","imagePullPolicy":"IfNotPresent","lifecycle":{"preStop":{"exec":{"command":["sh","/var/lib/jenkins/pre-shutdown.sh"]}}},"livenessProbe":{"failureThreshold":30,"httpGet":{"path":"/login","port":8080},"initialDelaySeconds":420,"timeoutSeconds":10},"name":"jenkins","ports":[{"containerPort":50000,"name":"slave"},{"containerPort":8080,"name":"http"}],"readinessProbe":{"httpGet":{"path":"/login","port":8080},"initialDelaySeconds":10,"timeoutSeconds":10},"resources":{"limits":{"cpu":"2","memory":"512Mi"},"requests":{"cpu":"0"}},"volumeMounts":[{"mountPath":"/var/lib/jenkins","name":"jenkins-home","readOnly":false},{"mountPath":"/opt/openshift/configuration/","name":"jenkins-config"}]}],"serviceAccountName":"jenkins","volumes":[{"name":"jenkins-home","persistentVolumeClaim":{"claimName":"jenkins-home"}},{"configMap":{"name":"jenkins"},"name":"jenkins-config"}]}},"triggers":[{"type":"ConfigChange"}]}}

        maven.fabric8.io/source-url=jar:file:/home/jenkins/workspace/bric8-tenant-jenkins_master-AGRITQKDOGHX6ZQLEFCNJVZP7VWPA5P7B6D2RTTKZ2KN4XI2LSCA/apps/jenkins/target/jenkins-4.0.37.jar!/META-INF/fabric8/openshift.yml
Latest Version: 1
Selector:   app=jenkins,group=io.fabric8.tenant.apps,provider=fabric8
Replicas:   1
Triggers:   Config
Strategy:   Recreate
Template:
Pod Template:
  Labels:       app=jenkins
            group=io.fabric8.tenant.apps
            provider=fabric8
            version=4.0.37
  Annotations:      configmap.fabric8.io/update-on-change=jenkins
            fabric8.io/git-branch=release-v4.0.37
            fabric8.io/git-commit=406a6dbfcc5da82114f2bdd4fafbd5b8dfe496d4
            fabric8.io/iconUrl=https://cdn.rawgit.com/fabric8-services/tenant-jenkins/tenant-jenkins-apps/jenkins/master/apps/jenkins/src/main/fabric8/icon.svg
            fabric8.io/metrics-path=dashboard/file/kubernetes-pods.json/?var-project=jenkins&var-version=4.0.37
            fabric8.io/scm-con-url=scm:git:git@github.com:fabric8-services/tenant-jenkins.git/tenant-jenkins-apps/jenkins
            fabric8.io/scm-devcon-url=scm:git:git@github.com:fabric8-services/tenant-jenkins.git/tenant-jenkins-apps/jenkins
            fabric8.io/scm-tag=fabric8-services/tenant-jenkins-1.0.0
            fabric8.io/scm-url=http://github.com/fabric8-services/tenant-jenkins/tenant-jenkins-apps/jenkins
            fabric8.io/target-platform=openshift
  Service Account:  jenkins
  Containers:
   jenkins:
    Image:  fabric8/jenkins-openshift:vb6fc097
    Ports:  50000/TCP, 8080/TCP
    Limits:
      cpu:  2
      memory:   512Mi
    Requests:
      cpu:  0
    Liveness:   http-get http://:8080/login delay=420s timeout=10s period=10s #success=1 #failure=30
    Readiness:  http-get http://:8080/login delay=10s timeout=10s period=10s #success=1 #failure=3
    Environment:
      PROJECT_NAMESPACE:        developer
      KUBERNETES_NAMESPACE:      (v1:metadata.namespace)
      GIT_COMMITTER_EMAIL:      fabric8@googlegroups.com
      GIT_COMMITTER_NAME:       fabric8
      OPENSHIFT_ENABLE_OAUTH:       true
      OPENSHIFT_ENABLE_REDIRECT_PROMPT: true
      KUBERNETES_TRUST_CERTIFICATES:    true
      KUBERNETES_MASTER:        https://kubernetes.default:443
      JAVA_GC_OPTS:         -XX:+UseParallelGC -XX:MinHeapFreeRatio=5 -XX:MaxHeapFreeRatio=10 -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90
      JAVA_OPTS:            -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -Dsun.zip.disableMemoryMapping=true
      OPENSHIFT_JENKINS_JVM_ARCH:   i686
      CONTAINER_INITIAL_PERCENT:    0.07
      CONTAINTER_INITIAL_PERCENT:   something-non-empty
      JENKINS_OPTS:         -Dgroovy.use.classvalue=true
      JAVA_HOME:            /etc/alternatives/java_sdk
      MAVEN_OPTS:           -Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransferListener=warn
      RECOMMENDER_API_TOKEN:        <set to the key 'token' in secret 'jenkins-recommender-api-token'>  Optional: false
    Mounts:
      /opt/openshift/configuration/ from jenkins-config (rw)
      /var/lib/jenkins from jenkins-home (rw)
  Volumes:
   jenkins-home:
    Type:   PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  jenkins-home
    ReadOnly:   false
   jenkins-config:
    Type:   ConfigMap (a volume populated by a ConfigMap)
    Name:   jenkins
    Optional:   false

Deployment #1 (latest):
    Name:       jenkins-1
    Created:    about a minute ago
    Status:     Failed
    Replicas:   0 current / 0 desired
    Selector:   app=jenkins,deployment=jenkins-1,deploymentconfig=jenkins,group=io.fabric8.tenant.apps,provider=fabric8
    Labels:     app=jenkins,group=io.fabric8.tenant.apps,openshift.io/deployment-config.name=jenkins,provider=fabric8,version=4.0.37
    Pods Status:    0 Running / 0 Waiting / 0 Succeeded / 0 Failed

Events:
  FirstSeen LastSeen    Count   From                SubObjectPath   Type        Reason          Message
  --------- --------    -----   ----                -------------   --------    ------          -------
  1m        1m      1   deploymentconfig-controller         Normal      DeploymentCreated   Created new replication controller "jenkins-1" for version 1

Took a long while to figure out whats up; saw these events:

7:52:42 AM | jenkins-1-deploy | Pod | Normal | Killing | Killing container with id docker://deployment:Need to kill Pod

i.e. openshift killed the deploy pods. I guess we now need more memory / CPU?

jstrachan commented 6 years ago

that was running minishift via minishift start --vm-driver=xhyve --memory=9000 --cpus=5 --disk-size=50g too btw

jstrachan commented 6 years ago

Triggering the DeploymentConfig's by hand in the openshift console seemed to fix it.

I guess we should ditch DeploymentConfigs for the jenkins tenant!

rawlingsj commented 6 years ago

Yeah! As a first step lets just switch the jenkins DC to a Deployment.