search

jenkinsci / google-kubernetes-engine-plugin

The Google Kubernetes Engine (GKE) Plugin allows you to deploy build artifacts to Kubernetes clusters running in GKE with Jenkins.
https://plugins.jenkins.io/google-kubernetes-engine/
Apache License 2.0
36 stars 66 forks source link

Deploy verification prints out too much on each iteration of checking #67

Open viglesiasce opened 5 years ago

viglesiasce commented 5 years ago

The full deployment JSON is being printed out during deployment validation. We should remove this and only add it back when we have a way to enable "debug" mode or log levels for this step.

Example log snippet here (it is 100x longer than this):

Cloning the remote Git repository
Cloning repository https://github.com/viglesiasce/sample-app
 > git init /home/jenkins/workspace/gke-plugin-pipeline # timeout=10
Fetching upstream changes from https://github.com/viglesiasce/sample-app
 > git --version # timeout=10
 > git fetch --tags --force --progress https://github.com/viglesiasce/sample-app +refs/heads/*:refs/remotes/origin/* # timeout=10
Checking out Revision 8834988fcdc3d6878b5ca6a4fcee4688da7c7597 (refs/remotes/origin/master)
Commit message: "Simplify Jenkinsfile"
[Pipeline] container
 > git config remote.origin.url https://github.com/viglesiasce/sample-app # timeout=10
 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
 > git config remote.origin.url https://github.com/viglesiasce/sample-app # timeout=10
Fetching upstream changes from https://github.com/viglesiasce/sample-app
 > git fetch --tags --force --progress https://github.com/viglesiasce/sample-app +refs/heads/*:refs/remotes/origin/* # timeout=10
 > git rev-parse refs/remotes/origin/master^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/master^{commit} # timeout=10
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 8834988fcdc3d6878b5ca6a4fcee4688da7c7597 # timeout=10
 > git branch -a -v --no-abbrev # timeout=10
 > git checkout -b master 8834988fcdc3d6878b5ca6a4fcee4688da7c7597 # timeout=10
 > git rev-list --no-walk 8834988fcdc3d6878b5ca6a4fcee4688da7c7597 # timeout=10
[Pipeline] {
[Pipeline] step
Executing shell script inside container [kubectl] of pod [default-jgrpt]
Executing command: "kubectl" "--kubeconfig" "/home/jenkins/workspace/gke-plugin-pipeline/.kube1437773298307485887config" "config" "use-context" "gke_vic-goog_us-west2-a_battleground" 
exit
Switched to context "gke_vic-goog_us-west2-a_battleground".
Executing shell script inside container [kubectl] of pod [default-jgrpt]
Executing command: "kubectl" "--kubeconfig" "/home/jenkins/workspace/gke-plugin-pipeline/.kube1437773298307485887config" "get" "--namespace" "default" "namespace" "default" "-o" "json" 
exit
{
    "apiVersion": "v1",
    "kind": "Namespace",
    "metadata": {
        "creationTimestamp": "2019-04-30T20:25:39Z",
        "name": "default",
        "resourceVersion": "27",
        "selfLink": "/api/v1/namespaces/default",
        "uid": "1ed7993b-6b86-11e9-9d9c-42010aa80079"
    },
    "spec": {
        "finalizers": [
            "kubernetes"
        ]
    },
    "status": {
        "phase": "Active"
    }
}
Executing shell script inside container [kubectl] of pod [default-jgrpt]
Executing command: "kubectl" "--kubeconfig" "/home/jenkins/workspace/gke-plugin-pipeline/.kube8595372679533270474config" "config" "use-context" "gke_vic-goog_us-west2-a_battleground" 
exit
Switched to context "gke_vic-goog_us-west2-a_battleground".
Executing shell script inside container [kubectl] of pod [default-jgrpt]
Executing command: "kubectl" "--kubeconfig" "/home/jenkins/workspace/gke-plugin-pipeline/.kube8595372679533270474config" "apply" "--namespace" "default" "-f" "/home/jenkins/workspace/gke-plugin-pipeline/k8s/staging" 
exit
deployment.extensions/gceme-backend-staging created
deployment.extensions/gceme-frontend-staging created
Verifying manifests: /home/jenkins/workspace/gke-plugin-pipeline/k8s/staging
Verifying 2 objects: 
Verifying: extensions/v1beta1/Deployment: gceme-backend-staging 
Executing shell script inside container [kubectl] of pod [default-jgrpt]
Executing command: "kubectl" "--kubeconfig" "/home/jenkins/workspace/gke-plugin-pipeline/.kube5019519250675376435config" "config" "use-context" "gke_vic-goog_us-west2-a_battleground" 
exit
Switched to context "gke_vic-goog_us-west2-a_battleground".
Executing shell script inside container [kubectl] of pod [default-jgrpt]
Executing command: "kubectl" "--kubeconfig" "/home/jenkins/workspace/gke-plugin-pipeline/.kube5019519250675376435config" "get" "--namespace" "default" "deployment" "gceme-backend-staging" "-o" "json" 
exit
{
    "apiVersion": "extensions/v1beta1",
    "kind": "Deployment",
    "metadata": {
        "annotations": {
            "deployment.kubernetes.io/revision": "1",
            "kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"extensions/v1beta1\",\"kind\":\"Deployment\",\"metadata\":{\"annotations\":{},\"labels\":{\"app.kubernetes.io/managed-by\":\"graphite-jenkins-gke\"},\"name\":\"gceme-backend-staging\",\"namespace\":\"default\"},\"spec\":{\"replicas\":1,\"template\":{\"metadata\":{\"labels\":{\"app\":\"gceme\",\"env\":\"staging\",\"role\":\"backend\"},\"name\":\"backend\"},\"spec\":{\"containers\":[{\"command\":[\"sh\",\"-c\",\"app -port=8080\"],\"image\":\"gcr.io/cloud-solutions-images/gceme:1.0.0\",\"imagePullPolicy\":\"Always\",\"name\":\"backend\",\"ports\":[{\"containerPort\":8080,\"name\":\"backend\"}],\"readinessProbe\":{\"httpGet\":{\"path\":\"/healthz\",\"port\":8080}},\"resources\":{\"limits\":{\"cpu\":\"100m\",\"memory\":\"500Mi\"}}}]}}}}\n"
        },
        "creationTimestamp": "2019-05-03T21:10:40Z",
        "generation": 1,
        "labels": {
            "app.kubernetes.io/managed-by": "graphite-jenkins-gke"
        },
        "name": "gceme-backend-staging",
        "namespace": "default",
        "resourceVersion": "769147",
        "selfLink": "/apis/extensions/v1beta1/namespaces/default/deployments/gceme-backend-staging",
        "uid": "e7a7cbc4-6de7-11e9-9a1a-42010aa8002d"
    },
    "spec": {
        "progressDeadlineSeconds": 2147483647,
        "replicas": 1,
        "revisionHistoryLimit": 10,
        "selector": {
            "matchLabels": {
                "app": "gceme",
                "env": "staging",
                "role": "backend"
            }
        },
        "strategy": {
            "rollingUpdate": {
                "maxSurge": 1,
                "maxUnavailable": 1
            },
            "type": "RollingUpdate"
        },
        "template": {
            "metadata": {
                "creationTimestamp": null,
                "labels": {
                    "app": "gceme",
                    "env": "staging",
                    "role": "backend"
                },
                "name": "backend"
            },
            "spec": {
                "containers": [
                    {
                        "command": [
                            "sh",
                            "-c",
                            "app -port=8080"
                        ],
                        "image": "gcr.io/cloud-solutions-images/gceme:1.0.0",
                        "imagePullPolicy": "Always",
                        "name": "backend",
                        "ports": [
                            {
                                "containerPort": 8080,
                                "name": "backend",
                                "protocol": "TCP"
                            }
                        ],
                        "readinessProbe": {
                            "failureThreshold": 3,
                            "httpGet": {
                                "path": "/healthz",
                                "port": 8080,
                                "scheme": "HTTP"
                            },
                            "periodSeconds": 10,
                            "successThreshold": 1,
                            "timeoutSeconds": 1
                        },
                        "resources": {
                            "limits": {
                                "cpu": "100m",
                                "memory": "500Mi"
                            }
                        },
                        "terminationMessagePath": "/dev/termination-log",
                        "terminationMessagePolicy": "File"
                    }
                ],
                "dnsPolicy": "ClusterFirst",
                "restartPolicy": "Always",
                "schedulerName": "default-scheduler",
                "securityContext": {},
                "terminationGracePeriodSeconds": 30
            }
        }
    },
    "status": {
        "conditions": [
            {
                "lastTransitionTime": "2019-05-03T21:10:40Z",
                "lastUpdateTime": "2019-05-03T21:10:40Z",
                "message": "Deployment has minimum availability.",
                "reason": "MinimumReplicasAvailable",
                "status": "True",
                "type": "Available"
            }
        ],
        "observedGeneration": 1,
        "replicas": 1,
        "unavailableReplicas": 1,
        "updatedReplicas": 1
    }
}
Verifying: extensions/v1beta1/Deployment: gceme-frontend-staging 
Executing shell script inside container [kubectl] of pod [default-jgrpt]
Executing command: "kubectl" "--kubeconfig" "/home/jenkins/workspace/gke-plugin-pipeline/.kube4621200451206075642config" "config" "use-context" "gke_vic-goog_us-west2-a_battleground" 
exit
Switched to context "gke_vic-goog_us-west2-a_battleground".
Executing shell script inside container [kubectl] of pod [default-jgrpt]
Executing command: "kubectl" "--kubeconfig" "/home/jenkins/workspace/gke-plugin-pipeline/.kube4621200451206075642config" "get" "--namespace" "default" "deployment" "gceme-frontend-staging" "-o" "json" 
exit
stephenashank commented 5 years ago

It seems like there is different behavior depending on the environment. Here's the console output from my build from the jenkins UI.

Started by user unknown or anonymous
Building in workspace /usr/local/google/home/stephenshank/Documents/google-kubernetes-engine-plugin/work/jobs/hello/workspace
No credentials specified
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/jenkinsci/google-kubernetes-engine-plugin.git # timeout=10
Fetching upstream changes from https://github.com/jenkinsci/google-kubernetes-engine-plugin.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/jenkinsci/google-kubernetes-engine-plugin.git +refs/heads/*:refs/remotes/origin/*
 > git rev-parse refs/remotes/origin/develop^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/develop^{commit} # timeout=10
Checking out Revision db14b1db74e4296c09f4cc36a3745520cc27d2fb (refs/remotes/origin/develop)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f db14b1db74e4296c09f4cc36a3745520cc27d2fb
Commit message: "Merge pull request #57 from stephenashank/feature/issue56instructionfix"
 > git rev-list --no-walk db14b1db74e4296c09f4cc36a3745520cc27d2fb # timeout=10
$ kubectl --kubeconfig /usr/local/google/home/stephenshank/Documents/google-kubernetes-engine-plugin/work/jobs/hello/workspace/.kube1027573903809928450config config use-context gke_graphitestephenshank_us-central1-a_standard-cluster-1
$ kubectl --kubeconfig /usr/local/google/home/stephenshank/Documents/google-kubernetes-engine-plugin/work/jobs/hello/workspace/.kube1027573903809928450config get --namespace default namespace default -o json
$ kubectl --kubeconfig /usr/local/google/home/stephenshank/Documents/google-kubernetes-engine-plugin/work/jobs/hello/workspace/.kube736431482384517979config config use-context gke_graphitestephenshank_us-central1-a_standard-cluster-1
$ kubectl --kubeconfig /usr/local/google/home/stephenshank/Documents/google-kubernetes-engine-plugin/work/jobs/hello/workspace/.kube736431482384517979config apply --namespace default -f /usr/local/google/home/stephenshank/Documents/google-kubernetes-engine-plugin/work/jobs/hello/workspace/docs/resources/manifest.yaml
Verifying manifests: /usr/local/google/home/stephenshank/Documents/google-kubernetes-engine-plugin/work/jobs/hello/workspace/docs/resources/manifest.yaml
Verifying 1 objects: 
Verifying: apps/v1/Deployment: nginx-deployment 
$ kubectl --kubeconfig /usr/local/google/home/stephenshank/Documents/google-kubernetes-engine-plugin/work/jobs/hello/workspace/.kube873112613718848570config config use-context gke_graphitestephenshank_us-central1-a_standard-cluster-1
$ kubectl --kubeconfig /usr/local/google/home/stephenshank/Documents/google-kubernetes-engine-plugin/work/jobs/hello/workspace/.kube873112613718848570config get --namespace default deployment nginx-deployment -o json
Verifying: apps/v1/Deployment: nginx-deployment 
$ kubectl --kubeconfig /usr/local/google/home/stephenshank/Documents/google-kubernetes-engine-plugin/work/jobs/hello/workspace/.kube4682006455367160805config config use-context gke_graphitestephenshank_us-central1-a_standard-cluster-1
$ kubectl --kubeconfig /usr/local/google/home/stephenshank/Documents/google-kubernetes-engine-plugin/work/jobs/hello/workspace/.kube4682006455367160805config get --namespace default deployment nginx-deployment -o json
Successfully verified apps/v1/Deployment: nginx-deployment
AvailableReplicas = 3, MinimumReplicas = 3

Finished: SUCCESS

This is what came from the command line where I was running mvn hpi:run:

May 03, 2019 2:28:57 PM com.google.jenkins.plugins.k8sengine.KubernetesEngineBuilder perform
INFO: GKE Deploying, projectId: graphitestephenshank cluster: standard-cluster-1 zone: us-central1-a
May 03, 2019 2:28:59 PM com.google.jenkins.plugins.k8sengine.KubernetesEngineBuilder verify
INFO: GKE verifying deployment to, projectId: graphitestephenshank cluster: standard-cluster-1 zone: us-central1-a manifests: /usr/local/google/home/stephenshank/Documents/google-kubernetes-engine-plugin/work/jobs/hello/workspace/docs/resources/manifest.yaml
May 03, 2019 2:28:59 PM com.google.jenkins.plugins.k8sengine.KubernetesVerifiers$DeploymentVerifier verify
INFO: Verifying deployment, nginx-deployment
May 03, 2019 2:29:04 PM com.google.jenkins.plugins.k8sengine.KubernetesVerifiers$DeploymentVerifier verify
INFO: Verifying deployment, nginx-deployment
May 03, 2019 2:29:05 PM com.google.jenkins.plugins.k8sengine.VerificationTask verifyObjects
INFO: 0 error results
May 03, 2019 2:29:05 PM hudson.model.Run execute
INFO: hello #25 main build action completed: SUCCESS
viglesiasce commented 5 years ago

Was this done on an existing deployment? If not can you try to delete and re-create it with the plugin?

On Fri, May 3, 2019 at 2:34 PM Stephen Shank notifications@github.com wrote:

It seems like there is different behavior depending on the environment. Here's the console output from my build from the jenkins UI.

Started by user unknown or anonymous Building in workspace /usr/local/google/home/stephenshank/Documents/google-kubernetes-engine-plugin/work/jobs/hello/workspace No credentials specified

git rev-parse --is-inside-work-tree # timeout=10 Fetching changes from the remote Git repository git config remote.origin.url https://github.com/jenkinsci/google-kubernetes-engine-plugin.git # timeout=10 Fetching upstream changes from https://github.com/jenkinsci/google-kubernetes-engine-plugin.git git --version # timeout=10 git fetch --tags --progress https://github.com/jenkinsci/google-kubernetes-engine-plugin.git +refs/heads/:refs/remotes/origin/ git rev-parse refs/remotes/origin/develop^{commit} # timeout=10 git rev-parse refs/remotes/origin/origin/develop^{commit} # timeout=10 Checking out Revision db14b1db74e4296c09f4cc36a3745520cc27d2fb (refs/remotes/origin/develop) git config core.sparsecheckout # timeout=10 git checkout -f db14b1db74e4296c09f4cc36a3745520cc27d2fb Commit message: "Merge pull request #57 from stephenashank/feature/issue56instructionfix" git rev-list --no-walk db14b1db74e4296c09f4cc36a3745520cc27d2fb # timeout=10 $ kubectl --kubeconfig /usr/local/google/home/stephenshank/Documents/google-kubernetes-engine-plugin/work/jobs/hello/workspace/.kube1027573903809928450config config use-context gke_graphitestephenshank_us-central1-a_standard-cluster-1 $ kubectl --kubeconfig /usr/local/google/home/stephenshank/Documents/google-kubernetes-engine-plugin/work/jobs/hello/workspace/.kube1027573903809928450config get --namespace default namespace default -o json $ kubectl --kubeconfig /usr/local/google/home/stephenshank/Documents/google-kubernetes-engine-plugin/work/jobs/hello/workspace/.kube736431482384517979config config use-context gke_graphitestephenshank_us-central1-a_standard-cluster-1 $ kubectl --kubeconfig /usr/local/google/home/stephenshank/Documents/google-kubernetes-engine-plugin/work/jobs/hello/workspace/.kube736431482384517979config apply --namespace default -f /usr/local/google/home/stephenshank/Documents/google-kubernetes-engine-plugin/work/jobs/hello/workspace/docs/resources/manifest.yaml Verifying manifests: /usr/local/google/home/stephenshank/Documents/google-kubernetes-engine-plugin/work/jobs/hello/workspace/docs/resources/manifest.yaml Verifying 1 objects: Verifying: apps/v1/Deployment: nginx-deployment $ kubectl --kubeconfig /usr/local/google/home/stephenshank/Documents/google-kubernetes-engine-plugin/work/jobs/hello/workspace/.kube873112613718848570config config use-context gke_graphitestephenshank_us-central1-a_standard-cluster-1 $ kubectl --kubeconfig /usr/local/google/home/stephenshank/Documents/google-kubernetes-engine-plugin/work/jobs/hello/workspace/.kube873112613718848570config get --namespace default deployment nginx-deployment -o json Verifying: apps/v1/Deployment: nginx-deployment $ kubectl --kubeconfig /usr/local/google/home/stephenshank/Documents/google-kubernetes-engine-plugin/work/jobs/hello/workspace/.kube4682006455367160805config config use-context gke_graphitestephenshank_us-central1-a_standard-cluster-1 $ kubectl --kubeconfig /usr/local/google/home/stephenshank/Documents/google-kubernetes-engine-plugin/work/jobs/hello/workspace/.kube4682006455367160805config get --namespace default deployment nginx-deployment -o json Successfully verified apps/v1/Deployment: nginx-deployment AvailableReplicas = 3, MinimumReplicas = 3

Finished: SUCCESS

This is what came from the command line where I was running mvn hpi:run:

May 03, 2019 2:28:57 PM com.google.jenkins.plugins.k8sengine.KubernetesEngineBuilder perform INFO: GKE Deploying, projectId: graphitestephenshank cluster: standard-cluster-1 zone: us-central1-a May 03, 2019 2:28:59 PM com.google.jenkins.plugins.k8sengine.KubernetesEngineBuilder verify INFO: GKE verifying deployment to, projectId: graphitestephenshank cluster: standard-cluster-1 zone: us-central1-a manifests: /usr/local/google/home/stephenshank/Documents/google-kubernetes-engine-plugin/work/jobs/hello/workspace/docs/resources/manifest.yaml May 03, 2019 2:28:59 PM com.google.jenkins.plugins.k8sengine.KubernetesVerifiers$DeploymentVerifier verify INFO: Verifying deployment, nginx-deployment May 03, 2019 2:29:04 PM com.google.jenkins.plugins.k8sengine.KubernetesVerifiers$DeploymentVerifier verify INFO: Verifying deployment, nginx-deployment May 03, 2019 2:29:05 PM com.google.jenkins.plugins.k8sengine.VerificationTask verifyObjects INFO: 0 error results May 03, 2019 2:29:05 PM hudson.model.Run execute INFO: hello #25 main build action completed: SUCCESS

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/jenkinsci/google-kubernetes-engine-plugin/issues/67#issuecomment-489247075, or mute the thread https://github.com/notifications/unsubscribe-auth/AADEFJ5KGQEZFHJVNAI4WLTPTSVXVANCNFSM4HKXBB2Q .

stephenashank commented 5 years ago

It's possible that the Jenkins logs don't include the logs from the launcher's process. I suspect the reason you see the deployment like that is because of how we run kubectl get with the -o json flag, which is currently required for verification.

It might be possible to replace the use of kubectl get with calls to the GKE API, but that also might run into the same issues that we have with kubectl apply where we can't use the GKE API

stephenashank commented 5 years ago

No, this was not for an existing deployment

johnlabarge commented 5 years ago

I explored using the Kubernetes SDK and it did not make sense in light of the fact that we are using kubectl already and other reasons I can disclose offline.

stephenashank commented 5 years ago

Need to re-enable a subset of logs