search

openshift / origin

Conformance test suite for OpenShift
http://www.openshift.org
Apache License 2.0
8.49k stars 4.7k forks source link

Running v3.7.0-rc.0 on Travis fails with MountVolume.SetUp failed for volume ... : exit status 1 #17210

Closed donovanmuller closed 6 years ago

donovanmuller commented 7 years ago

Using a Travis job to run integration tests against various Origin versions, using the v3.7.0-rc.0 release fails to create/deploy any Pod with the error:

MountVolume.SetUp failed for volume "deployer-token-bt4hk" : exit status 1

Using v3.6.1 works as expected.

See https://github.com/donovanmuller/spring-cloud-deployer-openshift/issues/48 for context.

Version
$ oc version
oc v3.7.0-rc.0+e92d5c5
kubernetes v1.7.6+a08f5eeb62
features: Basic-Auth GSSAPI Kerberos SPNEGO

Server https://127.0.0.1:8443
openshift v3.7.0-rc.0+e92d5c5
kubernetes v1.7.6+a08f5eeb62
Steps To Reproduce
  1. oc cluster up --version=v3.7.0-rc.0 (see Travis config for all steps)
  2. Run integration test via Maven
  3. Pods fail to be created:
$ oc get pods
NAME                              READY     STATUS              RESTARTS   AGE
app-testsimpledeployme-1-deploy   0/1       ContainerCreating   0          20s

$ oc get events
LASTSEEN   FIRSTSEEN   COUNT     NAME                              KIND               SUBOBJECT   TYPE      REASON              SOURCE                        MESSAGE
59s        59s         1         app-testsimpledeployme-1-deploy   Pod                            Normal    Scheduled           default-scheduler             Successfully assigned app-testsimpledeployme-1-deploy to localhost
27s        59s         7         app-testsimpledeployme-1-deploy   Pod                            Warning   FailedMount         kubelet, localhost            MountVolume.SetUp failed for volume "deployer-token-g4hl2" : exit status 1
59s        59s         1         app-testsimpledeployme            DeploymentConfig               Normal    DeploymentCreated   deploymentconfig-controller   Created new replication controller "app-testsimpledeployme-1" for version 1
Current Result

Pods fail to mount volumes for any deployments/builds/etc.

Expected Result

Pods are scheduled successfully

Additional Information
$ oc adm diagnostics
[Note] Determining if client configuration exists for client/cluster diagnostics
Info:  Successfully read a client config file at '/home/travis/.kube/config'
Info:  Using context for cluster-admin access: 'scdf/127-0-0-1:8443/system:admin'

[Note] Running diagnostic: ConfigContexts[/127-0-0-1:8443/developer]
       Description: Validate client config context is complete and has connectivity

Info:  For client config context '/127-0-0-1:8443/developer':
       The server URL is 'https://127.0.0.1:8443'
       The user authentication is 'developer/127-0-0-1:8443'
       The current project is 'default'
       Successfully requested project list; has access to project(s):
         [myproject]

[Note] Running diagnostic: ConfigContexts[default/127-0-0-1:8443/system:admin]
       Description: Validate client config context is complete and has connectivity

Info:  For client config context 'default/127-0-0-1:8443/system:admin':
       The server URL is 'https://127.0.0.1:8443'
       The user authentication is 'system:admin/127-0-0-1:8443'
       The current project is 'default'
       Successfully requested project list; has access to project(s):
         [default kube-public kube-system myproject openshift openshift-infra openshift-node scdf]

[Note] Running diagnostic: DiagnosticPod
       Description: Create a pod to run diagnostics from the application standpoint

WARN:  [DCli2006 from diagnostic DiagnosticPod@openshift/origin/pkg/diagnostics/client/run_diagnostics_pod.go:157]
       Timed out preparing diagnostic pod logs for streaming, so this diagnostic cannot run.
       It is likely that the image 'openshift/origin-deployer:v3.7.0-rc.0' was not pulled and running yet.
       Last error: (*errors.StatusError[2]) container "pod-diagnostics" in pod "pod-diagnostic-test-27ds4" is waiting to start: ContainerCreating

[Note] Running diagnostic: NetworkCheck
       Description: Create a pod on all schedulable nodes and run network diagnostics from the application standpoint

Info:  Skipping network diagnostics check. Reason: Not using openshift network plugin.

[Note] Skipping diagnostic: AggregatedLogging
       Description: Check aggregated logging integration for proper configuration
       Because: No master config file was provided

[Note] Running diagnostic: ClusterRegistry
       Description: Check that there is a working Docker registry

ERROR: [DClu1006 from diagnostic ClusterRegistry@openshift/origin/pkg/diagnostics/cluster/registry.go:206]
       The "docker-registry" service exists but has no associated pods, so it
       is not available. Builds and deployments that use the registry will fail.

[Note] Running diagnostic: ClusterRoleBindings
       Description: Check that the default ClusterRoleBindings are present and contain the expected subjects

Info:  clusterrolebinding/system:controller:horizontal-pod-autoscaler has more subjects than expected.

       Use the `oc adm policy reconcile-cluster-role-bindings` command to update the role binding to remove extra subjects.

Info:  clusterrolebinding/system:controller:horizontal-pod-autoscaler has extra subject {ServiceAccount  horizontal-pod-autoscaler openshift-infra}.

Info:  clusterrolebinding/system:controller:horizontal-pod-autoscaler has more subjects than expected.

       Use the `oc adm policy reconcile-cluster-role-bindings` command to update the role binding to remove extra subjects.

Info:  clusterrolebinding/system:controller:horizontal-pod-autoscaler has extra subject {ServiceAccount  horizontal-pod-autoscaler kube-system}.

Info:  clusterrolebinding/cluster-admin has more subjects than expected.

       Use the `oc adm policy reconcile-cluster-role-bindings` command to update the role binding to remove extra subjects.

Info:  clusterrolebinding/cluster-admin has extra subject {ServiceAccount  pvinstaller default}.

[Note] Running diagnostic: ClusterRoles
       Description: Check that the default ClusterRoles are present and contain the expected permissions

[Note] Running diagnostic: ClusterRouterName
       Description: Check there is a working router

ERROR: [DClu2007 from diagnostic ClusterRouter@openshift/origin/pkg/diagnostics/cluster/router.go:157]
       The "router" DeploymentConfig exists but has no running pods, so it
       is not available. Apps will not be externally accessible via the router.

[Note] Running diagnostic: MasterNode
       Description: Check if master is also running node (for Open vSwitch)

WARN:  [DClu3004 from diagnostic MasterNode@openshift/origin/pkg/diagnostics/cluster/master_node.go:162]
       Unable to find a node matching the cluster server IP.
       This may indicate the master is not also running a node, and is unable
       to proxy to pods over the Open vSwitch SDN.

[Note] Skipping diagnostic: MetricsApiProxy
       Description: Check the integrated heapster metrics can be reached via the API proxy
       Because: The heapster service does not exist in the openshift-infra project at this time,
       so it is not available for the Horizontal Pod Autoscaler to use as a source of metrics.

[Note] Running diagnostic: NodeDefinitions
       Description: Check node records on master

[Note] Running diagnostic: RouteCertificateValidation
       Description: Check all route certificates for certificates that might be rejected by extended validation.

[Note] Skipping diagnostic: ServiceExternalIPs
       Description: Check for existing services with ExternalIPs that are disallowed by master config
       Because: No master config file was detected

[Note] Summary of diagnostics execution (version v3.7.0-rc.0+e92d5c5):
[Note] Warnings seen: 2
[Note] Errors seen: 2

Diagnostic Pod fails to deploy because of this issue:

$  oc get events
...
1m         1m          1         pod-diagnostic-test-mr2dr         Pod                            Normal    Scheduled           default-scheduler             Successfully assigned pod-diagnostic-test-mr2dr to localhost
32s        1m          8         pod-diagnostic-test-mr2dr         Pod                            Warning   FailedMount         kubelet, localhost            MountVolume.SetUp failed for volume "default-token-r52s2" : exit status 1
$ oc get all -o json -n scdf
{
    "apiVersion": "v1",
    "items": [
        {
            "apiVersion": "v1",
            "kind": "DeploymentConfig",
            "metadata": {
                "annotations": {
                    "spring-cloud-deployer/redeploy-timestamp": "1510027157087"
                },
                "creationTimestamp": "2017-11-07T03:59:16Z",
                "generation": 1,
                "labels": {
                    "spring-app-id": "app-testsimpledeployme",
                    "spring-deployment-id": "app-testsimpledeployme"
                },
                "name": "app-testsimpledeployme",
                "namespace": "scdf",
                "resourceVersion": "1101",
                "selfLink": "/oapi/v1/namespaces/scdf/deploymentconfigs/app-testsimpledeployme",
                "uid": "06205cef-c370-11e7-9d53-42010a14002f"
            },
            "spec": {
                "replicas": 1,
                "selector": {
                    "spring-app-id": "app-testsimpledeployme",
                    "spring-deployment-id": "app-testsimpledeployme"
                },
                "strategy": {
                    "activeDeadlineSeconds": 21600,
                    "resources": {
                        "limits": {
                            "cpu": "0",
                            "memory": "0"
                        }
                    },
                    "rollingParams": {
                        "intervalSeconds": 1,
                        "maxSurge": "25%",
                        "maxUnavailable": "25%",
                        "timeoutSeconds": 600,
                        "updatePeriodSeconds": 1
                    },
                    "type": "Rolling"
                },
                "template": {
                    "metadata": {
                        "creationTimestamp": null,
                        "labels": {
                            "spring-app-id": "app-testsimpledeployme",
                            "spring-deployment-id": "app-testsimpledeployme"
                        }
                    },
                    "spec": {
                        "containers": [
                            {
                                "env": [
                                    {
                                        "name": "SPRING_CLOUD_APPLICATION_GUID",
                                        "value": "${HOSTNAME}"
                                    },
                                    {
                                        "name": "INSTANCE_INDEX",
                                        "value": "0"
                                    },
                                    {
                                        "name": "SPRING_APPLICATION_INDEX",
                                        "value": "0"
                                    }
                                ],
                                "image": "springcloud/spring-cloud-deployer-spi-test-app:latest",
                                "imagePullPolicy": "IfNotPresent",
                                "livenessProbe": {
                                    "failureThreshold": 3,
                                    "httpGet": {
                                        "path": "/health",
                                        "port": 8080,
                                        "scheme": "HTTP"
                                    },
                                    "initialDelaySeconds": 20,
                                    "periodSeconds": 60,
                                    "successThreshold": 1,
                                    "timeoutSeconds": 2
                                },
                                "name": "app-testsimpledeployme",
                                "ports": [
                                    {
                                        "containerPort": 8080,
                                        "protocol": "TCP"
                                    }
                                ],
                                "readinessProbe": {
                                    "failureThreshold": 3,
                                    "httpGet": {
                                        "path": "/info",
                                        "port": 8080,
                                        "scheme": "HTTP"
                                    },
                                    "initialDelaySeconds": 20,
                                    "periodSeconds": 10,
                                    "successThreshold": 1,
                                    "timeoutSeconds": 2
                                },
                                "resources": {
                                    "limits": {
                                        "cpu": "0",
                                        "memory": "0"
                                    }
                                },
                                "terminationMessagePath": "/dev/termination-log",
                                "terminationMessagePolicy": "File"
                            }
                        ],
                        "dnsPolicy": "ClusterFirst",
                        "restartPolicy": "Always",
                        "schedulerName": "default-scheduler",
                        "securityContext": {},
                        "terminationGracePeriodSeconds": 30
                    }
                },
                "test": false,
                "triggers": [
                    {
                        "type": "ConfigChange"
                    }
                ]
            },
            "status": {
                "availableReplicas": 0,
                "conditions": [
                    {
                        "lastTransitionTime": "2017-11-07T03:59:16Z",
                        "lastUpdateTime": "2017-11-07T03:59:16Z",
                        "message": "Deployment config does not have minimum availability.",
                        "status": "False",
                        "type": "Available"
                    },
                    {
                        "lastTransitionTime": "2017-11-07T03:59:16Z",
                        "lastUpdateTime": "2017-11-07T03:59:16Z",
                        "message": "replication controller \"app-testsimpledeployme-1\" is waiting for pod \"app-testsimpledeployme-1-deploy\" to run",
                        "status": "Unknown",
                        "type": "Progressing"
                    }
                ],
                "details": {
                    "causes": [
                        {
                            "type": "ConfigChange"
                        }
                    ],
                    "message": "config change"
                },
                "latestVersion": 1,
                "observedGeneration": 1,
                "replicas": 0,
                "unavailableReplicas": 0,
                "updatedReplicas": 0
            }
        },
        {
            "apiVersion": "v1",
            "kind": "Pod",
            "metadata": {
                "annotations": {
                    "openshift.io/deployment.name": "app-testsimpledeployme-1",
                    "openshift.io/scc": "restricted"
                },
                "creationTimestamp": "2017-11-07T03:59:16Z",
                "labels": {
                    "openshift.io/deployer-pod-for.name": "app-testsimpledeployme-1"
                },
                "name": "app-testsimpledeployme-1-deploy",
                "namespace": "scdf",
                "ownerReferences": [
                    {
                        "apiVersion": "v1",
                        "kind": "ReplicationController",
                        "name": "app-testsimpledeployme-1",
                        "uid": "06256f34-c370-11e7-9d53-42010a14002f"
                    }
                ],
                "resourceVersion": "1089",
                "selfLink": "/api/v1/namespaces/scdf/pods/app-testsimpledeployme-1-deploy",
                "uid": "062c1108-c370-11e7-9d53-42010a14002f"
            },
            "spec": {
                "activeDeadlineSeconds": 21600,
                "containers": [
                    {
                        "env": [
                            {
                                "name": "KUBERNETES_MASTER",
                                "value": "https://127.0.0.1:8443"
                            },
                            {
                                "name": "OPENSHIFT_MASTER",
                                "value": "https://127.0.0.1:8443"
                            },
                            {
                                "name": "BEARER_TOKEN_FILE",
                                "value": "/var/run/secrets/kubernetes.io/serviceaccount/token"
                            },
                            {
                                "name": "OPENSHIFT_CA_DATA",
                                "value": "-----BEGIN CERTIFICATE-----\nMIIC6jCCAdKgAwIBAgIBATANBgkqhkiG9w0BAQsFADAmMSQwIgYDVQQDDBtvcGVu\nc2hpZnQtc2lnbmVyQDE1MTAwMjcwODkwHhcNMTcxMTA3MDM1ODA5WhcNMjIxMTA2\nMDM1ODEwWjAmMSQwIgYDVQQDDBtvcGVuc2hpZnQtc2lnbmVyQDE1MTAwMjcwODkw\nggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDnBGS7ApMPtzCn+mgsRQ8H\ng+f2NIq/8sj7AtOFjVyv5Xc/YNhoZQWDQ+0G2C85Hlgi9EitTEw9ifschrr+jzQO\napZet/IQPoTmcqVEFBZon9121TOz/yPln9w5WRg7TQgmllL3glK6WdZemsC4EjqG\nMcdVxPcUEdmM7GEAEPVZN6aLgz2d1N15Dm0qHI8coY/PnqPfpW8tpkOiLUJPZFpe\nibuTxK5Z4XA1wpkWvTFC8IcCdmUKezH++nWAjylPTHss47NzghtSioiN66UlOz+Q\naCRpn319De6daRtQ88rMwECXHcg41uXTS1AKE+/SrP7k9MW+cvlUESguPZvg+Ypf\nAgMBAAGjIzAhMA4GA1UdDwEB/wQEAwICpDAPBgNVHRMBAf8EBTADAQH/MA0GCSqG\nSIb3DQEBCwUAA4IBAQB6paV4Qlg4DfknWJUxITxTMMHvlkF1QJW/Irdy+NKWlHFP\nP8Gol/JeiEkyZTYZgEzcaetXLm8DkWyBR9NnAGn/37txMbCv/0ETXv1GdJcNWLL0\nTjwxcSi61rIbOHHro8Ayu3ZI8AbO0mNsWYiSqOjNekCsJZcU5khhZS13IaMDqh7x\n1kMCi23ACUNC8xgJaOcC7jaqfcxXQIsh89hR6G4z1BIfnNMiBjVKpl/D8v8C0J5G\nLVqrg8sBY/08aIiioejZbbMYGpjW/mY61dgal5zX83V19BPtW2tJAt+Si20y3yvE\nG+InPTx9A1G8qqksTqxV+WoUNF7Q62tszWXLQaXq\n-----END CERTIFICATE-----\n"
                            },
                            {
                                "name": "OPENSHIFT_DEPLOYMENT_NAME",
                                "value": "app-testsimpledeployme-1"
                            },
                            {
                                "name": "OPENSHIFT_DEPLOYMENT_NAMESPACE",
                                "value": "scdf"
                            }
                        ],
                        "image": "openshift/origin-deployer:v3.7.0-rc.0",
                        "imagePullPolicy": "IfNotPresent",
                        "name": "deployment",
                        "resources": {
                            "limits": {
                                "cpu": "0",
                                "memory": "0"
                            },
                            "requests": {
                                "cpu": "0",
                                "memory": "0"
                            }
                        },
                        "securityContext": {
                            "capabilities": {
                                "drop": [
                                    "KILL",
                                    "MKNOD",
                                    "SETGID",
                                    "SETUID"
                                ]
                            },
                            "privileged": false,
                            "runAsUser": 1000070000,
                            "seLinuxOptions": {
                                "level": "s0:c8,c7"
                            }
                        },
                        "terminationMessagePath": "/dev/termination-log",
                        "terminationMessagePolicy": "File",
                        "volumeMounts": [
                            {
                                "mountPath": "/var/run/secrets/kubernetes.io/serviceaccount",
                                "name": "deployer-token-gzpdf",
                                "readOnly": true
                            }
                        ]
                    }
                ],
                "dnsPolicy": "ClusterFirst",
                "imagePullSecrets": [
                    {
                        "name": "deployer-dockercfg-shb2k"
                    }
                ],
                "nodeName": "localhost",
                "restartPolicy": "Never",
                "schedulerName": "default-scheduler",
                "securityContext": {
                    "fsGroup": 1000070000,
                    "seLinuxOptions": {
                        "level": "s0:c8,c7"
                    }
                },
                "serviceAccount": "deployer",
                "serviceAccountName": "deployer",
                "terminationGracePeriodSeconds": 10,
                "volumes": [
                    {
                        "name": "deployer-token-gzpdf",
                        "secret": {
                            "defaultMode": 420,
                            "secretName": "deployer-token-gzpdf"
                        }
                    }
                ]
            },
            "status": {
                "conditions": [
                    {
                        "lastProbeTime": null,
                        "lastTransitionTime": "2017-11-07T03:59:16Z",
                        "status": "True",
                        "type": "Initialized"
                    },
                    {
                        "lastProbeTime": null,
                        "lastTransitionTime": "2017-11-07T03:59:16Z",
                        "message": "containers with unready status: [deployment]",
                        "reason": "ContainersNotReady",
                        "status": "False",
                        "type": "Ready"
                    },
                    {
                        "lastProbeTime": null,
                        "lastTransitionTime": "2017-11-07T03:59:16Z",
                        "status": "True",
                        "type": "PodScheduled"
                    }
                ],
                "containerStatuses": [
                    {
                        "image": "openshift/origin-deployer:v3.7.0-rc.0",
                        "imageID": "",
                        "lastState": {},
                        "name": "deployment",
                        "ready": false,
                        "restartCount": 0,
                        "state": {
                            "waiting": {
                                "reason": "ContainerCreating"
                            }
                        }
                    }
                ],
                "hostIP": "10.20.0.47",
                "phase": "Pending",
                "qosClass": "BestEffort",
                "startTime": "2017-11-07T03:59:16Z"
            }
        },
        {
            "apiVersion": "v1",
            "kind": "ReplicationController",
            "metadata": {
                "annotations": {
                    "kubectl.kubernetes.io/desired-replicas": "1",
                    "openshift.io/deployer-pod.created-at": "2017-11-07 03:59:16 +0000 UTC",
                    "openshift.io/deployer-pod.name": "app-testsimpledeployme-1-deploy",
                    "openshift.io/deployment-config.latest-version": "1",
                    "openshift.io/deployment-config.name": "app-testsimpledeployme",
                    "openshift.io/deployment.phase": "Pending",
                    "openshift.io/deployment.replicas": "0",
                    "openshift.io/deployment.status-reason": "config change",
                    "openshift.io/encoded-deployment-config": "{\"kind\":\"DeploymentConfig\",\"apiVersion\":\"v1\",\"metadata\":{\"name\":\"app-testsimpledeployme\",\"namespace\":\"scdf\",\"selfLink\":\"/apis/apps.openshift.io/v1/namespaces/scdf/deploymentconfigs/app-testsimpledeployme\",\"uid\":\"06205cef-c370-11e7-9d53-42010a14002f\",\"resourceVersion\":\"1081\",\"generation\":1,\"creationTimestamp\":\"2017-11-07T03:59:16Z\",\"labels\":{\"spring-app-id\":\"app-testsimpledeployme\",\"spring-deployment-id\":\"app-testsimpledeployme\"}},\"spec\":{\"strategy\":{\"type\":\"Rolling\",\"rollingParams\":{\"updatePeriodSeconds\":1,\"intervalSeconds\":1,\"timeoutSeconds\":600,\"maxUnavailable\":\"25%\",\"maxSurge\":\"25%\"},\"resources\":{\"limits\":{\"cpu\":\"0\",\"memory\":\"0\"}},\"activeDeadlineSeconds\":21600},\"triggers\":[{\"type\":\"ConfigChange\"}],\"replicas\":1,\"test\":false,\"selector\":{\"spring-app-id\":\"app-testsimpledeployme\",\"spring-deployment-id\":\"app-testsimpledeployme\"},\"template\":{\"metadata\":{\"creationTimestamp\":null,\"labels\":{\"spring-app-id\":\"app-testsimpledeployme\",\"spring-deployment-id\":\"app-testsimpledeployme\"}},\"spec\":{\"containers\":[{\"name\":\"app-testsimpledeployme\",\"image\":\"springcloud/spring-cloud-deployer-spi-test-app:latest\",\"ports\":[{\"containerPort\":8080,\"protocol\":\"TCP\"}],\"env\":[{\"name\":\"SPRING_CLOUD_APPLICATION_GUID\",\"value\":\"${HOSTNAME}\"},{\"name\":\"INSTANCE_INDEX\",\"value\":\"0\"},{\"name\":\"SPRING_APPLICATION_INDEX\",\"value\":\"0\"}],\"resources\":{\"limits\":{\"cpu\":\"0\",\"memory\":\"0\"}},\"livenessProbe\":{\"httpGet\":{\"path\":\"/health\",\"port\":8080,\"scheme\":\"HTTP\"},\"initialDelaySeconds\":20,\"timeoutSeconds\":2,\"periodSeconds\":60,\"successThreshold\":1,\"failureThreshold\":3},\"readinessProbe\":{\"httpGet\":{\"path\":\"/info\",\"port\":8080,\"scheme\":\"HTTP\"},\"initialDelaySeconds\":20,\"timeoutSeconds\":2,\"periodSeconds\":10,\"successThreshold\":1,\"failureThreshold\":3},\"terminationMessagePath\":\"/dev/termination-log\",\"terminationMessagePolicy\":\"File\",\"imagePullPolicy\":\"IfNotPresent\"}],\"restartPolicy\":\"Always\",\"terminationGracePeriodSeconds\":30,\"dnsPolicy\":\"ClusterFirst\",\"securityContext\":{},\"schedulerName\":\"default-scheduler\"}}},\"status\":{\"latestVersion\":1,\"observedGeneration\":1,\"replicas\":0,\"updatedReplicas\":0,\"availableReplicas\":0,\"unavailableReplicas\":0,\"details\":{\"message\":\"config change\",\"causes\":[{\"type\":\"ConfigChange\"}]},\"conditions\":[{\"type\":\"Available\",\"status\":\"False\",\"lastUpdateTime\":\"2017-11-07T03:59:16Z\",\"lastTransitionTime\":\"2017-11-07T03:59:16Z\",\"message\":\"Deployment config does not have minimum availability.\"}]}}\n"
                },
                "creationTimestamp": "2017-11-07T03:59:16Z",
                "generation": 1,
                "labels": {
                    "openshift.io/deployment-config.name": "app-testsimpledeployme",
                    "spring-app-id": "app-testsimpledeployme",
                    "spring-deployment-id": "app-testsimpledeployme"
                },
                "name": "app-testsimpledeployme-1",
                "namespace": "scdf",
                "ownerReferences": [
                    {
                        "apiVersion": "apps.openshift.io/v1",
                        "blockOwnerDeletion": true,
                        "controller": true,
                        "kind": "DeploymentConfig",
                        "name": "app-testsimpledeployme",
                        "uid": "06205cef-c370-11e7-9d53-42010a14002f"
                    }
                ],
                "resourceVersion": "1090",
                "selfLink": "/api/v1/namespaces/scdf/replicationcontrollers/app-testsimpledeployme-1",
                "uid": "06256f34-c370-11e7-9d53-42010a14002f"
            },
            "spec": {
                "replicas": 0,
                "selector": {
                    "deployment": "app-testsimpledeployme-1",
                    "deploymentconfig": "app-testsimpledeployme",
                    "spring-app-id": "app-testsimpledeployme",
                    "spring-deployment-id": "app-testsimpledeployme"
                },
                "template": {
                    "metadata": {
                        "annotations": {
                            "openshift.io/deployment-config.latest-version": "1",
                            "openshift.io/deployment-config.name": "app-testsimpledeployme",
                            "openshift.io/deployment.name": "app-testsimpledeployme-1"
                        },
                        "creationTimestamp": null,
                        "labels": {
                            "deployment": "app-testsimpledeployme-1",
                            "deploymentconfig": "app-testsimpledeployme",
                            "spring-app-id": "app-testsimpledeployme",
                            "spring-deployment-id": "app-testsimpledeployme"
                        }
                    },
                    "spec": {
                        "containers": [
                            {
                                "env": [
                                    {
                                        "name": "SPRING_CLOUD_APPLICATION_GUID",
                                        "value": "${HOSTNAME}"
                                    },
                                    {
                                        "name": "INSTANCE_INDEX",
                                        "value": "0"
                                    },
                                    {
                                        "name": "SPRING_APPLICATION_INDEX",
                                        "value": "0"
                                    }
                                ],
                                "image": "springcloud/spring-cloud-deployer-spi-test-app:latest",
                                "imagePullPolicy": "IfNotPresent",
                                "livenessProbe": {
                                    "failureThreshold": 3,
                                    "httpGet": {
                                        "path": "/health",
                                        "port": 8080,
                                        "scheme": "HTTP"
                                    },
                                    "initialDelaySeconds": 20,
                                    "periodSeconds": 60,
                                    "successThreshold": 1,
                                    "timeoutSeconds": 2
                                },
                                "name": "app-testsimpledeployme",
                                "ports": [
                                    {
                                        "containerPort": 8080,
                                        "protocol": "TCP"
                                    }
                                ],
                                "readinessProbe": {
                                    "failureThreshold": 3,
                                    "httpGet": {
                                        "path": "/info",
                                        "port": 8080,
                                        "scheme": "HTTP"
                                    },
                                    "initialDelaySeconds": 20,
                                    "periodSeconds": 10,
                                    "successThreshold": 1,
                                    "timeoutSeconds": 2
                                },
                                "resources": {
                                    "limits": {
                                        "cpu": "0",
                                        "memory": "0"
                                    }
                                },
                                "terminationMessagePath": "/dev/termination-log",
                                "terminationMessagePolicy": "File"
                            }
                        ],
                        "dnsPolicy": "ClusterFirst",
                        "restartPolicy": "Always",
                        "schedulerName": "default-scheduler",
                        "securityContext": {},
                        "terminationGracePeriodSeconds": 30
                    }
                }
            },
            "status": {
                "observedGeneration": 1,
                "replicas": 0
            }
        },
        {
            "apiVersion": "v1",
            "kind": "Service",
            "metadata": {
                "annotations": {
                    "service.alpha.openshift.io/dependencies": "[]"
                },
                "creationTimestamp": "2017-11-07T03:59:16Z",
                "labels": {
                    "spring-app-id": "app-testsimpledeployme",
                    "spring-deployment-id": "app-testsimpledeployme"
                },
                "name": "app-testsimpledeployme",
                "namespace": "scdf",
                "resourceVersion": "1095",
                "selfLink": "/api/v1/namespaces/scdf/services/app-testsimpledeployme",
                                    }
                                ],
                                "readinessProbe": {
                                    "failureThreshold": 3,
                                    "httpGet": {
                                        "path": "/info",
                                        "port": 8080,
                                        "scheme": "HTTP"
                                    },
                                    "initialDelaySeconds": 20,
                                    "periodSeconds": 10,
                                    "successThreshold": 1,
                                    "timeoutSeconds": 2
                                },
                                "resources": {
                                    "limits": {
                                        "cpu": "0",
                                        "memory": "0"
                                    }
                                },
                                "terminationMessagePath": "/dev/termination-log",
                                "terminationMessagePolicy": "File"
                            }
                        ],
                        "dnsPolicy": "ClusterFirst",
                        "restartPolicy": "Always",
                        "schedulerName": "default-scheduler",
                        "securityContext": {},
                        "terminationGracePeriodSeconds": 30
                    }
                }
            },
            "status": {
                "observedGeneration": 1,
                "replicas": 0
            }
        },
        {
            "apiVersion": "v1",
            "kind": "Service",
            "metadata": {
                "annotations": {
                    "service.alpha.openshift.io/dependencies": "[]"
                },
                "creationTimestamp": "2017-11-07T03:59:16Z",
                "labels": {
                    "spring-app-id": "app-testsimpledeployme",
                    "spring-deployment-id": "app-testsimpledeployme"
                },
                "name": "app-testsimpledeployme",
                "namespace": "scdf",
                "resourceVersion": "1095",
                "selfLink": "/api/v1/namespaces/scdf/services/app-testsimpledeployme",
                "uid": "0669f851-c370-11e7-9d53-42010a14002f"
            },
            "spec": {
                "clusterIP": "172.30.126.130",
                "ports": [
                    {
                        "port": 8080,
                        "protocol": "TCP",
                        "targetPort": 8080
                    }
                ],
                "selector": {
                    "spring-app-id": "app-testsimpledeployme",
                    "spring-deployment-id": "app-testsimpledeployme"
                },
                "sessionAffinity": "None",
                "type": "ClusterIP"
            },
            "status": {
                "loadBalancer": {}
            }
        }
    ],
    "kind": "List",
    "metadata": {
        "resourceVersion": "",
        "selfLink": ""
    }
}
$ docker logs origin
...
E1107 04:01:25.306896    6117 nestedpendingoperations.go:264] Operation for "\"kubernetes.io/secret/062c1108-c370-11e7-9d53-42010a14002f-deployer-token-gzpdf\" (\"062c1108-c370-11e7-9d53-42010a14002f\")" failed. No retries permitted until 2017-11-07 04:03:27.30686189 +0000 UTC (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "deployer-token-gzpdf" (UniqueName: "kubernetes.io/secret/062c1108-c370-11e7-9d53-42010a14002f-deployer-token-gzpdf") pod "app-testsimpledeployme-1-deploy" (UID: "062c1108-c370-11e7-9d53-42010a14002f") : exit status 1
W1107 04:01:29.704521    6117 status_manager.go:448] Failed to update status for pod "pod-diagnostic-test-27ds4_scdf(09658c01-c370-11e7-9d53-42010a14002f)": Operation cannot be fulfilled on pods "pod-diagnostic-test-27ds4": StorageError: invalid object, Code: 4, Key: /kubernetes.io/pods/scdf/pod-diagnostic-test-27ds4, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 09658c01-c370-11e7-9d53-42010a14002f, UID in object meta:
I1107 04:01:29.870446    6117 reconciler.go:290] Volume detached for volume "default-token-7xqfn" (UniqueName: "kubernetes.io/secret/09658c01-c370-11e7-9d53-42010a14002f-default-token-7xqfn") on node "localhost" DevicePath ""
E1107 04:01:30.663073    6117 kubelet_volumes.go:128] Orphaned pod "09658c01-c370-11e7-9d53-42010a14002f" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
I1107 04:01:31.881302    6117 reconciler.go:186] operationExecutor.UnmountVolume started for volume "default-token-7xqfn" (UniqueName: "kubernetes.io/secret/09658c01-c370-11e7-9d53-42010a14002f-default-token-7xqfn") pod "09658c01-c370-11e7-9d53-42010a14002f" (UID: "09658c01-c370-11e7-9d53-42010a14002f")
I1107 04:01:31.905126    6117 operation_generator.go:532] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09658c01-c370-11e7-9d53-42010a14002f-default-token-7xqfn" (OuterVolumeSpecName: "default-token-7xqfn") pod "09658c01-c370-11e7-9d53-42010a14002f" (UID: "09658c01-c370-11e7-9d53-42010a14002f"). InnerVolumeSpecName "default-token-7xqfn". PluginName "kubernetes.io/secret", VolumeGidValue ""
I1107 04:01:31.982029    6117 reconciler.go:290] Volume detached for volume "default-token-7xqfn" (UniqueName: "kubernetes.io/secret/09658c01-c370-11e7-9d53-42010a14002f-default-token-7xqfn") on node "localhost" DevicePath ""
E1107 04:02:46.724106    6117 nestedpendingoperations.go:264] Operation for "\"kubernetes.io/secret/eded16ae-c36f-11e7-9d53-42010a14002f-pvinstaller-token-rtc9s\" (\"eded16ae-c36f-11e7-9d53-42010a14002f\")" failed. No retries permitted until 2017-11-07 04:04:48.724062748 +0000 UTC (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "pvinstaller-token-rtc9s" (UniqueName: "kubernetes.io/secret/eded16ae-c36f-11e7-9d53-42010a14002f-pvinstaller-token-rtc9s") pod "persistent-volume-setup-5v9r2" (UID: "eded16ae-c36f-11e7-9d53-42010a14002f") : exit status 1
E1107 04:02:48.521920    6117 nestedpendingoperations.go:264] Operation for "\"kubernetes.io/secret/eefdbbda-c36f-11e7-9d53-42010a14002f-deployer-token-fn26t\" (\"eefdbbda-c36f-11e7-9d53-42010a14002f\")" failed. No retries permitted until 2017-11-07 04:04:50.521881089 +0000 UTC (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "deployer-token-fn26t" (UniqueName: "kubernetes.io/secret/eefdbbda-c36f-11e7-9d53-42010a14002f-deployer-token-fn26t") pod "router-1-deploy" (UID: "eefdbbda-c36f-11e7-9d53-42010a14002f") : exit status 1
E1107 04:02:48.621384    6117 nestedpendingoperations.go:264] Operation for "\"kubernetes.io/secret/ef0ec675-c36f-11e7-9d53-42010a14002f-deployer-token-fn26t\" (\"ef0ec675-c36f-11e7-9d53-42010a14002f\")" failed. No retries permitted until 2017-11-07 04:04:50.621341774 +0000 UTC (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "deployer-token-fn26t" (UniqueName: "kubernetes.io/secret/ef0ec675-c36f-11e7-9d53-42010a14002f-deployer-token-fn26t") pod "docker-registry-1-deploy" (UID: "ef0ec675-c36f-11e7-9d53-42010a14002f") : exit status 1
E1107 04:02:54.659123    6117 kubelet.go:1594] Unable to mount volumes for pod "persistent-volume-setup-5v9r2_default(eded16ae-c36f-11e7-9d53-42010a14002f)": timeout expired waiting for volumes to attach/mount for pod "default"/"persistent-volume-setup-5v9r2". list of unattached/unmounted volumes=[pvinstaller-token-rtc9s]; skipping pod
E1107 04:02:54.659238    6117 pod_workers.go:186] Error syncing pod eded16ae-c36f-11e7-9d53-42010a14002f ("persistent-volume-setup-5v9r2_default(eded16ae-c36f-11e7-9d53-42010a14002f)"), skipping: timeout expired waiting for volumes to attach/mount for pod "default"/"persistent-volume-setup-5v9r2". list of unattached/unmounted volumes=[pvinstaller-token-rtc9s]
E1107 04:02:55.659182    6117 kubelet.go:1594] Unable to mount volumes for pod "docker-registry-1-deploy_default(ef0ec675-c36f-11e7-9d53-42010a14002f)": timeout expired waiting for volumes to attach/mount for pod "default"/"docker-registry-1-deploy". list of unattached/unmounted volumes=[deployer-token-fn26t]; skipping pod
E1107 04:02:55.659215    6117 pod_workers.go:186] Error syncing pod ef0ec675-c36f-11e7-9d53-42010a14002f ("docker-registry-1-deploy_default(ef0ec675-c36f-11e7-9d53-42010a14002f)"), skipping: timeout expired waiting for volumes to attach/mount for pod "default"/"docker-registry-1-deploy". list of unattached/unmounted volumes=[deployer-token-fn26t]
E1107 04:02:57.659358    6117 kubelet.go:1594] Unable to mount volumes for pod "router-1-deploy_default(eefdbbda-c36f-11e7-9d53-42010a14002f)": timeout expired waiting for volumes to attach/mount for pod "default"/"router-1-deploy". list of unattached/unmounted volumes=[deployer-token-fn26t]; skipping pod
E1107 04:02:57.659393    6117 pod_workers.go:186] Error syncing pod eefdbbda-c36f-11e7-9d53-42010a14002f ("router-1-deploy_default(eefdbbda-c36f-11e7-9d53-42010a14002f)"), skipping: timeout expired waiting for volumes to attach/mount for pod "default"/"router-1-deploy". list of unattached/unmounted volumes=[deployer-token-fn26t]
E1107 04:03:27.372278    6117 nestedpendingoperations.go:264] Operation for "\"kubernetes.io/secret/062c1108-c370-11e7-9d53-42010a14002f-deployer-token-gzpdf\" (\"062c1108-c370-11e7-9d53-42010a14002f\")" failed. No retries permitted until 2017-11-07 04:05:29.372254495 +0000 UTC (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "deployer-token-gzpdf" (UniqueName: "kubernetes.io/secret/062c1108-c370-11e7-9d53-42010a14002f-deployer-token-gzpdf") pod "app-testsimpledeployme-1-deploy" (UID: "062c1108-c370-11e7-9d53-42010a14002f") : exit status 1
E1107 04:03:33.659422    6117 kubelet.go:1594] Unable to mount volumes for pod "app-testsimpledeployme-1-deploy_scdf(062c1108-c370-11e7-9d53-42010a14002f)": timeout expired waiting for volumes to attach/mount for pod "scdf"/"app-testsimpledeployme-1-deploy". list of unattached/unmounted volumes=[deployer-token-gzpdf]; skipping pod
E1107 04:03:33.659475    6117 pod_workers.go:186] Error syncing pod 062c1108-c370-11e7-9d53-42010a14002f ("app-testsimpledeployme-1-deploy_scdf(062c1108-c370-11e7-9d53-42010a14002f)"), skipping: timeout expired waiting for volumes to attach/mount for pod "scdf"/"app-testsimpledeployme-1-deploy". list of unattached/unmounted volumes=[deployer-token-gzpdf]
E1107 04:04:48.791253    6117 nestedpendingoperations.go:264] Operation for "\"kubernetes.io/secret/eded16ae-c36f-11e7-9d53-42010a14002f-pvinstaller-token-rtc9s\" (\"eded16ae-c36f-11e7-9d53-42010a14002f\")" failed. No retries permitted until 2017-11-07 04:06:50.79121944 +0000 UTC (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "pvinstaller-token-rtc9s" (UniqueName: "kubernetes.io/secret/eded16ae-c36f-11e7-9d53-42010a14002f-pvinstaller-token-rtc9s") pod "persistent-volume-setup-5v9r2" (UID: "eded16ae-c36f-11e7-9d53-42010a14002f") : exit status 1
E1107 04:04:50.607384    6117 nestedpendingoperations.go:264] Operation for "\"kubernetes.io/secret/eefdbbda-c36f-11e7-9d53-42010a14002f-deployer-token-fn26t\" (\"eefdbbda-c36f-11e7-9d53-42010a14002f\")" failed. No retries permitted until 2017-11-07 04:06:52.60735181 +0000 UTC (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "deployer-token-fn26t" (UniqueName: "kubernetes.io/secret/eefdbbda-c36f-11e7-9d53-42010a14002f-deployer-token-fn26t") pod "router-1-deploy" (UID: "eefdbbda-c36f-11e7-9d53-42010a14002f") : exit status 1
E1107 04:04:50.721963    6117 nestedpendingoperations.go:264] Operation for "\"kubernetes.io/secret/ef0ec675-c36f-11e7-9d53-42010a14002f-deployer-token-fn26t\" (\"ef0ec675-c36f-11e7-9d53-42010a14002f\")" failed. No retries permitted until 2017-11-07 04:06:52.721930853 +0000 UTC (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "deployer-token-fn26t" (UniqueName: "kubernetes.io/secret/ef0ec675-c36f-11e7-9d53-42010a14002f-deployer-token-fn26t") pod "docker-registry-1-deploy" (UID: "ef0ec675-c36f-11e7-9d53-42010a14002f") : exit status 1
E1107 04:05:09.659461    6117 kubelet.go:1594] Unable to mount volumes for pod "docker-registry-1-deploy_default(ef0ec675-c36f-11e7-9d53-42010a14002f)": timeout expired waiting for volumes to attach/mount for pod "default"/"docker-registry-1-deploy". list of unattached/unmounted volumes=[deployer-token-fn26t]; skipping pod
E1107 04:05:09.659586    6117 pod_workers.go:186] Error syncing pod ef0ec675-c36f-11e7-9d53-42010a14002f ("docker-registry-1-deploy_default(ef0ec675-c36f-11e7-9d53-42010a14002f)"), skipping: timeout expired waiting for volumes to attach/mount for pod "default"/"docker-registry-1-deploy". list of unattached/unmounted volumes=[deployer-token-fn26t]
E1107 04:05:11.659090    6117 kubelet.go:1594] Unable to mount volumes for pod "router-1-deploy_default(eefdbbda-c36f-11e7-9d53-42010a14002f)": timeout expired waiting for volumes to attach/mount for pod "default"/"router-1-deploy". list of unattached/unmounted volumes=[deployer-token-fn26t]; skipping pod
E1107 04:05:11.659127    6117 pod_workers.go:186] Error syncing pod eefdbbda-c36f-11e7-9d53-42010a14002f ("router-1-deploy_default(eefdbbda-c36f-11e7-9d53-42010a14002f)"), skipping: timeout expired waiting for volumes to attach/mount for pod "default"/"router-1-deploy". list of unattached/unmounted volumes=[deployer-token-fn26t]
E1107 04:05:11.662403    6117 kubelet.go:1594] Unable to mount volumes for pod "persistent-volume-setup-5v9r2_default(eded16ae-c36f-11e7-9d53-42010a14002f)": timeout expired waiting for volumes to attach/mount for pod "default"/"persistent-volume-setup-5v9r2". list of unattached/unmounted volumes=[pvinstaller-token-rtc9s]; skipping pod
E1107 04:05:11.662424    6117 pod_workers.go:186] Error syncing pod eded16ae-c36f-11e7-9d53-42010a14002f ("persistent-volume-setup-5v9r2_default(eded16ae-c36f-11e7-9d53-42010a14002f)"), skipping: timeout expired waiting for volumes to attach/mount for pod "default"/"persistent-volume-setup-5v9r2". list of unattached/unmounted volumes=[pvinstaller-token-rtc9s]
I1107 04:05:21.715978    6117 replication_controller.go:588] Replication Controller has been deleted scdf/app-testsimpledeployme-1
I1107 04:05:21.758952    6117 reconciler.go:290] Volume detached for volume "deployer-token-gzpdf" (UniqueName: "kubernetes.io/secret/062c1108-c370-11e7-9d53-42010a14002f-deployer-token-gzpdf") on node "localhost" DevicePath ""
E1107 04:05:28.662939    6117 kubelet_volumes.go:128] Orphaned pod "062c1108-c370-11e7-9d53-42010a14002f" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
E1107 04:05:30.664298    6117 kubelet_volumes.go:128] Orphaned pod "062c1108-c370-11e7-9d53-42010a14002f" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
E1107 04:05:32.666247    6117 kubelet_volumes.go:128] Orphaned pod "062c1108-c370-11e7-9d53-42010a14002f" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.  Turn up verbosity to see them.
E1107 04:05:34.662019    6117 kubelet_volumes.go:128] Orphaned pod "062c1108-c370-11e7-9d53-42010a14002f" found, but volume paths are still present on disk. : There were a total of 1 errors similar to this.   Turn up verbosity to see them.
pweil- commented 7 years ago

since this is for the token /cc @simo5

cf532088799 commented 6 years ago

Hello，I can`t start api-server,the logs tells: Nov 27 22:47:48 localhost kubelet: E1127 22:47:48.679948 2374 kubelet_volumes.go:114] Orphaned pod "4d166379-d370-11e7-bed4-0e85c26ea6ca" found, but volume paths are still present on disk. Nov 27 22:47:48 localhost kubelet: E1127 22:47:48.680037 2374 kubelet_volumes.go:114] Orphaned pod "d8d80b9a-d354-11e7-bed4-0e85c26ea6ca" found, but volume paths are still present on disk. Nov 27 22:47:48 localhost kubelet: E1127 22:47:48.680115 2374 kubelet_volumes.go:114] Orphaned pod "d8d827be-d354-11e7-bed4-0e85c26ea6ca" found, but volume paths are still present on disk. Do you resolve the problem？ can you help？

hrishin commented 6 years ago

New release https://github.com/openshift/origin/releases/tag/v3.7.0 seems to be working well.

donovanmuller commented 6 years ago

This does still not work for me in context of a Travis build: https://travis-ci.org/donovanmuller/spring-cloud-deployer-openshift/jobs/316810682

scdf          2m         6m          10        app-testsimpledeployme-1-deploy   Pod                            Warning   FailedMount                kubelet, localhost            MountVolume.SetUp failed for volume "deployer-token-blkj9" : exit status 1
default       2s         4m          3         router-1-deploy                   Pod                            Warning   FailedMount                kubelet, localhost            Unable to mount volumes for pod "router-1-deploy_default(b97153e2-e170-11e7-a033-42010a1400d9)": timeout expired waiting for volumes to attach/mount for pod "default"/"router-1-deploy". list of unattached/unmounted volumes=[deployer-token-fdjp7]
default       2s         4m          3         docker-registry-1-deploy          Pod                            Warning   FailedMount                kubelet, localhost            Unable to mount volumes for pod "docker-registry-1-deploy_default(b963df7c-e170-11e7-a033-42010a1400d9)": timeout expired waiting for volumes to attach/mount for pod "default"/"docker-registry-1-deploy". list of unattached/unmounted volumes=[deployer-token-fdjp7]
default       2s         4m          3         persistent-volume-setup-pz5zw     Pod                            Warning   FailedMount                kubelet, localhost            Unable to mount volumes for pod "persistent-volume-setup-pz5zw_default(b981f00e-e170-11e7-a033-42010a1400d9)": timeout expired waiting for volumes to attach/mount for pod "default"/"persistent-volume-setup-pz5zw". list of unattached/unmounted volumes=[pvinstaller-token-rz9z4]
scdf          1m         4m          2         app-testsimpledeployme-1-deploy   Pod                            Warning   FailedMount                kubelet, localhost            Unable to mount volumes for pod "app-testsimpledeployme-1-deploy_scdf(c5e624d6-e170-11e7-a033-42010a1400d9)": timeout expired waiting for volumes to attach/mount for pod "scdf"/"app-testsimpledeployme-1-deploy". list of unattached/unmounted volumes=[deployer-token-blkj9]
donovanmuller commented 6 years ago

Using the 3.9.0-alpha.3 release produces a successful build. See https://travis-ci.org/donovanmuller/spring-cloud-deployer-openshift/jobs/337055121

I'll leave this issue open until the 3.9.0 release produces a successful build.

openshift-bot commented 6 years ago

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close. Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

donovanmuller commented 6 years ago

Confirmed this is working in the 3.9.0 release.