search

deis / builder

Git server and application builder for Deis Workflow
https://deis.com
MIT License
40 stars 41 forks source link

Better error reporting when builder pod failure happens #288

Closed helgi closed 8 years ago

helgi commented 8 years ago

While running e2e I ran into a builder problem where it was failing due to out of disk problems.

Getting back the whole Pod object helps me understand the problem but we should try to error out with a bit smaller message and also it seems the test kept going as if the proper error code wasn't returned

remote: 2016/04/05 05:45:00 Error running git receive hook [watching events for builder pod startup (Giving up; pod went into failed status:
remote: &api.Pod{TypeMeta:unversioned.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:api.ObjectMeta{Name:"slugbuild-test-50362358-779ca3d7-3c4b4798", GenerateName:"", Namespace:"deis", SelfLink:"/api/v1/namespaces/deis/pods/slugbuild-test-50362358-779ca3d7-3c4b4798", UID:"896b387a-faf1-11e5-91ae-080027969e8a", ResourceVersion:"32712", Generation:0, CreationTimestamp:unversioned.Time{Time:time.Time{sec:63595431900, nsec:0, loc:(*time.Location)(0x14ec860)}}, DeletionTimestamp:(*unversioned.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{"heritage":"slugbuild-test-50362358-779ca3d7-3c4b4798"}, Annotations:map[string]string(nil)}, Spec:api.PodSpec{Volumes:[]api.Volume{api.Volume{Name:"objectstorage-keyfile", VolumeSource:api.VolumeSource{HostPath:(*api.HostPathVolumeSource)(nil), EmptyDir:(*api.EmptyDirVolumeSource)(nil), GCEPersistentDisk:(*api.GCEPersistentDiskVolumeSource)(nil), AWSElasticBlockStore:(*api.AWSElasticBlockStoreVolumeSource)(nil), GitRepo:(*api.GitRepoVolumeSource)(nil), Secret:(*api.SecretVolumeSource)(0xc82031b4a0), NFS:(*api.NFSVolumeSource)(nil), ISCSI:(*api.ISCSIVolumeSource)(nil), Glusterfs:(*api.GlusterfsVolumeSource)(nil), PersistentVolumeClaim:(*api.PersistentVolumeClaimVolumeSource)(nil), RBD:(*api.RBDVolumeSource)(nil), Cinder:(*api.CinderVolumeSource)(nil), CephFS:(*api.CephFSVolumeSource)(nil), Flocker:(*api.FlockerVolumeSource)(nil), DownwardAPI:(*api.DownwardAPIVolumeSource)(nil), FC:(*api.FCVolumeSource)(nil)}}, api.Volume{Name:"default-token-qwhiw", VolumeSource:api.VolumeSource{HostPath:(*api.HostPathVolumeSource)(nil), EmptyDir:(*api.EmptyDirVolumeSource)(nil), GCEPersistentDisk:(*api.GCEPersistentDiskVolumeSource)(nil), AWSElasticBlockStore:(*api.AWSElasticBlockStoreVolumeSource)(nil), GitRepo:(*api.GitRepoVolumeSource)(nil), Secret:(*api.SecretVolumeSource)(0xc82031b4c0), NFS:(*api.NFSVolumeSource)(nil), ISCSI:(*api.ISCSIVolumeSource)(nil), Glusterfs:(*api.GlusterfsVolumeSource)(nil), PersistentVolumeClaim:(*api.PersistentVolumeClaimVolumeSource)(nil), RBD:(*api.RBDVolumeSource)(nil), Cinder:(*api.CinderVolumeSource)(nil), CephFS:(*api.CephFSVolumeSource)(nil), Flocker:(*api.FlockerVolumeSource)(nil), DownwardAPI:(*api.DownwardAPIVolumeSource)(nil), FC:(*api.FCVolumeSource)(nil)}}}, Containers:[]api.Container{api.Container{Name:"deis-slugbuilder", Image:"quay.io/deisci/slugbuilder:canary", Command:[]string(nil), Args:[]string(nil), WorkingDir:"", Ports:[]api.ContainerPort(nil), Env:[]api.EnvVar{api.EnvVar{Name:"DEBUG", Value:"1", ValueFrom:(*api.EnvVarSource)(nil)}, api.EnvVar{Name:"TAR_PATH", Value:"home/test-50362358:git-779ca3d7/tar", ValueFrom:(*api.EnvVarSource)(nil)}, api.EnvVar{Name:"PUT_PATH", Value:"home/test-50362358:git-779ca3d7/push", ValueFrom:(*api.EnvVarSource)(nil)}, api.EnvVar{Name:"BUILDER_STORAGE", Value:"minio", ValueFrom:(*api.EnvVarSource)(nil)}}, Resources:api.ResourceRequirements{Limits:api.ResourceList(nil), Requests:api.ResourceList(nil)}, VolumeMounts:[]api.VolumeMount{api.VolumeMount{Name:"objectstorage-keyfile", ReadOnly:true, MountPath:"/var/run/secrets/deis/objectstore/creds"}, api.VolumeMount{Name:"default-token-qwhiw", ReadOnly:true, MountPath:"/var/run/secrets/kubernetes.io/serviceaccount"}}, LivenessProbe:(*api.Probe)(nil), ReadinessProbe:(*api.Probe)(nil), Lifecycle:(*api.Lifecycle)(nil), TerminationMessagePath:"/dev/termination-log", ImagePullPolicy:"Always", SecurityContext:(*api.SecurityContext)(nil), Stdin:false, StdinOnce:false, TTY:false}}, RestartPolicy:"Never", TerminationGracePeriodSeconds:(*int64)(0xc82031b4f0), ActiveDeadlineSeconds:(*int64)(nil), DNSPolicy:"ClusterFirst", NodeSelector:map[string]string(nil), ServiceAccountName:"default", NodeName:"172.17.8.100", HostNetwork:false, HostPID:false, HostIPC:false, ImagePullSecrets:[]api.LocalObjectReference(nil)}, Status:api.PodStatus{Phase:"Failed", Conditions:[]api.PodCondition(nil), Message:"Pod cannot be started due to lack of disk space.", Reason:"OutOfDisk", HostIP:"", PodIP:"", StartTime:(*unversioned.Time)(0xc8203f0200), ContainerStatuses:[]api.ContainerStatus(nil)}})]

To ssh://git@deis.10.3.0.208.nip.io:2222/test-50362358.git
 * [new branch]      master -> master
[It] restarts all of 1
  /go/src/github.com/deis/workflow-e2e/vendor/github.com/onsi/ginkgo/extensions/table/table_entry.go:46
$ deis ps:scale web=1 --app=test-50362358
Scaling processes... but first, coffee!
Error:
400 Bad Request
detail: No build associated with this release