search

containerbuilding / cbi

Container Builder Interface for Kubernetes with support for several backends (Docker, BuildKit, Buildah, kaniko, img, Google Cloud Container Builder, Azure Container Registry Build, OpenShift Source-to-Image...)
Apache License 2.0
243 stars 18 forks source link

Segfault in apimachinery #69

Closed gopenguin closed 6 years ago

gopenguin commented 6 years ago

First of all, thanks for this great projekt. After some testing it seems to work realy great. But while iterating over different configurations the cbid keeps segfaulting. Building a image during the crash cycles still worked. Here are two of the logs:

W0527 17:12:41.949819       1 client_config.go:533] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0527 17:12:41.955578       1 controller.go:109] Creating event broadcaster
I0527 17:12:41.958554       1 controller.go:127] Setting up event handlers
I0527 17:12:41.958735       1 controller.go:168] Starting BuildJob controller
I0527 17:12:41.958766       1 controller.go:171] Waiting for informer caches to sync
I0527 17:12:41.959552       1 reflector.go:202] Starting reflector *v1.Job (30s) from github.com/containerbuilding/cbi/vendor/k8s.io/client-go/informers/factory.go:87
I0527 17:12:41.959620       1 reflector.go:240] Listing and watching *v1.Job from github.com/containerbuilding/cbi/vendor/k8s.io/client-go/informers/factory.go:87
I0527 17:12:41.960526       1 reflector.go:202] Starting reflector *v1alpha1.BuildJob (30s) from github.com/containerbuilding/cbi/pkg/client/informers/externalversions/factory.go:117
I0527 17:12:41.960599       1 reflector.go:240] Listing and watching *v1alpha1.BuildJob from github.com/containerbuilding/cbi/pkg/client/informers/externalversions/factory.go:117
I0527 17:12:42.128097       1 controller.go:386] Processing object: cluster-init-secure
I0527 17:12:42.128161       1 controller.go:386] Processing object: emp-hufflepuff-m
I0527 17:12:42.128189       1 controller.go:386] Processing object: minimal-ldap-proxy-test-job
I0527 17:12:42.158966       1 shared_informer.go:123] caches populated
I0527 17:12:42.159034       1 controller.go:176] Starting workers
I0527 17:12:42.159093       1 controller.go:182] Started workers
E0527 17:12:42.162489       1 runtime.go:66] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
/go/src/github.com/containerbuilding/cbi/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:72
/go/src/github.com/containerbuilding/cbi/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65
/go/src/github.com/containerbuilding/cbi/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51
/usr/local/go/src/runtime/asm_amd64.s:573
/usr/local/go/src/runtime/panic.go:502
/usr/local/go/src/runtime/panic.go:63
/usr/local/go/src/runtime/signal_unix.go:388
/go/src/github.com/containerbuilding/cbi/pkg/cbid/controller/job.go:60
/go/src/github.com/containerbuilding/cbi/pkg/cbid/controller/controller.go:287
/go/src/github.com/containerbuilding/cbi/pkg/cbid/controller/controller.go:232
/go/src/github.com/containerbuilding/cbi/pkg/cbid/controller/controller.go:240
/go/src/github.com/containerbuilding/cbi/pkg/cbid/controller/controller.go:193
/go/src/github.com/containerbuilding/cbi/pkg/cbid/controller/controller.go:179
/go/src/github.com/containerbuilding/cbi/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133
/go/src/github.com/containerbuilding/cbi/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134
/go/src/github.com/containerbuilding/cbi/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88
/usr/local/go/src/runtime/asm_amd64.s:2361
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
    panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0xf4c93b]

goroutine 149 [running]:
github.com/containerbuilding/cbi/vendor/k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
    /go/src/github.com/containerbuilding/cbi/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:58 +0x107
panic(0x1073d60, 0x1a04550)
    /usr/local/go/src/runtime/panic.go:502 +0x229
github.com/containerbuilding/cbi/pkg/cbid/controller.newJob(0x12ec680, 0xc4200be020, 0x12d4620, 0xc4200c0030, 0xc420579200, 0xc4203bec60, 0x17, 0x0)
    /go/src/github.com/containerbuilding/cbi/pkg/cbid/controller/job.go:60 +0x14b
github.com/containerbuilding/cbi/pkg/cbid/controller.(*Controller).syncHandler(0xc4203c2380, 0xc4203bed60, 0x1d, 0x1010c80, 0xc4202afd70)
    /go/src/github.com/containerbuilding/cbi/pkg/cbid/controller/controller.go:287 +0x186
github.com/containerbuilding/cbi/pkg/cbid/controller.(*Controller).processNextWorkItem.func1(0xc4203c2380, 0x1010c80, 0xc4202afd70, 0x0, 0x0)
    /go/src/github.com/containerbuilding/cbi/pkg/cbid/controller/controller.go:232 +0xda
github.com/containerbuilding/cbi/pkg/cbid/controller.(*Controller).processNextWorkItem(0xc4203c2380, 0xc420320210)
    /go/src/github.com/containerbuilding/cbi/pkg/cbid/controller/controller.go:240 +0x4f
github.com/containerbuilding/cbi/pkg/cbid/controller.(*Controller).runWorker(0xc4203c2380)
    /go/src/github.com/containerbuilding/cbi/pkg/cbid/controller/controller.go:193 +0x2b
github.com/containerbuilding/cbi/pkg/cbid/controller.(*Controller).(github.com/containerbuilding/cbi/pkg/cbid/controller.runWorker)-fm()
    /go/src/github.com/containerbuilding/cbi/pkg/cbid/controller/controller.go:179 +0x2a
github.com/containerbuilding/cbi/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc4202ae160)
    /go/src/github.com/containerbuilding/cbi/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x54
github.com/containerbuilding/cbi/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc4202ae160, 0x3b9aca00, 0x0, 0x1, 0xc4203c0ea0)
    /go/src/github.com/containerbuilding/cbi/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134 +0xbd
github.com/containerbuilding/cbi/vendor/k8s.io/apimachinery/pkg/util/wait.Until(0xc4202ae160, 0x3b9aca00, 0xc4203c0ea0)
    /go/src/github.com/containerbuilding/cbi/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88 +0x4d
created by github.com/containerbuilding/cbi/pkg/cbid/controller.(*Controller).Run
    /go/src/github.com/containerbuilding/cbi/pkg/cbid/controller/controller.go:179 +0x1fb

W0527 17:15:31.311108       1 client_config.go:533] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0527 17:15:31.315551       1 controller.go:109] Creating event broadcaster
I0527 17:15:31.316720       1 controller.go:127] Setting up event handlers
I0527 17:15:31.317110       1 controller.go:168] Starting BuildJob controller
I0527 17:15:31.317206       1 controller.go:171] Waiting for informer caches to sync
I0527 17:15:31.317992       1 reflector.go:202] Starting reflector *v1.Job (30s) from github.com/containerbuilding/cbi/vendor/k8s.io/client-go/informers/factory.go:87
I0527 17:15:31.318203       1 reflector.go:240] Listing and watching *v1.Job from github.com/containerbuilding/cbi/vendor/k8s.io/client-go/informers/factory.go:87
I0527 17:15:31.319428       1 reflector.go:202] Starting reflector *v1alpha1.BuildJob (30s) from github.com/containerbuilding/cbi/pkg/client/informers/externalversions/factory.go:117
I0527 17:15:31.319570       1 reflector.go:240] Listing and watching *v1alpha1.BuildJob from github.com/containerbuilding/cbi/pkg/client/informers/externalversions/factory.go:117
I0527 17:15:31.458542       1 controller.go:386] Processing object: cluster-init-secure
I0527 17:15:31.458613       1 controller.go:386] Processing object: emp-hufflepuff-m
I0527 17:15:31.458679       1 controller.go:386] Processing object: minimal-ldap-proxy-test-job
I0527 17:15:31.517913       1 shared_informer.go:123] caches populated
I0527 17:15:31.517982       1 controller.go:176] Starting workers
I0527 17:15:31.518026       1 controller.go:182] Started workers
I0527 17:15:31.590351       1 controller.go:238] Successfully synced 'build/minimal-ldap-proxy-test'
I0527 17:15:31.590766       1 event.go:218] Event(v1.ObjectReference{Kind:"BuildJob", Namespace:"build", Name:"minimal-ldap-proxy-test", UID:"306554a4-61d0-11e8-a63f-d05099532f33", APIVersion:"cbi.containerbuilding.github.io/v1alpha1", ResourceVersion:"15380348", FieldPath:""}): type: 'Normal' reason: 'Synced' BuildJob synced successfully
I0527 17:16:01.384681       1 reflector.go:286] github.com/containerbuilding/cbi/pkg/client/informers/externalversions/factory.go:117: forcing resync
I0527 17:16:01.418535       1 controller.go:238] Successfully synced 'build/minimal-ldap-proxy-test'
I0527 17:16:01.418663       1 event.go:218] Event(v1.ObjectReference{Kind:"BuildJob", Namespace:"build", Name:"minimal-ldap-proxy-test", UID:"306554a4-61d0-11e8-a63f-d05099532f33", APIVersion:"cbi.containerbuilding.github.io/v1alpha1", ResourceVersion:"15380348", FieldPath:""}): type: 'Normal' reason: 'Synced' BuildJob synced successfully
I0527 17:16:01.459749       1 reflector.go:286] github.com/containerbuilding/cbi/vendor/k8s.io/client-go/informers/factory.go:87: forcing resync
I0527 17:16:31.385613       1 reflector.go:286] github.com/containerbuilding/cbi/pkg/client/informers/externalversions/factory.go:117: forcing resync
I0527 17:16:31.460144       1 reflector.go:286] github.com/containerbuilding/cbi/vendor/k8s.io/client-go/informers/factory.go:87: forcing resync
I0527 17:16:31.469399       1 controller.go:238] Successfully synced 'build/minimal-ldap-proxy-test'
I0527 17:16:31.469503       1 event.go:218] Event(v1.ObjectReference{Kind:"BuildJob", Namespace:"build", Name:"minimal-ldap-proxy-test", UID:"306554a4-61d0-11e8-a63f-d05099532f33", APIVersion:"cbi.containerbuilding.github.io/v1alpha1", ResourceVersion:"15380348", FieldPath:""}): type: 'Normal' reason: 'Synced' BuildJob synced successfully
I0527 17:17:01.386571       1 reflector.go:286] github.com/containerbuilding/cbi/pkg/client/informers/externalversions/factory.go:117: forcing resync
I0527 17:17:01.416023       1 controller.go:238] Successfully synced 'build/minimal-ldap-proxy-test'
I0527 17:17:01.416174       1 event.go:218] Event(v1.ObjectReference{Kind:"BuildJob", Namespace:"build", Name:"minimal-ldap-proxy-test", UID:"306554a4-61d0-11e8-a63f-d05099532f33", APIVersion:"cbi.containerbuilding.github.io/v1alpha1", ResourceVersion:"15380348", FieldPath:""}): type: 'Normal' reason: 'Synced' BuildJob synced successfully
I0527 17:17:01.460527       1 reflector.go:286] github.com/containerbuilding/cbi/vendor/k8s.io/client-go/informers/factory.go:87: forcing resync
I0527 17:17:31.387140       1 reflector.go:286] github.com/containerbuilding/cbi/pkg/client/informers/externalversions/factory.go:117: forcing resync
E0527 17:17:31.395272       1 runtime.go:66] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
/go/src/github.com/containerbuilding/cbi/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:72
/go/src/github.com/containerbuilding/cbi/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65
/go/src/github.com/containerbuilding/cbi/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51
/usr/local/go/src/runtime/asm_amd64.s:573
/usr/local/go/src/runtime/panic.go:502
/usr/local/go/src/runtime/panic.go:63
/usr/local/go/src/runtime/signal_unix.go:388
/go/src/github.com/containerbuilding/cbi/pkg/cbid/controller/job.go:60
/go/src/github.com/containerbuilding/cbi/pkg/cbid/controller/controller.go:287
/go/src/github.com/containerbuilding/cbi/pkg/cbid/controller/controller.go:232
/go/src/github.com/containerbuilding/cbi/pkg/cbid/controller/controller.go:240
/go/src/github.com/containerbuilding/cbi/pkg/cbid/controller/controller.go:193
/go/src/github.com/containerbuilding/cbi/pkg/cbid/controller/controller.go:179
/go/src/github.com/containerbuilding/cbi/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133
/go/src/github.com/containerbuilding/cbi/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134
/go/src/github.com/containerbuilding/cbi/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88
/usr/local/go/src/runtime/asm_amd64.s:2361
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
    panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0xf4c93b]

goroutine 123 [running]:
github.com/containerbuilding/cbi/vendor/k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
    /go/src/github.com/containerbuilding/cbi/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:58 +0x107
panic(0x1073d60, 0x1a04550)
    /usr/local/go/src/runtime/panic.go:502 +0x229
github.com/containerbuilding/cbi/pkg/cbid/controller.newJob(0x12ec680, 0xc4200be020, 0x12d4620, 0xc4200fa070, 0xc4205e4d00, 0xc42038ed40, 0x17, 0x0)
    /go/src/github.com/containerbuilding/cbi/pkg/cbid/controller/job.go:60 +0x14b
github.com/containerbuilding/cbi/pkg/cbid/controller.(*Controller).syncHandler(0xc420384b80, 0xc42038e100, 0x1d, 0x1010c80, 0xc4202ba0b0)
    /go/src/github.com/containerbuilding/cbi/pkg/cbid/controller/controller.go:287 +0x186
github.com/containerbuilding/cbi/pkg/cbid/controller.(*Controller).processNextWorkItem.func1(0xc420384b80, 0x1010c80, 0xc4202ba0b0, 0x0, 0x0)
    /go/src/github.com/containerbuilding/cbi/pkg/cbid/controller/controller.go:232 +0xda
github.com/containerbuilding/cbi/pkg/cbid/controller.(*Controller).processNextWorkItem(0xc420384b80, 0xc420302401)
    /go/src/github.com/containerbuilding/cbi/pkg/cbid/controller/controller.go:240 +0x4f
github.com/containerbuilding/cbi/pkg/cbid/controller.(*Controller).runWorker(0xc420384b80)
    /go/src/github.com/containerbuilding/cbi/pkg/cbid/controller/controller.go:193 +0x2b
github.com/containerbuilding/cbi/pkg/cbid/controller.(*Controller).(github.com/containerbuilding/cbi/pkg/cbid/controller.runWorker)-fm()
    /go/src/github.com/containerbuilding/cbi/pkg/cbid/controller/controller.go:179 +0x2a
github.com/containerbuilding/cbi/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc4202b2230)
    /go/src/github.com/containerbuilding/cbi/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x54
github.com/containerbuilding/cbi/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc4202b2230, 0x3b9aca00, 0x0, 0x1, 0xc4202e6de0)
    /go/src/github.com/containerbuilding/cbi/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134 +0xbd
github.com/containerbuilding/cbi/vendor/k8s.io/apimachinery/pkg/util/wait.Until(0xc4202b2230, 0x3b9aca00, 0xc4202e6de0)
    /go/src/github.com/containerbuilding/cbi/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88 +0x4d
created by github.com/containerbuilding/cbi/pkg/cbid/controller.(*Controller).Run
    /go/src/github.com/containerbuilding/cbi/pkg/cbid/controller/controller.go:179 +0x1fb

If you have any questions, I'm very happy to answer.

AkihiroSuda commented 6 years ago

Thanks for reporting!

This seems because err is not checked here: https://github.com/containerbuilding/cbi/blob/9787b4f1a4fafed89aed158863497e2bd86a5b6b/pkg/cbid/controller/job.go#L58

I'll fix ASAP

gopenguin commented 6 years ago

Thanks. That was quick