search

vmware-archive / dispatch

Dispatch is a framework for deploying and managing serverless style applications.
http://dispatchframework.io
Apache License 2.0
532 stars 58 forks source link

Event Driver stuck on DELETING #252

Closed tenczar closed 6 years ago

tenczar commented 6 years ago

Detailed Description

It seems that the control loop gets stuck in the DELETING state under specific circumstances. I created a vcenter event driver called vxrail. I deleted this event driver and confirmed that it was no longer present when calling dispatch get event-driver. I then immediately made a call to create a new event driver with the same name. This resulted in the newly created event-driver being in an ERROR state. Attempting to delete this event-driver then results in the resource being stuck in the deleting state.

I assume the pod containing the old event driver was not done terminating when the new create call was made. That resulted in the create call erroring out without creating a pod. The delete call is now failing because it can't find the pod to delete. I suspect we might have similar bugs in the other control loop handlers.

Log output

time="2018-03-02T21:55:00Z" level=debug msg="[BEGIN] [github.com/vmware/dispatch/pkg/event-manager.(*driverEntityHandler).Delete:57] name 'vxrail'"
time="2018-03-02T21:55:00Z" level=error msg="k8s: deployment=event-driver-vcenter-vxrail not found: deployments.extensions \"event-driver-vcenter-vxrail\" not found"
time="2018-03-02T21:55:00Z" level=debug msg="[END  ] [github.com/vmware/dispatch/pkg/event-manager.(*driverEntityHandler).Delete:57] [3.507112ms] name 'vxrail'"
time="2018-03-02T21:55:00Z" level=debug msg="[END  ] [github.com/vmware/dispatch/pkg/controller.(*DefaultController).processItem:123] [3.580151ms] "
time="2018-03-02T21:55:00Z" level=error msg="error deleting driver: k8s: deployment=event-driver-vcenter-vxrail not found: deployments.extensions \"event-driver-vcenter-vxrail\" not found"
time="2018-03-02T21:56:00Z" level=debug msg="[BEGIN] [github.com/vmware/dispatch/pkg/controller.(*DefaultController).run.func2.1:252]"
time="2018-03-02T21:56:00Z" level=info msg="periodic syncing with the underlying driver"
time="2018-03-02T21:56:00Z" level=debug msg="[BEGIN] [github.com/vmware/dispatch/pkg/controller.(*DefaultController).sync:193]"
time="2018-03-02T21:56:00Z" level=debug msg="[BEGIN] [github.com/vmware/dispatch/pkg/event-manager.(*subscriptionEntityHandler).Sync:73]"
time="2018-03-02T21:56:00Z" level=debug msg="[BEGIN] [github.com/vmware/dispatch/pkg/event-manager.(*subscriptionEntityHandler).Type:26]"
time="2018-03-02T21:56:00Z" level=debug msg="[END  ] [github.com/vmware/dispatch/pkg/event-manager.(*subscriptionEntityHandler).Type:26] [180.784µs] "
time="2018-03-02T21:56:00Z" level=debug msg="[BEGIN] [github.com/vmware/dispatch/pkg/controller.defaultSyncFilter:151]"
time="2018-03-02T21:56:00Z" level=debug msg="[END  ] [github.com/vmware/dispatch/pkg/controller.defaultSyncFilter:151] [66.273µs] "
time="2018-03-02T21:56:00Z" level=debug msg="[END  ] [github.com/vmware/dispatch/pkg/event-manager.(*subscriptionEntityHandler).Sync:73] [1.872434ms] "
time="2018-03-02T21:56:00Z" level=debug msg="[BEGIN] [github.com/vmware/dispatch/pkg/event-manager.(*driverEntityHandler).Sync:75]"
time="2018-03-02T21:56:00Z" level=debug msg="[BEGIN] [github.com/vmware/dispatch/pkg/event-manager.(*driverEntityHandler).Type:26]"
time="2018-03-02T21:56:00Z" level=debug msg="[END  ] [github.com/vmware/dispatch/pkg/event-manager.(*driverEntityHandler).Type:26] [10.107µs] "
time="2018-03-02T21:56:00Z" level=debug msg="[BEGIN] [github.com/vmware/dispatch/pkg/controller.defaultSyncFilter:151]"
time="2018-03-02T21:56:00Z" level=debug msg="[END  ] [github.com/vmware/dispatch/pkg/controller.defaultSyncFilter:151] [13.822µs] "
time="2018-03-02T21:56:00Z" level=debug msg="[END  ] [github.com/vmware/dispatch/pkg/event-manager.(*driverEntityHandler).Sync:75] [1.27889ms] "
time="2018-03-02T21:56:00Z" level=debug msg="[END  ] [github.com/vmware/dispatch/pkg/controller.(*DefaultController).sync:193] [3.4311ms] "
time="2018-03-02T21:56:00Z" level=debug msg="[END  ] [github.com/vmware/dispatch/pkg/controller.(*DefaultController).run.func2.1:252] [3.565905ms] "
time="2018-03-02T21:56:00Z" level=info msg="sync: processing entity vxrail"
time="2018-03-02T21:56:00Z" level=debug msg="[BEGIN] [github.com/vmware/dispatch/pkg/controller.(*DefaultController).processItem:123]"
time="2018-03-02T21:56:00Z" level=debug msg="[BEGIN] [github.com/vmware/dispatch/pkg/event-manager.(*driverEntityHandler).Delete:57] name 'vxrail'"
time="2018-03-02T21:56:00Z" level=error msg="k8s: deployment=event-driver-vcenter-vxrail not found: deployments.extensions \"event-driver-vcenter-vxrail\" not found"
time="2018-03-02T21:56:00Z" level=debug msg="[END  ] [github.com/vmware/dispatch/pkg/event-manager.(*driverEntityHandler).Delete:57] [3.816118ms] name 'vxrail'"
time="2018-03-02T21:56:00Z" level=debug msg="[END  ] [github.com/vmware/dispatch/pkg/controller.(*DefaultController).processItem:123] [3.869478ms] "
time="2018-03-02T21:56:00Z" level=error msg="error deleting driver: k8s: deployment=event-driver-vcenter-vxrail not found: deployments.extensions \"event-driver-vcenter-vxrail\" not found"
time="2018-03-02T21:56:24Z" level=info msg="cookie auth: unset\n"
time="2018-03-02T21:56:24Z" level=debug msg="[BEGIN] [github.com/vmware/dispatch/pkg/event-manager.(*Handlers).getDrivers:442] getDrivers"
time="2018-03-02T21:56:24Z" level=debug msg="[BEGIN] [github.com/vmware/dispatch/pkg/event-manager.driverEntityToModel:138] type: vxrail, name: vcenter"
time="2018-03-02T21:56:24Z" level=debug msg="[END  ] [github.com/vmware/dispatch/pkg/event-manager.(*Handlers).getDrivers:442] [1.330043ms] getDrivers"

Context

Possible Implementation

Complexity

Impact

Your Environment

kars7e commented 6 years ago

This should not happen anymore with the work from #439