search

kube-object-storage / lib-bucket-provisioner

Library for the dynamic provisioning of object store buckets to be used by object store providers.
Apache License 2.0
20 stars 22 forks source link

New OBC create secret already exists fail and fail to cleanup #146

Open guymguym opened 5 years ago

guymguym commented 5 years ago

Created a new OBC, the secret did not exist, but the provisioner failed because it got an "AlreadyExists" error on the secret. Maybe due to create retry? Then it tries to clean everything up and fails - Perhaps the name is not initialized for those resources when the creation is partially done?

I0904 09:30:06.270492       1 resourcehandlers.go:139]  "level"=0 "msg"="creating Secret" "key"="noobaa-2/kaka" "name"="noobaa-2/kaka"
E0904 09:30:06.369702       1 controller.go:243]  "msg"="cleaning up reconcile artifacts" "error"="error creating secret for OBC \"noobaa-2/kaka\": secrets \"kaka\" already exists" "key"="noobaa-2/kaka"
I0904 09:30:06.370114       1 controller.go:245]  "level"=0 "msg"="deleting storage artifacts" "key"="noobaa-2/kaka"
time="2019-09-04T09:30:06Z" level=info msg="Delete: got request to delete bucket \"kaka-36f0517b-4016-464c-a2c0-5
07a63041f8f\" and account \"obc-account.kaka-36f0517b-4016-464c-a2c0-507a63041f8f.5d6f841d@noobaa.io\"" provision
er=noobaa.io/noobaa-2.bucket
time="2019-09-04T09:30:06Z" level=info msg="deleting account \"obc-account.kaka-36f0517b-4016-464c-a2c0-507a63041
f8f.5d6f841d@noobaa.io\"" provisioner=noobaa.io/noobaa-2.bucket
time="2019-09-04T09:30:06Z" level=info msg="✈️  RPC: account.delete_account() Request: {Email:obc-account.kaka-36f
0517b-4016-464c-a2c0-507a63041f8f.5d6f841d@noobaa.io}"
time="2019-09-04T09:30:06Z" level=info msg="UpdateStatus: Done generation 1" sys=noobaa-2/noobaa
time="2019-09-04T09:30:06Z" level=info msg="✅ RPC: account.delete_account() Response OK: &{Op:res RequestID:1@https://::ffff:172.17.0.1:44186(cxevzhbc) Took:30.417999999946915 Error:<nil>}"
time="2019-09-04T09:30:06Z" level=info msg="✅ Successfully deleted account \"obc-account.kaka-36f0517b-4016-464c-a2c0-507a63041f8f.5d6f841d@noobaa.io\"" provisioner=noobaa.io/noobaa-2.bucket
I0904 09:30:06.421912       1 resourcehandlers.go:230]  "level"=0 "msg"="removing ObjectBucket's finalizer" "key"="noobaa-2/kaka" "name"=""
E0904 09:30:06.422056       1 controller.go:423]  "msg"="error deleting objectBucket" "error"="resource name may not be empty" "key"="noobaa-2/kaka" "/"=null
E0904 09:30:06.422168       1 controller.go:429]  "msg"="error releasing secret" "error"="resource name may not be empty" "key"="noobaa-2/kaka" "/"=null
jeffvance commented 5 years ago

@guymguym I've tried several times to re-create this in minikube and have not been able to.

More info:

handleProvision()'s defer handles cleanup for all errors. This is where deleteResources() is called. Prior to the secret create err, Provision() was called successfully and it returned a partially filled-in ob (so the ob is not nil). We try to create a secret and that fails (for some reason). The CM (nil) and OB (not nil) are skipped and the defer block is invoked. Since the ob is not nil we try to handle it in deleteObjectBucket() and get the Update error in the log.

I'd like to add to the delete<ResourceType> funcs where they check if the resource is nil, to also check that the name is empty, and only then return. Eg.

        if ob == nil || ob.Name == "" {
                return nil
        }

I still don't know why the secret creation failed but this issue highlights a lib error recovery bug.

jeffvance commented 4 years ago

@guymguym with pr #153 merged can we close this issue?