search

kube-object-storage / lib-bucket-provisioner

Library for the dynamic provisioning of object store buckets to be used by object store providers.
Apache License 2.0
20 stars 22 forks source link

attempting to create obc to and external rgw endpoint failing #188

Closed pcuzner closed 4 years ago

pcuzner commented 4 years ago

I have an external ceph cluster (rhcs4), and have created backingstoreclass/storageclass for noobaa - but when I attempt to create an obc I see this in the operator log

time="2020-02-17T02:52:29Z" level=error msg="⚠️  RPC: bucket.read_bucket() Response Error: Code=NO_SUCH_BUCKET Message=No such bucket: mybucket-6bcf1e98-1208-4649-aeef-a7724e3cad01"          
time="2020-02-17T02:52:29Z" level=info msg="✈️  RPC: tier.create_tier() Request: {Name:mybucket-6bcf1e98-1208-4649-aeef-a7724e3cad01.5e49ffed.0 DataPlacement:SPREAD AttachedPools:[rgw-backing
-store] ChunkCoderConfig:<nil>}"                                                                                                                                                              
time="2020-02-17T02:52:29Z" level=info msg="✅ RPC: tier.create_tier() Response OK: took 14.3ms"                                                                                              
time="2020-02-17T02:52:29Z" level=info msg="✈️  RPC: tiering_policy.create_policy() Request: {Name:mybucket-6bcf1e98-1208-4649-aeef-a7724e3cad01.5e49ffed Tiers:[{Order:0 Tier:mybucket-6bcf1e9
8-1208-4649-aeef-a7724e3cad01.5e49ffed.0 Mode:}] ChunkSplitConfig:<nil> DataCapacity:<nil> StorageCapacity:<nil>}"                                                                            
time="2020-02-17T02:52:29Z" level=info msg="✅ RPC: tiering_policy.create_policy() Response OK: took 19.2ms"                                                                                  
time="2020-02-17T02:52:29Z" level=info msg="✈️  RPC: bucket.create_bucket() Request: {Name:mybucket-6bcf1e98-1208-4649-aeef-a7724e3cad01 Tiering:mybucket-6bcf1e98-1208-4649-aeef-a7724e3cad01.
5e49ffed BucketClaim:0xc000742b00}"                                                                                                                                                           
time="2020-02-17T02:52:29Z" level=info msg="✅ RPC: bucket.create_bucket() Response OK: took 22.0ms"                                                                                          
time="2020-02-17T02:52:29Z" level=info msg="✅ Successfully created bucket \"mybucket-6bcf1e98-1208-4649-aeef-a7724e3cad01\"" provisioner=default.noobaa.io/obc                               
time="2020-02-17T02:52:29Z" level=info msg="✈️  RPC: account.create_account() Request: {Name:obc-account.mybucket-6bcf1e98-1208-4649-aeef-a7724e3cad01.5e49ffed@noobaa.io Email:obc-account.myb
ucket-6bcf1e98-1208-4649-aeef-a7724e3cad01.5e49ffed@noobaa.io HasLogin:false S3Access:true AllowBucketCreate:false AllowedBuckets:{FullPermission:false PermissionList:[mybucket-6bcf1e98-1208
-4649-aeef-a7724e3cad01]} DefaultPool:rgw-backing-store}"                                                                                                                                     
time="2020-02-17T02:52:29Z" level=info msg="✅ RPC: account.create_account() Response OK: took 16.4ms"                                                                                        
time="2020-02-17T02:52:29Z" level=info msg="✅ Successfully created account \"obc-account.mybucket-6bcf1e98-1208-4649-aeef-a7724e3cad01.5e49ffed@noobaa.io\" with access to bucket \"mybucket-
6bcf1e98-1208-4649-aeef-a7724e3cad01\"" provisioner=default.noobaa.io/obc                                                                                                                     
I0217 02:52:29.633498       1 resourcehandlers.go:138]  "msg"="creating Secret" "key"="default/mybucket" "name"="default/mybucket"                                                            
E0217 02:52:29.641754       1 controller.go:286]  "msg"="cleaning up reconcile artifacts" "error"="error creating secret for OBC: secrets \"mybucket\" already exists" "key"="default/mybucket
"                                                                                                                                                                                             
I0217 02:52:29.641779       1 controller.go:288]  "msg"="deleting storage artifacts" "key"="default/mybucket"                                                                                 
time="2020-02-17T02:52:29Z" level=info msg="✅ Exists: NooBaa \"noobaa\"\n"                                                                                                                   
time="2020-02-17T02:52:29Z" level=info msg="✅ Exists: Service \"noobaa-mgmt\"\n"                                                                                                             
time="2020-02-17T02:52:29Z" level=info msg="✅ Exists: Secret \"noobaa-operator\"\n"                                                                                                          
time="2020-02-17T02:52:29Z" level=info msg="Delete: got request to delete bucket \"mybucket-6bcf1e98-1208-4649-aeef-a7724e3cad01\" and account \"obc-account.mybucket-6bcf1e98-1208-4649-aeef-
a7724e3cad01.5e49ffed@noobaa.io\"" provisioner=default.noobaa.io/obc                                                                                                                          
time="2020-02-17T02:52:29Z" level=info msg="deleting account \"obc-account.mybucket-6bcf1e98-1208-4649-aeef-a7724e3cad01.5e49ffed@noobaa.io\"" provisioner=default.noobaa.io/obc              
time="2020-02-17T02:52:29Z" level=info msg="✈️  RPC: account.delete_account() Request: {Email:obc-account.mybucket-6bcf1e98-1208-4649-aeef-a7724e3cad01.5e49ffed@noobaa.io}"                   
time="2020-02-17T02:52:29Z" level=info msg="✅ RPC: account.delete_account() Response OK: took 19.9ms"                                                                                        
time="2020-02-17T02:52:29Z" level=info msg="✅ Successfully deleted account \"obc-account.mybucket-6bcf1e98-1208-4649-aeef-a7724e3cad01.5e49ffed@noobaa.io\"" provisioner=default.noobaa.io/ob
c                                                                                                                                                                                             
E0217 02:52:29.682260       1 controller.go:509]  "msg"="error releasing secret" "error"="resource name may not be empty" "key"="default/mybucket"                                            
I0217 02:52:29.682276       1 resourcehandlers.go:182]  "msg"="got nil configmap, skipping" "key"="default/mybucket"

so the resource is created, but then immediately deleted. BTW, before I run the oc create, I make sure the mybucket secret doesn't exist!

I'm also confused when I look in the noobaa GUI, since 3 buckets appear with my bucket prefix - but given the above I would expect them to have been cleaned up?

Quite possibly a "user error" - I'm just starting to get a handle on noobaa goodness!

noobaa-operator version : 2.0.10

jeniawhite commented 4 years ago

Hello, In noobaa-operator version 2.0.7 or above you can use the diagnose command in the CLI (it should collect all of the relevant logs). In this case, I assume that the problem is within the lib-bucket-provisioner or the noobaa-operator itself. I reckon that a complete noobaa-operator pod log will be sufficient (please attach if possible).

From the log snippet that you've attached, I can conclude the following.

1) I assume that you've deleted the secret (in default namespace) and validated the completion of the operation prior to running the creation. I will validate and check lib-bucket-provisioner's querying and creation of secrets.

I0217 02:52:29.633498       1 resourcehandlers.go:138]  "msg"="creating Secret" "key"="default/mybucket" "name"="default/mybucket"                                                            
E0217 02:52:29.641754       1 controller.go:286]  "msg"="cleaning up reconcile artifacts" "error"="error creating secret for OBC: secrets \"mybucket\" already exists" "key"="default/mybucket
"                                                                                                                                                                                             
I0217 02:52:29.641779       1 controller.go:288]  "msg"="deleting storage artifacts" "key"="default/mybucket"

2) The buckets should have been removed as part of the deletion of storage artifacts, but probably failed for this reason or a different reason later on (don't have full log):

E0217 02:52:29.682260       1 controller.go:509]  "msg"="error releasing secret" "error"="resource name may not be empty" "key"="default/mybucket"                                            
I0217 02:52:29.682276       1 resourcehandlers.go:182]  "msg"="got nil configmap, skipping" "key"="default/mybucket"

I'm attempting to reproduce the problem in order to investigate it (without any success so far). If you have any additional information that can help, please update (full logs, step by step reproduction, access to a system that has this issue). Thank you.

pcuzner commented 4 years ago

I think the issue related to the sc being misconfigured. After recreating the bucketclass/storageclass OBC's are working correctly!

Thanks for the diagnose command tip!