search

kube-object-storage / lib-bucket-provisioner

Library for the dynamic provisioning of object store buckets to be used by object store providers.
Apache License 2.0
21 stars 22 forks source link

repeated errors when provisioning new obc #71

Closed jeffvance closed 5 years ago

jeffvance commented 5 years ago

E0416 14:30:26.636238 27193 resourcehandlers.go:285] "msg"="possibly intermittent, retrying" "error"="Operation cannot be fulfilled on objectbucketclaims.objectbucket.io \"screeley-provb-3\": the object has been modified; please apply your changes to the latest version and try again" "request"="s3-provisioner/screeley-provb-3"

Full log:

./jeff-prov --kubeconfig=/var/run/kubernetes/admin.kubeconfig --master=https://localhost:6443 -alsologtostderr -v=2
I0416 14:30:06.738227   27193 aws-s3-provisioner.go:499] AWS S3 Provisioner - main
I0416 14:30:06.738429   27193 aws-s3-provisioner.go:500] flags: kubeconfig="/var/run/kubernetes/admin.kubeconfig"; masterURL="https://localhost:6443"
I0416 14:30:06.740993   27193 manager.go:81] objectbucket.io/provisioner-manager "level"=0 "msg"="constructing new Provisioner"  "name"="aws-s3.io/bucket"
I0416 14:30:06.741456   27193 manager.go:92] objectbucket.io/provisioner-manager "level"=0 "msg"="generating controller manager"  
I0416 14:30:06.781048   27193 manager.go:99] objectbucket.io/provisioner-manager "level"=0 "msg"="adding schemes to manager"  
I0416 14:30:06.781209   27193 manager.go:104] objectbucket.io/provisioner-manager "level"=0 "msg"="constructing new ObjectBucketClaimReconciler"  
I0416 14:30:06.781236   27193 reconiler.go:50] objectbucket.io/claim-reconciler "level"=0 "msg"="constructing new reconciler"  "provisioner"="aws-s3.io/bucket"
I0416 14:30:06.781268   27193 reconiler.go:55] objectbucket.io/claim-reconciler "level"=1 "msg"="retry loop setting"  "RetryBaseInterval"=3
I0416 14:30:06.781288   27193 reconiler.go:59] objectbucket.io/claim-reconciler "level"=1 "msg"="retry loop setting"  "RetryTimeout"=30
I0416 14:30:06.781305   27193 manager.go:133] objectbucket.io/provisioner-manager "level"=0 "msg"="building controller manager"  
I0416 14:30:06.783062   27193 :0] objectbucket.io/provisioner-manager/kubebuilder/controller "level"=0 "msg"="Starting EventSource"  "controller"="objectbucketclaim-application" "source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{"storageClassName":"","bucketName":"","ssl":false,"cannedBucketAcl":"","versioned":false,"additionalConfig":null,"ObjectBucketName":""},"status":{"Phase":""}}}
I0416 14:30:06.783160   27193 aws-s3-provisioner.go:517] main: running aws-s3.io/bucket provisioner...
I0416 14:30:06.783200   27193 manager.go:150] objectbucket.io/provisioner-manager "level"=0 "msg"="Starting manager"  "provisioner"="aws-s3.io/bucket"
I0416 14:30:06.786099   27193 manager.go:114] objectbucket.io/provisioner-manager "level"=1 "msg"="event: Create() "  "Kind"={"Group":"objectbucket.io","Version":"v1alpha1","Kind":"ObjectBucketClaim"} "Name"="screeley-provb-3"
I0416 14:30:06.883479   27193 :0] objectbucket.io/provisioner-manager/kubebuilder/controller "level"=0 "msg"="Starting Controller"  "controller"="objectbucketclaim-application"
I0416 14:30:06.983670   27193 :0] objectbucket.io/provisioner-manager/kubebuilder/controller "level"=0 "msg"="Starting workers"  "controller"="objectbucketclaim-application" "worker count"=1
I0416 14:30:06.983759   27193 reconiler.go:81]  "level"=0 "msg"="new Reconcile iteration" "request"="s3-provisioner/screeley-provb-3" 
I0416 14:30:06.983776   27193 helpers.go:60]  "level"=0 "msg"="getting claim for key" "request"="s3-provisioner/screeley-provb-3" 
I0416 14:30:06.983810   27193 helpers.go:21]  "level"=0 "msg"="validating claim for provisioning" "request"="s3-provisioner/screeley-provb-3" 
I0416 14:30:06.983827   27193 helpers.go:120]  "level"=0 "msg"="getting storageClass for claim" "request"="s3-provisioner/screeley-provb-3" 
I0416 14:30:06.983843   27193 helpers.go:127]  "level"=0 "msg"="OBC defined class" "request"="s3-provisioner/screeley-provb-3" "name"="s3-buckets"
I0416 14:30:06.983858   27193 helpers.go:130]  "level"=0 "msg"="getting storage class" "request"="s3-provisioner/screeley-provb-3" "name"="s3-buckets"
I0416 14:30:07.084186   27193 helpers.go:141]  "level"=0 "msg"="successfully got class" "request"="s3-provisioner/screeley-provb-3" "name"=null
I0416 14:30:07.088365   27193 helpers.go:91]  "level"=0 "msg"="determining bucket name" "request"="s3-provisioner/screeley-provb-3" 
I0416 14:30:07.088388   27193 helpers.go:101]  "level"=0 "msg"="bucket name generated" "request"="s3-provisioner/screeley-provb-3" "name"="screeley-provb-3"
I0416 14:30:07.088401   27193 helpers.go:21]  "level"=0 "msg"="validating claim for provisioning" "request"="s3-provisioner/screeley-provb-3" 
I0416 14:30:07.088421   27193 reconiler.go:189]  "level"=0 "msg"="provisioning" "request"="s3-provisioner/screeley-provb-3" "bucket"="screeley-provb-3"
I0416 14:30:07.088434   27193 helpers.go:60]  "level"=0 "msg"="getting claim for key" "request"="s3-provisioner/screeley-provb-3" 
I0416 14:30:07.088465   27193 aws-s3-provisioner.go:233] initializing and setting CreateOrGrant services
I0416 14:30:07.088477   27193 util.go:34] getting storage class "s3-buckets"...
I0416 14:30:07.088698   27193 manager.go:118] objectbucket.io/provisioner-manager "level"=1 "msg"="event: Update (ignored)"  "Kind"="/, Kind=" "Name"="screeley-provb-3"
I0416 14:30:07.091027   27193 iam.go:345] storage class flag createBucketUser's value, or absence of flag, indicates to create a new user
I0416 14:30:07.091042   27193 aws-s3-provisioner.go:215] Creating AWS session based on storageclass "s3-buckets"
I0416 14:30:07.091051   27193 util.go:64] getting secret "s3-provisioner/s3-bucket-owner"...
I0416 14:30:07.093978   27193 aws-s3-provisioner.go:202] Creating AWS session using credentials from storage class s3-buckets's secret
I0416 14:30:07.094035   27193 aws-s3-provisioner.go:221] Creating S3 service based on storageclass "s3-buckets"
I0416 14:30:07.094075   27193 aws-s3-provisioner.go:342] Creating bucket "screeley-provb-3"
I0416 14:30:08.411336   27193 aws-s3-provisioner.go:162] Bucket screeley-provb-3 successfully created
I0416 14:30:08.434997   27193 util.go:109] Generated user screeley-provb-3-X5gsu after 1 iterations
I0416 14:30:08.435020   27193 iam.go:49] creating user and policy for bucket "screeley-provb-3"
I0416 14:30:08.435029   27193 iam.go:303] creating IAM user "screeley-provb-3-X5gsu"
I0416 14:30:08.495822   27193 iam.go:327] successfully created IAM user "screeley-provb-3-X5gsu" with access keys
I0416 14:30:08.495856   27193 iam.go:150] createBucketPolicyDocument for bucket "screeley-provb-3" and ARN "arn:aws:s3:::screeley-provb-3"
I0416 14:30:08.560980   27193 iam.go:217] createUserPolicy "screeley-provb-3-X5gsu" successfully created
I0416 14:30:08.561005   27193 iam.go:241] attach policy "screeley-provb-3-X5gsu" to user
I0416 14:30:08.561012   27193 iam.go:223] getting ARN for policy "screeley-provb-3-X5gsu"
I0416 14:30:08.561020   27193 iam.go:259] creating new user "screeley-provb-3-X5gsu"
I0416 14:30:08.582382   27193 iam.go:272] created user "screeley-provb-3-X5gsu" and accountID "939345161466"
I0416 14:30:08.582403   27193 iam.go:234] successfully got PolicyARN "arn:aws:iam::939345161466:policy/screeley-provb-3-X5gsu" for AccountID 939345161466's Policy "screeley-provb-3-X5gsu"
I0416 14:30:08.625058   27193 iam.go:252] successfully attached policy "screeley-provb-3-X5gsu" to user "screeley-provb-3-X5gsu"
I0416 14:30:08.625083   27193 iam.go:87] successfully created user and policy for bucket "screeley-provb-3"
I0416 14:30:08.625135   27193 helpers.go:86]  "level"=0 "msg"="setting OB name" "request"="s3-provisioner/screeley-provb-3" "name"=""
I0416 14:30:08.625155   27193 helpers.go:60]  "level"=0 "msg"="getting claim for key" "request"="s3-provisioner/screeley-provb-3" 
I0416 14:30:08.625195   27193 resourcehandlers.go:120]  "level"=0 "msg"="creating ObjectBucket" "request"="s3-provisioner/screeley-provb-3" "name"="obc-s3-provisioner-screeley-provb-3"
I0416 14:30:08.625228   27193 resourcehandlers.go:45]  "level"=0 "msg"="creating object until timeout" "request"="s3-provisioner/screeley-provb-3" "interval"=3000000000 "timeout"=30000000000
E0416 14:30:08.633512   27193 resourcehandlers.go:285]  "msg"="possibly intermittent, retrying" "error"="Operation cannot be fulfilled on objectbucketclaims.objectbucket.io \"screeley-provb-3\": the object has been modified; please apply your changes to the latest version and try again" "request"="s3-provisioner/screeley-provb-3" 
E0416 14:30:11.636487   27193 resourcehandlers.go:285]  "msg"="possibly intermittent, retrying" "error"="Operation cannot be fulfilled on objectbucketclaims.objectbucket.io \"screeley-provb-3\": the object has been modified; please apply your changes to the latest version and try again" "request"="s3-provisioner/screeley-provb-3" 
E0416 14:30:14.636467   27193 resourcehandlers.go:285]  "msg"="possibly intermittent, retrying" "error"="Operation cannot be fulfilled on objectbucketclaims.objectbucket.io \"screeley-provb-3\": the object has been modified; please apply your changes to the latest version and try again" "request"="s3-provisioner/screeley-provb-3" 
E0416 14:30:17.636340   27193 resourcehandlers.go:285]  "msg"="possibly intermittent, retrying" "error"="Operation cannot be fulfilled on objectbucketclaims.objectbucket.io \"screeley-provb-3\": the object has been modified; please apply your changes to the latest version and try again" "request"="s3-provisioner/screeley-provb-3" 
E0416 14:30:20.636376   27193 resourcehandlers.go:285]  "msg"="possibly intermittent, retrying" "error"="Operation cannot be fulfilled on objectbucketclaims.objectbucket.io \"screeley-provb-3\": the object has been modified; please apply your changes to the latest version and try again" "request"="s3-provisioner/screeley-provb-3" 
E0416 14:30:23.636373   27193 resourcehandlers.go:285]  "msg"="possibly intermittent, retrying" "error"="Operation cannot be fulfilled on objectbucketclaims.objectbucket.io \"screeley-provb-3\": the object has been modified; please apply your changes to the latest version and try again" "request"="s3-provisioner/screeley-provb-3" 
E0416 14:30:26.636238   27193 resourcehandlers.go:285]  "msg"="possibly intermittent, retrying" "error"="Operation cannot be fulfilled on objectbucketclaims.objectbucket.io \"screeley-provb-3\": the object has been modified; please apply your changes to the latest version and try again" "request"="s3-provisioner/screeley-provb-3" 
E0416 14:30:29.636205   27193 resourcehandlers.go:285]  "msg"="possibly intermittent, retrying" "error"="Operation cannot be fulfilled on objectbucketclaims.objectbucket.io \"screeley-provb-3\": the object has been modified; please apply your changes to the latest version and try again" "request"="s3-provisioner/screeley-provb-3" 
E0416 14:30:32.636338   27193 resourcehandlers.go:285]  "msg"="possibly intermittent, retrying" "error"="Operation cannot be fulfilled on objectbucketclaims.objectbucket.io \"screeley-provb-3\": the object has been modified; please apply your changes to the latest version and try again" "request"="s3-provisioner/screeley-provb-3" 
E0416 14:30:35.636463   27193 resourcehandlers.go:285]  "msg"="possibly intermittent, retrying" "error"="Operation cannot be fulfilled on objectbucketclaims.objectbucket.io \"screeley-provb-3\": the object has been modified; please apply your changes to the latest version and try again" "request"="s3-provisioner/screeley-provb-3" 
E0416 14:30:38.636270   27193 resourcehandlers.go:285]  "msg"="possibly intermittent, retrying" "error"="Operation cannot be fulfilled on objectbucketclaims.objectbucket.io \"screeley-provb-3\": the object has been modified; please apply your changes to the latest version and try again" "request"="s3-provisioner/screeley-provb-3" 
E0416 14:30:38.637899   27193 resourcehandlers.go:285]  "msg"="possibly intermittent, retrying" "error"="Operation cannot be fulfilled on objectbucketclaims.objectbucket.io \"screeley-provb-3\": the object has been modified; please apply your changes to the latest version and try again" "request"="s3-provisioner/screeley-provb-3" 
E0416 14:30:38.637929   27193 reconiler.go:152]  "msg"="cleaning up reconcile artifacts" "error"="error updating phase: timed out waiting for the condition" "request"="s3-provisioner/screeley-provb-3" 
I0416 14:30:38.637956   27193 reconiler.go:154]  "level"=0 "msg"="deleting bucket" "request"="s3-provisioner/screeley-provb-3" "name"="screeley-provb-3"
I0416 14:30:38.637967   27193 aws-s3-provisioner.go:395] Deleting bucket "screeley-provb-3" for OB "obc-s3-provisioner-screeley-provb-3"
I0416 14:30:38.637979   27193 util.go:34] getting storage class "s3-buckets"...
I0416 14:30:38.639315   27193 aws-s3-provisioner.go:215] Creating AWS session based on storageclass "s3-buckets"
I0416 14:30:38.639331   27193 util.go:64] getting secret "s3-provisioner/s3-bucket-owner"...
I0416 14:30:38.640574   27193 aws-s3-provisioner.go:202] Creating AWS session using credentials from storage class s3-buckets's secret
I0416 14:30:38.640617   27193 aws-s3-provisioner.go:221] Creating S3 service based on storageclass "s3-buckets"
I0416 14:30:38.640644   27193 iam.go:93] deleting user and policy for bucket "screeley-provb-3"
I0416 14:30:38.672963   27193 iam.go:107] successfully detached policy "arn:aws:iam::939345161466:policy/screeley-provb-3-X5gsu", user "screeley-provb-3-X5gsu"
I0416 14:30:38.720315   27193 iam.go:117] successfully deleted policy "arn:aws:iam::939345161466:policy/screeley-provb-3-X5gsu"
I0416 14:30:38.720340   27193 iam.go:279] getting access key for user "screeley-provb-3-X5gsu"
I0416 14:30:38.802054   27193 iam.go:129] successfully deleted access key for user "screeley-provb-3-X5gsu"
I0416 14:30:38.802078   27193 iam.go:133] Deleting User "screeley-provb-3-X5gsu"
I0416 14:30:38.830286   27193 iam.go:142] successfully deleted user and policy for bucket "screeley-provb-3"
I0416 14:30:38.830315   27193 aws-s3-provisioner.go:424] Deleting all objects in bucket "screeley-provb-3" (from OB "obc-s3-provisioner-screeley-provb-3")
I0416 14:30:39.140218   27193 aws-s3-provisioner.go:430] Deleting empty bucket "screeley-provb-3" from OB "obc-s3-provisioner-screeley-provb-3"
I0416 14:30:39.472895   27193 aws-s3-provisioner.go:437] Deleted bucket "screeley-provb-3" from OB "obc-s3-provisioner-screeley-provb-3"
I0416 14:30:39.472950   27193 resourcehandlers.go:219]  "level"=0 "msg"="deleting ObjectBucket" "request"="s3-provisioner/screeley-provb-3" "name"="obc-s3-provisioner-screeley-provb-3"
I0416 14:30:39.472975   27193 helpers.go:170]  "level"=0 "msg"="checking for finalizer" "request"="s3-provisioner/screeley-provb-3" "object"="obc-s3-provisioner-screeley-provb-3" "value"="objectbucket.io/finalizer"
I0416 14:30:39.472990   27193 helpers.go:173]  "level"=0 "msg"="found finalizer in obj" "request"="s3-provisioner/screeley-provb-3" 
I0416 14:30:39.473004   27193 helpers.go:182]  "level"=0 "msg"="removing finalizer from object" "request"="s3-provisioner/screeley-provb-3" "name"="obc-s3-provisioner-screeley-provb-3"
I0416 14:30:39.473017   27193 helpers.go:191]  "level"=0 "msg"="found finalizer, deleting and updating API" "request"="s3-provisioner/screeley-provb-3" 
I0416 14:30:39.476087   27193 helpers.go:197]  "level"=0 "msg"="finalizer deletion successful" "request"="s3-provisioner/screeley-provb-3" 
I0416 14:30:39.479563   27193 resourcehandlers.go:186]  "level"=0 "msg"="got nil secret, skipping" "request"="s3-provisioner/screeley-provb-3" 
I0416 14:30:39.479586   27193 resourcehandlers.go:156]  "level"=0 "msg"="got nil configMap pointer, skipping delete" "request"="s3-provisioner/screeley-provb-3" 
E0416 14:30:39.479642   27193 :0] objectbucket.io/provisioner-manager/kubebuilder/controller "msg"="Reconciler error" "error"="error updating phase: timed out waiting for the condition"  "controller"="objectbucketclaim-application" "request"={"Namespace":"s3-provisioner","Name":"screeley-provb-3"}
I0416 14:30:40.479896   27193 reconiler.go:81]  "level"=0 "msg"="new Reconcile iteration" "request"="s3-provisioner/screeley-provb-3" 
I0416 14:30:40.479927   27193 helpers.go:60]  "level"=0 "msg"="getting claim for key" "request"="s3-provisioner/screeley-provb-3" 
I0416 14:30:40.480215   27193 helpers.go:21]  "level"=0 "msg"="validating claim for provisioning" "request"="s3-provisioner/screeley-provb-3" 
I0416 14:30:40.480239   27193 helpers.go:120]  "level"=0 "msg"="getting storageClass for claim" "request"="s3-provisioner/screeley-provb-3" 
I0416 14:30:40.480265   27193 helpers.go:127]  "level"=0 "msg"="OBC defined class" "request"="s3-provisioner/screeley-provb-3" "name"="s3-buckets"
I0416 14:30:40.480287   27193 helpers.go:130]  "level"=0 "msg"="getting storage class" "request"="s3-provisioner/screeley-provb-3" "name"="s3-buckets"
I0416 14:30:40.480316   27193 helpers.go:141]  "level"=0 "msg"="successfully got class" "request"="s3-provisioner/screeley-provb-3" "name"=null
I0416 14:30:40.484126   27193 helpers.go:91]  "level"=0 "msg"="determining bucket name" "request"="s3-provisioner/screeley-provb-3" 
I0416 14:30:40.484153   27193 helpers.go:101]  "level"=0 "msg"="bucket name generated" "request"="s3-provisioner/screeley-provb-3" "name"="screeley-provb-3"
I0416 14:30:40.484167   27193 helpers.go:21]  "level"=0 "msg"="validating claim for provisioning" "request"="s3-provisioner/screeley-provb-3" 
I0416 14:30:40.484183   27193 reconiler.go:189]  "level"=0 "msg"="provisioning" "request"="s3-provisioner/screeley-provb-3" "bucket"="screeley-provb-3"
I0416 14:30:40.484195   27193 helpers.go:60]  "level"=0 "msg"="getting claim for key" "request"="s3-provisioner/screeley-provb-3" 
I0416 14:30:40.484210   27193 aws-s3-provisioner.go:233] initializing and setting CreateOrGrant services
I0416 14:30:40.484219   27193 util.go:34] getting storage class "s3-buckets"...
I0416 14:30:40.485737   27193 iam.go:345] storage class flag createBucketUser's value, or absence of flag, indicates to create a new user
I0416 14:30:40.485752   27193 aws-s3-provisioner.go:215] Creating AWS session based on storageclass "s3-buckets"
I0416 14:30:40.485761   27193 util.go:64] getting secret "s3-provisioner/s3-bucket-owner"...
I0416 14:30:40.487057   27193 aws-s3-provisioner.go:202] Creating AWS session using credentials from storage class s3-buckets's secret
I0416 14:30:40.487147   27193 aws-s3-provisioner.go:221] Creating S3 service based on storageclass "s3-buckets"
I0416 14:30:40.487175   27193 aws-s3-provisioner.go:342] Creating bucket "screeley-provb-3"
I0416 14:30:41.229210   27193 aws-s3-provisioner.go:162] Bucket screeley-provb-3 successfully created
I0416 14:30:41.245367   27193 util.go:109] Generated user screeley-provb-3-QdoNY after 1 iterations
I0416 14:30:41.245386   27193 iam.go:49] creating user and policy for bucket "screeley-provb-3"
I0416 14:30:41.245394   27193 iam.go:303] creating IAM user "screeley-provb-3-QdoNY"
I0416 14:30:41.306068   27193 iam.go:327] successfully created IAM user "screeley-provb-3-QdoNY" with access keys
I0416 14:30:41.306091   27193 iam.go:150] createBucketPolicyDocument for bucket "screeley-provb-3" and ARN "arn:aws:s3:::screeley-provb-3"
I0416 14:30:41.360274   27193 iam.go:217] createUserPolicy "screeley-provb-3-QdoNY" successfully created
I0416 14:30:41.360299   27193 iam.go:241] attach policy "screeley-provb-3-QdoNY" to user
I0416 14:30:41.360306   27193 iam.go:223] getting ARN for policy "screeley-provb-3-QdoNY"
I0416 14:30:41.360312   27193 iam.go:259] creating new user "screeley-provb-3-QdoNY"
I0416 14:30:41.380312   27193 iam.go:272] created user "screeley-provb-3-QdoNY" and accountID "939345161466"
I0416 14:30:41.380333   27193 iam.go:234] successfully got PolicyARN "arn:aws:iam::939345161466:policy/screeley-provb-3-QdoNY" for AccountID 939345161466's Policy "screeley-provb-3-QdoNY"
I0416 14:30:41.420277   27193 iam.go:252] successfully attached policy "screeley-provb-3-QdoNY" to user "screeley-provb-3-QdoNY"
I0416 14:30:41.420301   27193 iam.go:87] successfully created user and policy for bucket "screeley-provb-3"
I0416 14:30:41.420335   27193 helpers.go:86]  "level"=0 "msg"="setting OB name" "request"="s3-provisioner/screeley-provb-3" "name"=""
I0416 14:30:41.420353   27193 helpers.go:60]  "level"=0 "msg"="getting claim for key" "request"="s3-provisioner/screeley-provb-3" 
I0416 14:30:41.420383   27193 resourcehandlers.go:120]  "level"=0 "msg"="creating ObjectBucket" "request"="s3-provisioner/screeley-provb-3" "name"="obc-s3-provisioner-screeley-provb-3"
I0416 14:30:41.420402   27193 resourcehandlers.go:45]  "level"=0 "msg"="creating object until timeout" "request"="s3-provisioner/screeley-provb-3" "interval"=3000000000 "timeout"=30000000000
I0416 14:30:41.427987   27193 manager.go:118] objectbucket.io/provisioner-manager "level"=1 "msg"="event: Update (ignored)"  "Kind"="/, Kind=" "Name"="screeley-provb-3"
I0416 14:30:41.428538   27193 resourcehandlers.go:133]  "level"=0 "msg"="creating Secret" "request"="s3-provisioner/screeley-provb-3" "name"="screeley-provb-3" "namespace"="s3-provisioner"
I0416 14:30:41.428562   27193 resourcehandlers.go:45]  "level"=0 "msg"="creating object until timeout" "request"="s3-provisioner/screeley-provb-3" "interval"=3000000000 "timeout"=30000000000
I0416 14:30:41.431599   27193 resourcehandlers.go:71]  "level"=0 "msg"="defining new configMap" "request"="s3-provisioner/screeley-provb-3" "for claim"="s3-provisioner/screeley-provb-3"
I0416 14:30:41.431627   27193 resourcehandlers.go:146]  "level"=0 "msg"="creating configMap" "request"="s3-provisioner/screeley-provb-3" "name"="screeley-provb-3" "namespace"="s3-provisioner"
I0416 14:30:41.431644   27193 resourcehandlers.go:45]  "level"=0 "msg"="creating object until timeout" "request"="s3-provisioner/screeley-provb-3" "interval"=3000000000 "timeout"=30000000000
I0416 14:30:41.436236   27193 reconiler.go:246]  "level"=0 "msg"="provisioning succeeded" "request"="s3-provisioner/screeley-provb-3" 
I0416 14:30:41.436275   27193 controller.go:236] objectbucket.io/provisioner-manager/kubebuilder/controller "level"=1 "msg"="Successfully Reconciled"  "controller"="objectbucketclaim-application" "request"={"Namespace":"s3-provisioner","Name":"screeley-provb-3"}

Eventually recovers and the k8s resources are created. There seems to be a lot of overhead creating and then deleting aws and k8s resources in this code path...

copejon commented 5 years ago

Eventually recovers and the k8s resources are created. There seems to be a lot of overhead creating and then deleting aws and k8s resources in this code path...

That's true. If a reconcile iteration can't succeed, it cleans up any resources that were created. I still think that's a necessary step. There are also smaller internal loops that will retry k8s resource operations on error a few times in quick succession in case of errors that may not be fatal.

What you're seeing is Reconcile() being retried by the controller at an exponential backoff. Conceptually, do we want to do as k8s does and continuously retry until actual state of the world == desired state of the world? If not, then we need a way to differentiate errors returned by provisioners that doesn't have us parsing strings. Maybe typed errors?

copejon commented 5 years ago

@screeley44 @jeffvance Has the bug resurfaced or can we close this?

jeffvance commented 5 years ago

Have not seen this for a while...