search

cert-manager / aws-privateca-issuer

Addon for cert-manager that issues certificates using AWS ACM PCA.
Apache License 2.0
192 stars 81 forks source link

[Troubleshoot]: Reconciler error StorageError: invalid object, Code: 4 #329

Closed FabioAntunes closed 3 months ago

FabioAntunes commented 3 months ago

Describe the expected outcome

Create certificates.

Describe the actual outcome

We are getting the following errors can someone point us what could be the possible root cause?

{
  "level": "error",
  "ts": "2024-07-11T20:58:35Z",
  "msg": "Reconciler error",
  "controller": "certificaterequest",
  "controllerGroup": "cert-manager.io",
  "controllerKind": "CertificateRequest",
  "CertificateRequest": {
    "name": "istio-csr-vx59z",
    "namespace": "istio-system"
  },
  "namespace": "istio-system",
  "name": "istio-csr-vx59z",
  "reconcileID": "9bef08f4-dc04-4929-b582-800a054a2495",
  "error": "Operation cannot be fulfilled on certificaterequests.cert-manager.io \"istio-csr-vx59z\": StorageError: invalid object, Code: 4, Key: /registry/cert-manager.io/certificaterequests/istio-system/istio-csr-vx59z, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: cdc694e7-89bb-4c87-8745-2898ec235b66, UID in object meta: ",
  "stacktrace": "sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:235"
}

This was fresh installation on a cluster we were on version 1.2.5 and even after updating to 1.2.7 the issue persists.

The pod starts fine and signs a couple of certificates and after a while it just keeps on erroring:

{"level":"info","ts":"2024-07-11T21:21:19Z","logger":"controllers.CertificateRequest","msg":"Created certificate with arn: arn:aws:acm-pca:eu-west-1:111111111111:certificate-authority/cccccccc-dddd-4444-bbbb-dddddddddddd/certificate/086c62b520dbc87632f7b1c70841c8a3","certificaterequest":{"name":"istio-csr-74cd2","namespace":"istio-system"}
{"level":"info","ts":"2024-07-11T21:21:22Z","logger":"controllers.CertificateRequest","msg":"Created certificate with arn: arn:aws:acm-pca:eu-west-1:111111111111:certificate-authority/cccccccc-dddd-4444-bbbb-dddddddddddd/certificate/ce46ac96b76b9b76ed85c6a1045b337e","certificaterequest":{"name":"istio-csr-xr6sr","namespace":"istio-system"}}
{"level":"info","ts":"2024-07-11T21:21:25Z","logger":"controllers.CertificateRequest","msg":"Created certificate with arn: arn:aws:acm-pca:eu-west-1:111111111111:certificate-authority/cccccccc-dddd-4444-bbbb-dddddddddddd/certificate/acccf3f7b2a5d5720630549dbee4eebe","certificaterequest":{"name":"istio-csr-4sgdc","namespace":"istio-system"}}
{"level":"info","ts":"2024-07-11T21:21:28Z","logger":"controllers.CertificateRequest","msg":"Created certificate with arn: arn:aws:acm-pca:eu-west-1:111111111111:certificate-authority/cccccccc-dddd-4444-bbbb-dddddddddddd/certificate/1363d551d5008673f78b328cbadc081a","certificaterequest":{"name":"istio-csr-tsqqb","namespace":"istio-system"}}
{"level":"info","ts":"2024-07-11T21:21:32Z","logger":"controllers.CertificateRequest","msg":"Created certificate with arn: arn:aws:acm-pca:eu-west-1:111111111111:certificate-authority/cccccccc-dddd-4444-bbbb-dddddddddddd/certificate/63eed2d60abab75686790d8067528eef","certificaterequest":{"name":"istio-csr-w2nbq","namespace":"istio-system"}}
{"level":"info","ts":"2024-07-11T21:21:35Z","logger":"controllers.CertificateRequest","msg":"Created certificate with arn: arn:aws:acm-pca:eu-west-1:111111111111:certificate-authority/cccccccc-dddd-4444-bbbb-dddddddddddd/certificate/a1c53cc8b8a777f70b1271cc4bb0550e","certificaterequest":{"name":"istio-csr-2hp4b","namespace":"istio-system"}}
{"level":"info","ts":"2024-07-11T21:21:38Z","logger":"controllers.CertificateRequest","msg":"Created certificate with arn: arn:aws:acm-pca:eu-west-1:111111111111:certificate-authority/cccccccc-dddd-4444-bbbb-dddddddddddd/certificate/5668b6797cf7426045738499ee4d4611","certificaterequest":{"name":"istio-csr-pw2bc","namespace":"istio-system"}}
{"level":"info","ts":"2024-07-11T21:21:41Z","logger":"controllers.CertificateRequest","msg":"Created certificate with arn: arn:aws:acm-pca:eu-west-1:111111111111:certificate-authority/cccccccc-dddd-4444-bbbb-dddddddddddd/certificate/4a809e692c639eba0af1d1cbb28bbeb4","certificaterequest":{"name":"istio-csr-jsz85","namespace":"istio-system"}}
{"level":"info","ts":"2024-07-11T21:21:45Z","logger":"controllers.CertificateRequest","msg":"Created certificate with arn: arn:aws:acm-pca:eu-west-1:111111111111:certificate-authority/cccccccc-dddd-4444-bbbb-dddddddddddd/certificate/e52c16c5b59fc154c68c8002d6a2224d","certificaterequest":{"name":"istio-csr-fr6q4","namespace":"istio-system"}}
{"level":"info","ts":"2024-07-11T21:21:48Z","logger":"controllers.CertificateRequest","msg":"Created certificate with arn: arn:aws:acm-pca:eu-west-1:111111111111:certificate-authority/cccccccc-dddd-4444-bbbb-dddddddddddd/certificate/ed2d4a085176ce0ad0916ab97905f295","certificaterequest":{"name":"istio-csr-mlqzn","namespace":"istio-system"}}
{"level":"info","ts":"2024-07-11T21:21:51Z","logger":"controllers.CertificateRequest","msg":"Created certificate with arn: arn:aws:acm-pca:eu-west-1:111111111111:certificate-authority/cccccccc-dddd-4444-bbbb-dddddddddddd/certificate/8651e097d7e90c7408c1f93ddf20cea0","certificaterequest":{"name":"istio-csr-5lxdp","namespace":"istio-system"}}
{"level":"info","ts":"2024-07-11T21:21:54Z","logger":"controllers.CertificateRequest","msg":"Created certificate with arn: arn:aws:acm-pca:eu-west-1:111111111111:certificate-authority/cccccccc-dddd-4444-bbbb-dddddddddddd/certificate/92508cd2360f0e5a822095296c26b36b","certificaterequest":{"name":"istio-csr-vtzh4","namespace":"istio-system"}}
{"level":"info","ts":"2024-07-11T21:21:58Z","logger":"controllers.CertificateRequest","msg":"Created certificate with arn: arn:aws:acm-pca:eu-west-1:111111111111:certificate-authority/cccccccc-dddd-4444-bbbb-dddddddddddd/certificate/48501e5b439555d1f9161edff48fc42f","certificaterequest":{"name":"istio-csr-7ld4m","namespace":"istio-system"}}
{"level":"info","ts":"2024-07-11T21:22:01Z","logger":"controllers.CertificateRequest","msg":"Created certificate with arn: arn:aws:acm-pca:eu-west-1:111111111111:certificate-authority/cccccccc-dddd-4444-bbbb-dddddddddddd/certificate/c859a2762b454de1f00562921970fcea","certificaterequest":{"name":"istio-csr-bwfj7","namespace":"istio-system"}}
{"level":"info","ts":"2024-07-11T21:22:04Z","logger":"controllers.CertificateRequest","msg":"Created certificate with arn: arn:aws:acm-pca:eu-west-1:111111111111:certificate-authority/cccccccc-dddd-4444-bbbb-dddddddddddd/certificate/fbd889f73d30699dc6882e8669510566","certificaterequest":{"name":"istio-csr-65rck","namespace":"istio-system"}}
{"level":"info","ts":"2024-07-11T21:22:07Z","logger":"controllers.CertificateRequest","msg":"Created certificate with arn: arn:aws:acm-pca:eu-west-1:111111111111:certificate-authority/cccccccc-dddd-4444-bbbb-dddddddddddd/certificate/2314a4e1d389fba753f778eca59e73d3","certificaterequest":{"name":"istio-csr-hctps","namespace":"istio-system"}}
{"level":"info","ts":"2024-07-11T21:22:11Z","logger":"controllers.CertificateRequest","msg":"Created certificate with arn: arn:aws:acm-pca:eu-west-1:111111111111:certificate-authority/cccccccc-dddd-4444-bbbb-dddddddddddd/certificate/1c6ac1721ab5db1ed7a3afbb2c41e0c8","certificaterequest":{"name":"istio-csr-h6s5x","namespace":"istio-system"}}
{"level":"info","ts":"2024-07-11T21:22:14Z","logger":"controllers.CertificateRequest","msg":"Created certificate with arn: arn:aws:acm-pca:eu-west-1:111111111111:certificate-authority/cccccccc-dddd-4444-bbbb-dddddddddddd/certificate/5bfb97f192ece6d7c787eca66cc384a1","certificaterequest":{"name":"istio-csr-j8jb7","namespace":"istio-system"}}
{"level":"info","ts":"2024-07-11T21:22:17Z","logger":"controllers.CertificateRequest","msg":"Created certificate with arn: arn:aws:acm-pca:eu-west-1:111111111111:certificate-authority/cccccccc-dddd-4444-bbbb-dddddddddddd/certificate/0c66c4245dae37f31f740ffeddd659cd","certificaterequest":{"name":"istio-csr-w2kgd","namespace":"istio-system"}}
{"level":"info","ts":"2024-07-11T21:22:20Z","logger":"controllers.CertificateRequest","msg":"Created certificate with arn: arn:aws:acm-pca:eu-west-1:111111111111:certificate-authority/cccccccc-dddd-4444-bbbb-dddddddddddd/certificate/09e080cef89aba42cf47d833e8ef17a4","certificaterequest":{"name":"istio-csr-9rlvj","namespace":"istio-system"}}
{"level":"info","ts":"2024-07-11T21:22:23Z","logger":"controllers.CertificateRequest","msg":"Created certificate with arn: arn:aws:acm-pca:eu-west-1:111111111111:certificate-authority/cccccccc-dddd-4444-bbbb-dddddddddddd/certificate/38433e2520d2f78b6bc318217fabb61f","certificaterequest":{"name":"istio-csr-4nnjq","namespace":"istio-system"}}
{"level":"info","ts":"2024-07-11T21:22:27Z","logger":"controllers.CertificateRequest","msg":"Created certificate with arn: arn:aws:acm-pca:eu-west-1:111111111111:certificate-authority/cccccccc-dddd-4444-bbbb-dddddddddddd/certificate/43e4d8bb3f3bb7b2ba043534fcd5e2de","certificaterequest":{"name":"istio-csr-wldbt","namespace":"istio-system"}}
{"level":"info","ts":"2024-07-11T21:22:30Z","logger":"controllers.CertificateRequest","msg":"Created certificate with arn: arn:aws:acm-pca:eu-west-1:111111111111:certificate-authority/cccccccc-dddd-4444-bbbb-dddddddddddd/certificate/830dfa03537874f55018eff6593e3d41","certificaterequest":{"name":"istio-csr-s8m2h","namespace":"istio-system"}}
{"level":"error","ts":"2024-07-11T21:22:33Z","msg":"Reconciler error","controller":"certificaterequest","controllerGroup":"cert-manager.io","controllerKind":"CertificateRequest","CertificateRequest":{"name":"istio-csr-s8m2h","namespace":"istio-system"},"namespace":"istio-system","name":"istio-csr-s8m2h","reconcileID":"747887b4-eb48-4cd3-b7c7-31e9eec7a47d","error":"Operation cannot be fulfilled on certificaterequests.cert-manager.io \"istio-csr-s8m2h\": StorageError: invalid object, Code: 4, Key: /registry/cert-manager.io/certificaterequests/istio-system/istio-csr-s8m2h, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 5a74bd58-cf52-428d-83d1-3c396cc5bfc1, UID in object meta: ","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:227"}
{"level":"info","ts":"2024-07-11T21:22:33Z","logger":"controllers.CertificateRequest","msg":"Created certificate with arn: arn:aws:acm-pca:eu-west-1:111111111111:certificate-authority/cccccccc-dddd-4444-bbbb-dddddddddddd/certificate/bc12e5fd8524c9f9df7c5ca0daff3ca4","certificaterequest":{"name":"istio-csr-bbh8l","namespace":"istio-system"}}
{"level":"error","ts":"2024-07-11T21:22:36Z","msg":"Reconciler error","controller":"certificaterequest","controllerGroup":"cert-manager.io","controllerKind":"CertificateRequest","CertificateRequest":{"name":"istio-csr-bbh8l","namespace":"istio-system"},"namespace":"istio-system","name":"istio-csr-bbh8l","reconcileID":"a15f399d-d7a6-4235-9504-e00255c0a96b","error":"Operation cannot be fulfilled on certificaterequests.cert-manager.io \"istio-csr-bbh8l\": StorageError: invalid object, Code: 4, Key: /registry/cert-manager.io/certificaterequests/istio-system/istio-csr-bbh8l, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: c897e717-de1a-42cd-8ee7-7465e06e1933, UID in object meta: ","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:227"}
{"level":"info","ts":"2024-07-11T21:22:36Z","logger":"controllers.CertificateRequest","msg":"Created certificate with arn: arn:aws:acm-pca:eu-west-1:111111111111:certificate-authority/cccccccc-dddd-4444-bbbb-dddddddddddd/certificate/f2a9c272e41e64b0beee3fa701bcd267","certificaterequest":{"name":"istio-csr-rqcs6","namespace":"istio-system"}}
{"level":"error","ts":"2024-07-11T21:22:39Z","msg":"Reconciler error","controller":"certificaterequest","controllerGroup":"cert-manager.io","controllerKind":"CertificateRequest","CertificateRequest":{"name":"istio-csr-rqcs6","namespace":"istio-system"},"namespace":"istio-system","name":"istio-csr-rqcs6","reconcileID":"dbc0bf65-0047-42c7-b30e-777c59bca54b","error":"Operation cannot be fulfilled on certificaterequests.cert-manager.io \"istio-csr-rqcs6\": StorageError: invalid object, Code: 4, Key: /registry/cert-manager.io/certificaterequests/istio-system/istio-csr-rqcs6, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: d2a5535c-5764-41dd-8a67-8e796e70cf74, UID in object meta: ","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:227"}
{"level":"info","ts":"2024-07-11T21:22:40Z","logger":"controllers.CertificateRequest","msg":"Created certificate with arn: arn:aws:acm-pca:eu-west-1:111111111111:certificate-authority/cccccccc-dddd-4444-bbbb-dddddddddddd/certificate/9712d7fcc73159c23c8d30ee5c8dcf7e","certificaterequest":{"name":"istio-csr-jq58j","namespace":"istio-system"}}
{"level":"error","ts":"2024-07-11T21:22:43Z","msg":"Reconciler error","controller":"certificaterequest","controllerGroup":"cert-manager.io","controllerKind":"CertificateRequest","CertificateRequest":{"name":"istio-csr-jq58j","namespace":"istio-system"},"namespace":"istio-system","name":"istio-csr-jq58j","reconcileID":"1cdabc6c-fb1f-4d5a-9bb8-0ec25e709667","error":"Operation cannot be fulfilled on certificaterequests.cert-manager.io \"istio-csr-jq58j\": StorageError: invalid object, Code: 4, Key: /registry/cert-manager.io/certificaterequests/istio-system/istio-csr-jq58j, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 680d59a6-8d8e-4c46-86f7-8b408e862c1d, UID in object meta: ","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:227"}
{"level":"info","ts":"2024-07-11T21:22:43Z","logger":"controllers.CertificateRequest","msg":"Created certificate with arn: arn:aws:acm-pca:eu-west-1:111111111111:certificate-authority/cccccccc-dddd-4444-bbbb-dddddddddddd/certificate/21740b5f5867aadcf64f5b9d5c187838","certificaterequest":{"name":"istio-csr-hjpgr","namespace":"istio-system"}}
{"level":"error","ts":"2024-07-11T21:22:46Z","msg":"Reconciler error","controller":"certificaterequest","controllerGroup":"cert-manager.io","controllerKind":"CertificateRequest","CertificateRequest":{"name":"istio-csr-hjpgr","namespace":"istio-system"},"namespace":"istio-system","name":"istio-csr-hjpgr","reconcileID":"98423457-bd5c-4f17-853a-bc89d3cf04b1","error":"Operation cannot be fulfilled on certificaterequests.cert-manager.io \"istio-csr-hjpgr\": StorageError: invalid object, Code: 4, Key: /registry/cert-manager.io/certificaterequests/istio-system/istio-csr-hjpgr, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: f40d5db3-c9ae-4022-a6a1-e5c86daab72f, UID in object meta: ","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:227"}
{"level":"info","ts":"2024-07-11T21:22:46Z","logger":"controllers.CertificateRequest","msg":"Created certificate with arn: arn:aws:acm-pca:eu-west-1:111111111111:certificate-authority/cccccccc-dddd-4444-bbbb-dddddddddddd/certificate/bdec6e1422a60a8eea370dd767c79b33","certificaterequest":{"name":"istio-csr-zkp97","namespace":"istio-system"}}
{"level":"error","ts":"2024-07-11T21:22:49Z","msg":"Reconciler error","controller":"certificaterequest","controllerGroup":"cert-manager.io","controllerKind":"CertificateRequest","CertificateRequest":{"name":"istio-csr-zkp97","namespace":"istio-system"},"namespace":"istio-system","name":"istio-csr-zkp97","reconcileID":"09fb9a4e-1982-41db-b544-f791fca60950","error":"Operation cannot be fulfilled on certificaterequests.cert-manager.io \"istio-csr-zkp97\": StorageError: invalid object, Code: 4, Key: /registry/cert-manager.io/certificaterequests/istio-system/istio-csr-zkp97, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 41abe8bb-196f-4f06-8b49-eb5978f4d81c, UID in object meta: ","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:227"}
{"level":"info","ts":"2024-07-11T21:22:49Z","logger":"controllers.CertificateRequest","msg":"Created certificate with arn: arn:aws:acm-pca:eu-west-1:111111111111:certificate-authority/cccccccc-dddd-4444-bbbb-dddddddddddd/certificate/1a2c04b63b32917829eca0839ab4eb75","certificaterequest":{"name":"istio-csr-6kkm6","namespace":"istio-system"}}
{"level":"error","ts":"2024-07-11T21:22:52Z","msg":"Reconciler error","controller":"certificaterequest","controllerGroup":"cert-manager.io","controllerKind":"CertificateRequest","CertificateRequest":{"name":"istio-csr-6kkm6","namespace":"istio-system"},"namespace":"istio-system","name":"istio-csr-6kkm6","reconcileID":"5fe3288a-49be-4093-9895-fdd4dd867a04","error":"Operation cannot be fulfilled on certificaterequests.cert-manager.io \"istio-csr-6kkm6\": StorageError: invalid object, Code: 4, Key: /registry/cert-manager.io/certificaterequests/istio-system/istio-csr-6kkm6, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 1d5ff2f2-7801-4151-b9bf-6169d94b0af2, UID in object meta: ","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:227"}
{"level":"info","ts":"2024-07-11T21:22:52Z","logger":"controllers.CertificateRequest","msg":"Created certificate with arn: arn:aws:acm-pca:eu-west-1:111111111111:certificate-authority/cccccccc-dddd-4444-bbbb-dddddddddddd/certificate/4e1fe5e8945ff38e84cff0765a0899ce","certificaterequest":{"name":"istio-csr-d9wv4","namespace":"istio-system"}}
{"level":"error","ts":"2024-07-11T21:22:56Z","msg":"Reconciler error","controller":"certificaterequest","controllerGroup":"cert-manager.io","controllerKind":"CertificateRequest","CertificateRequest":{"name":"istio-csr-d9wv4","namespace":"istio-system"},"namespace":"istio-system","name":"istio-csr-d9wv4","reconcileID":"717028ce-a681-45be-8722-234838caa8eb","error":"Operation cannot be fulfilled on certificaterequests.cert-manager.io \"istio-csr-d9wv4\": StorageError: invalid object, Code: 4, Key: /registry/cert-manager.io/certificaterequests/istio-system/istio-csr-d9wv4, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 85c68e8f-b471-47ba-a478-f9213dbeb782, UID in object meta: ","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:227"}
{"level":"info","ts":"2024-07-11T21:22:56Z","logger":"controllers.CertificateRequest","msg":"Created certificate with arn: arn:aws:acm-pca:eu-west-1:111111111111:certificate-authority/cccccccc-dddd-4444-bbbb-dddddddddddd/certificate/7b954a013cbac45f0f1237a4dff09beb","certificaterequest":{"name":"istio-csr-rlsf5","namespace":"istio-system"}}
{"level":"error","ts":"2024-07-11T21:22:59Z","msg":"Reconciler error","controller":"certificaterequest","controllerGroup":"cert-manager.io","controllerKind":"CertificateRequest","CertificateRequest":{"name":"istio-csr-rlsf5","namespace":"istio-system"},"namespace":"istio-system","name":"istio-csr-rlsf5","reconcileID":"02672e7e-b3b7-4a8e-b2fc-1bbb59b61d29","error":"Operation cannot be fulfilled on certificaterequests.cert-manager.io \"istio-csr-rlsf5\": StorageError: invalid object, Code: 4, Key: /registry/cert-manager.io/certificaterequests/istio-system/istio-csr-rlsf5, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 5826ad83-8829-4bff-94ed-71263f76ca3d, UID in object meta: ","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:227"}
{"level":"info","ts":"2024-07-11T21:22:59Z","logger":"controllers.CertificateRequest","msg":"Created certificate with arn: arn:aws:acm-pca:eu-west-1:111111111111:certificate-authority/cccccccc-dddd-4444-bbbb-dddddddddddd/certificate/19f0e74278d056dbf60a8b3a56f41c3c","certificaterequest":{"name":"istio-csr-h6jmp","namespace":"istio-system"}}
{"level":"error","ts":"2024-07-11T21:23:02Z","msg":"Reconciler error","controller":"certificaterequest","controllerGroup":"cert-manager.io","controllerKind":"CertificateRequest","CertificateRequest":{"name":"istio-csr-h6jmp","namespace":"istio-system"},"namespace":"istio-system","name":"istio-csr-h6jmp","reconcileID":"75f699c7-be0b-4892-8d28-ca330c5f2a8e","error":"Operation cannot be fulfilled on certificaterequests.cert-manager.io \"istio-csr-h6jmp\": StorageError: invalid object, Code: 4, Key: /registry/cert-manager.io/certificaterequests/istio-system/istio-csr-h6jmp, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: f02725d5-0610-480d-97f6-a5ed0c0cb42f, UID in object meta: ","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:227"}
{"level":"info","ts":"2024-07-11T21:23:02Z","logger":"controllers.CertificateRequest","msg":"Created certificate with arn: arn:aws:acm-pca:eu-west-1:111111111111:certificate-authority/cccccccc-dddd-4444-bbbb-dddddddddddd/certificate/7f3f336abd9f53c09d0b022e406c1abb","certificaterequest":{"name":"istio-csr-g8wnd","namespace":"istio-system"}}
{"level":"error","ts":"2024-07-11T21:23:05Z","msg":"Reconciler error","controller":"certificaterequest","controllerGroup":"cert-manager.io","controllerKind":"CertificateRequest","CertificateRequest":{"name":"istio-csr-g8wnd","namespace":"istio-system"},"namespace":"istio-system","name":"istio-csr-g8wnd","reconcileID":"eff95137-95f6-4eaf-9462-e7d41f9f5ca8","error":"Operation cannot be fulfilled on certificaterequests.cert-manager.io \"istio-csr-g8wnd\": StorageError: invalid object, Code: 4, Key: /registry/cert-manager.io/certificaterequests/istio-system/istio-csr-g8wnd, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: e3ed7235-1a07-437e-aa84-85b6d73ca07f, UID in object meta: ","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:227"}
{"level":"info","ts":"2024-07-11T21:23:05Z","logger":"controllers.CertificateRequest","msg":"Created certificate with arn: arn:aws:acm-pca:eu-west-1:111111111111:certificate-authority/cccccccc-dddd-4444-bbbb-dddddddddddd/certificate/5ce31dfdb5b268392f132c3ce12a107c","certificaterequest":{"name":"istio-csr-xf4cr","namespace":"istio-system"}}
{"level":"error","ts":"2024-07-11T21:23:08Z","msg":"Reconciler error","controller":"certificaterequest","controllerGroup":"cert-manager.io","controllerKind":"CertificateRequest","CertificateRequest":{"name":"istio-csr-xf4cr","namespace":"istio-system"},"namespace":"istio-system","name":"istio-csr-xf4cr","reconcileID":"44456ff2-be9a-4509-a93b-fa125c114ef5","error":"Operation cannot be fulfilled on certificaterequests.cert-manager.io \"istio-csr-xf4cr\": StorageError: invalid object, Code: 4, Key: /registry/cert-manager.io/certificaterequests/istio-system/istio-csr-xf4cr, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 8acb81d2-2e3e-42a6-9f42-6d0105650f78, UID in object meta: ","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:227"}
{"level":"info","ts":"2024-07-11T21:23:09Z","logger":"controllers.CertificateRequest","msg":"Created certificate with arn: arn:aws:acm-pca:eu-west-1:111111111111:certificate-authority/cccccccc-dddd-4444-bbbb-dddddddddddd/certificate/49c841b505a4310e853768f86c7926d3","certificaterequest":{"name":"istio-csr-k45ld","namespace":"istio-system"}}
{"level":"error","ts":"2024-07-11T21:23:12Z","msg":"Reconciler error","controller":"certificaterequest","controllerGroup":"cert-manager.io","controllerKind":"CertificateRequest","CertificateRequest":{"name":"istio-csr-k45ld","namespace":"istio-system"},"namespace":"istio-system","name":"istio-csr-k45ld","reconcileID":"5aeb7130-603f-4d6d-9040-2b1c2b6bcc4d","error":"Operation cannot be fulfilled on certificaterequests.cert-manager.io \"istio-csr-k45ld\": StorageError: invalid object, Code: 4, Key: /registry/cert-manager.io/certificaterequests/istio-system/istio-csr-k45ld, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: d3e38bcf-fb73-418a-962c-b20dd21fbc04, UID in object meta: ","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:227"}
{"level":"info","ts":"2024-07-11T21:23:12Z","logger":"controllers.CertificateRequest","msg":"Created certificate with arn: arn:aws:acm-pca:eu-west-1:111111111111:certificate-authority/cccccccc-dddd-4444-bbbb-dddddddddddd/certificate/7c37a63465ce0a97f5195f59cb38cf90","certificaterequest":{"name":"istio-csr-b9rxv","namespace":"istio-system"}}
{"level":"error","ts":"2024-07-11T21:23:15Z","msg":"Reconciler error","controller":"certificaterequest","controllerGroup":"cert-manager.io","controllerKind":"CertificateRequest","CertificateRequest":{"name":"istio-csr-b9rxv","namespace":"istio-system"},"namespace":"istio-system","name":"istio-csr-b9rxv","reconcileID":"7e9df590-79fe-41a6-80b0-57b6b787b23b","error":"Operation cannot be fulfilled on certificaterequests.cert-manager.io \"istio-csr-b9rxv\": StorageError: invalid object, Code: 4, Key: /registry/cert-manager.io/certificaterequests/istio-system/istio-csr-b9rxv, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: e0a502c0-4c55-42f0-814d-a075edff07ec, UID in object meta: ","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:227"}
{"level":"info","ts":"2024-07-11T21:23:15Z","logger":"controllers.CertificateRequest","msg":"Created certificate with arn: arn:aws:acm-pca:eu-west-1:111111111111:certificate-authority/cccccccc-dddd-4444-bbbb-dddddddddddd/certificate/c4733808d820d80b1cd8241ad9c0f737","certificaterequest":{"name":"istio-csr-vg6dm","namespace":"istio-system"}}
...

Are we being rate limited by the AWS?

Steps to reproduce

No response

Relevant log output

No response

Version

v1.2.7

Have you tried the following?

Category

Supported Workflow Broken

Severity

Severity 1

bmsiegel commented 3 months ago

Looks like you're running into RequestInProgressException. This happens when the issue cert and get cert calls are too close to each other. If there's a way you can slow down your issuance on your end that'd be a good mitigation, I will prioritize handling this exception in the source code as well.

FabioAntunes commented 3 months ago

Thanks for the quick reply! Would this always be an issue? We are using this along with istio-csr if we have a surge on pod creation would we always be in this position? If so this partnership between istio-csr and aws-privateca might not be feasible for us. We maybe should use this project for our ingress certificates and then use regular cert-manager issuer for istio.

On Thu, 11 Jul 2024 at 23:19, Brady Siegel @.***> wrote:

Looks like you're running into RequestInProgressException. This happens when the issue cert and get cert calls are too close to each other. If there's a way you can slow down your issuance on your end that'd be a good mitigation, I will prioritize handling this exception in the source code as well.

— Reply to this email directly, view it on GitHub https://github.com/cert-manager/aws-privateca-issuer/issues/329#issuecomment-2224045750, or unsubscribe https://github.com/notifications/unsubscribe-auth/AATNIIM2XXCB5LO6QKOUGM3ZL4AF5AVCNFSM6AAAAABKXYRNTSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMRUGA2DKNZVGA . You are receiving this because you authored the thread.Message ID: @.***>

bmsiegel commented 3 months ago

So this will be an issue until we fix it. We are always open to a contribution from the community as well!

FabioAntunes commented 3 months ago

What I meant was: if the problem was on the aws side with the rate limiting and therefore it's an upstream issue, or is this on the aws-privateca-issuer?

anbaig commented 3 months ago

It's a client exception that the aws-privateca-issuer should just retry on when trying to drive a certificate to a completed state.

bmsiegel commented 3 months ago

This is to say we can support your use case when we prioritize the fix here.

anbaig commented 3 months ago

Per @bmsiegel -- The fix would be here: https://github.com/cert-manager/aws-privateca-issuer/blob/main/pkg/aws/pca.go#L80-L87

Probably want to just update the client config to retry if RequestInProgress Exception is encountered

bmsiegel commented 3 months ago

The fix for this is included in the release: https://github.com/cert-manager/aws-privateca-issuer/releases/tag/v1.3.0. Please reopen if you're still seeing this issue.