search

carvel-dev / kapp

kapp is a simple deployment tool focused on the concept of "Kubernetes application" — a set of resources with the same label
https://carvel.dev/kapp
Apache License 2.0
904 stars 108 forks source link

orphan delete-strategy for child resources not created by kapp #215

Open lnhrdt opened 3 years ago

lnhrdt commented 3 years ago

I have an app that I deploy and teardown all the time as part of an automated test process. Part of my app includes certificates generated by cert-manager and provisioned by Let's Encrypt.

Up until now, I excluded the certificate resources from the automated test environment because the act of creating and deleting the certificate resources leads to exceeding Let's Encrypt's rate limits, which causes my deployments to stop working.

I noticed kapp features a kapp.k14s.io/delete-strategy annotation that I might be able to use on the certificate to "skip deleting" it and reuse it on the next deployment. I thought this would be better for me because having the certificates in my test environment just like every other environment would make my test environment more production-like and simplify my test process by letting it assume there will always be valid certificates in any environment.

I ran into issues with the generated, "child resources" of the certificate.

What steps did you take:

I applied the kapp.k14s.io/delete-strategy: orphan annotation to my certificate.cert-manager.io resource. I deployed the app. I deleted the app.

What happened:

When the app deployed, cert-manager created the accompanying certificaterequest.cert-manager.io and order.acme.cert-manager.io resources (normal, desired behavior of cert-manager). A kubectl describe command for each of the generated resources shows that they have inherited the kapp.k14s.io/delete-strategy: orphan annotation. I don't know what mechanism applied that annotation to them (maybe cert-manger or k8s itself?), but that's exactly what I wanted so I'm happy about it. The three resources work as a unit so I wanted to treat all of the same way (i.e. leave them orphaned on delete).

Name:         my-app-qcmbr
Namespace:    my-app
Labels:       kapp.k14s.io/app=1619283995411166000
              kapp.k14s.io/association=v1.8dd04f03c81055b2dff27e72ffe84719
Annotations:  cert-manager.io/certificate-name: my-app
              cert-manager.io/certificate-revision: 1
              cert-manager.io/private-key-secret-name: my-app-tjh4z
              kapp.k14s.io/delete-strategy: orphan 👈 😁
              kapp.k14s.io/identity: v1;my-app/cert-manager.io/Certificate/my-app;cert-manager.io/v1
              kapp.k14s.io/original:
                {"apiVersion":"cert-manager.io/v1","kind":"Certificate","metadata":{"annotations":{"kapp.k14s.io/delete-strategy":"orphan"},"labels":{"kap...
              kapp.k14s.io/original-diff-md5: 58e0494c51d30eb3494f7c9198986bb9
API Version:  cert-manager.io/v1
Kind:         CertificateRequest

However when I deleted the app, kapp's diff didn't show the orphan strategy for the generated child resources.

Namespace  Name                     Kind                Conds.  Age  Op      Op st.  Wait to  Rs  Ri
(cluster)  my-app                   Namespace           -       9m   delete  orphan  delete   ok  -
my-app     api                      Deployment          2/2 t   19s  delete  -       delete   ok  -

... (abbreviated)

^          my-app                   Certificate         1/1 t   9m   delete  orphan  delete   ok  -
^          my-app-qcmbr             CertificateRequest  1/1 t   9m   -       -       delete   ok  - 👈 🤔
^          my-app-qcmbr-1942737023  Order               -       9m   -       -       delete   ok  - 👈 🤔

Oh no! This diff from kapp isn't representing kapp.k14s.io/delete-strategy: orphan annotations on CertificateRequest and Order resources! I went ahead and deleted the app anyway to see what would happen.

$ kapp -n apps delete -a my-app
... (abbreviated)
10:17:51AM: ---- waiting complete [28/28 done] ----

kapp: Error: Expected all resources to be gone, but found: order/my-app-qcmbr-1942737023 (acme.cert-manager.io/v1) namespace: my-app, certificaterequest/my-app-qcmbr (cert-manager.io/v1) namespace: my-app
$ echo $?
1

So, it seems it didn't delete the resources (yay) but printed a sad error message and exited unsuccessfully (boo).

What did you expect:

Anything else you would like to add:

The Carvel team rocks.

Environment:

Vote on this request

This is an invitation to the community to vote on issues, to help us prioritize our backlog. Use the "smiley face" up to the right of this comment to vote.

👍 "I would like to see this addressed as soon as possible" 👎 "There are other more important things to focus on right now"

We are also happy to receive and review Pull Requests if you want to help working on this issue.

cppforlife commented 3 years ago

hey @lnhrdt,

thanks for a detailed question. here is whats happening: kapp knows about resources it creates vs resources that get created by controllers/cluster as a response to kapp created resources.

if you take a look at the delete summary, there is a column called "Op" that indicates what operation kapp will take against that resource. in your case, 2 resources that were created by cert-manager are not owned by kapp directly, hence by default kapp will not explicitly delete them (though it will try to wait for them to be deleted -- wait to column shows that).

one possible solution here is to try adding owned-for-deletion annotation (https://carvel.dev/kapp/docs/latest/apply/#kappk14sioowned-for-deletion). im guessing that cert-manager copies down annotations from certificate resource to child resources so it should copy over this annotation as well. this annotation would tell kapp to consider these resources as owned by kapp. by combining it with delete-strategy annotation to orphan resources (instead of actually deleting it), you ll get desired combination.

ive not tried doing this myself but do let us k ow how this pans out.

lnhrdt commented 3 years ago

Hey @cppforlife

Your suggestion worked great! I applied the kapp.k14s.io/owned-for-deletion: "" annotation to my certificates. Something (presumably cert-manager) copied them over to the generated CertificateRequest and Order resources. When I used kapp to delete the app, it identified all 3 resources as having an orphan strategy and successfully deleted the app while leaving those 3 behind. Thanks for helping me find a solution to what I needed!

I don't know if it's too much of an edge case to care about, but I feel it might be worth pointing out what was kind of unexpected behavior along the way. When the resources generated by cert-manager (CertificateRequest and Order) had the Certificate resource's annotations:

kapp.k14s.io/delete-strategy: orphan
kapp.k14s.io/identity: v1;my-app/cert-manager.io/Certificate/my-app;cert-manager.io/v1
kapp.k14s.io/original:
  {"apiVersion":"cert-manager.io/v1","kind":"Certificate","metadata":{"annotations":{"kapp.k14s.io/delete-strategy":"orphan"},"labels":{"kap...
kapp.k14s.io/original-diff-md5: 58e0494c51d30eb3494f7c9198986bb9

A few weird things happened when deleting the app:

Again, I don't really need resolution on that for my needs but thought it might be worth pointing out what I noticed along the way before closing this issue. And thanks again for the quick help @cppforlife!

cppforlife commented 3 years ago

they showed up in kapp's diff, despite not being owned by the app

since we marked them with "owned-for-deletion" they are owned by the app (for deletion at least) from kapp's pov. if you have a diff snippet (with summary) it might be useful to see how it looks.

their presence failed the delete operation

can you tell me a bit more about this one? any errors that popped up?