Deleting Object successful but it does not remove respective objects

[ilteristabak]

I have a custom resource whose count in all namespaces is ~770. I ran janitor in debug mode and it prints

• X xxx-20191127-122231162 with 2m TTL is 6d19h16m42s old and will be deleted (rule require-completed-or-failed-on-not-default matches)
• https://172.20.0.1:443 "POST /api/v1/namespaces/default/events HTTP/1.1" 201 942
• Deleting X default/xxx-20191127-122231162..
• https://172.20.0.1:443 "DELETE /apis/xxx.xxx.dev/v1/namespaces/default/X/xxx-20191127-122231162 HTTP/1.1" 200 None

But it does not delete that object. I exported the object to my docker-for-desktop, it prints same output but this time it is deleted

It is not about permissions because I modified clusterrole and this time output is:

• Deleting X default/xxx-20191127-122231162..
• https://172.20.0.1:443 "DELETE /apis/xxx.xxx.dev/v1/namespaces/default/X/xxx-20191127-122231162 HTTP/1.1" 403 475
• Could not delete X default/xxx-20191127-122231162: X.xxx.xxx.dev "xxx-20191127-122231162" is forbidden: User "system:serviceaccount:non-default:kube-janitor" cannot delete resource "X" in API group "xxx.xxx.dev" in the namespace "default"

Here it is the last line of main cluster: Clean up run completed: resources-processed=3762, X-with-ttl=17, X-deleted=17, rule-require-completed-or-failed-on-not-default-matches=16

[hjacobs]

We observe the same problem with CRDs, kube-janitor reports success, but the object is stuck in deletion:

kubectl get pcs myobject -o yaml
apiVersion: zalando.org/v1
kind: PlatformCredentialsSet
metadata:
  annotations:
    deployment-time: "2019-09-03T08:20:02Z"
    janitor/ttl: 1w

  creationTimestamp: "2019-04-28T17:39:19Z"
  deletionGracePeriodSeconds: 0
  deletionTimestamp: "2019-07-25T09:32:28Z"
  finalizers:
  - foregroundDeletion
  generation: 4
  name: myobject
  namespace: default
  resourceVersion: "792468607"
  uid: 8d539110-69dc-11e9-8076-0aecc1ce036a
spec:
  application: myapp
  tokens:
    myapp:
      privileges: []
status:
  observedGeneration: 4
  problems: []
  processingStatus: Succeeded

[hjacobs]

kubectl uses the "Background" propagation policy:

I0131 10:59:05.824361   12718 request.go:947] Request Body: {"propagationPolicy":"Background"}
I0131 10:59:05.824566   12718 round_trippers.go:419] curl -k -v -XDELETE  -H "Accept: application/json" -H "Content-Type: application/json" -H "User-Agent: zkubectl-v1.15.6.103/v1.15.6 (linux/amd64) kubernetes/7015f71" -H "Authorization: Bearer XXX" 'https://kube/apis/zalando.org/v1/namespaces/default/platformcredentialssets/myobject'
I0131 10:59:05.872156   12718 round_trippers.go:438] DELETE https://kube/apis/zalando.org/v1/namespaces/default/platformcredentialssets/myobject 200 OK in 47 milliseconds