search

observatorium / thanos-receive-controller

Kubernetes controller to automatically configure Thanos receive hashrings
Apache License 2.0
93 stars 42 forks source link

main.go: enable cleanup of old PVCs #38

Open squat opened 4 years ago

squat commented 4 years ago

This commit enables the controller to clean up the PVCs of Thanos Receive Pods that are watched by it. Whenever a receiver is deleted, the controller will spawn a helper container that mounts all PVCs specified by the StatefulSet for that container, replicate the contents to object storage, and then rm -rf the contents.

Tested on a kind cluster. A follow up PR will add E2E tests and verify this functionality.

squat commented 4 years ago

This PR is currently blocked on https://github.com/observatorium/thanos-replicate/blob/eb8dd2444ac8a24b1cf2d458055d1e1c9f727591/scheme.go#L197.

Because there are no labels on the block before the receive shipper uploads them, thanos-replicate skips the block when running in the cleanup job.

Will make a PR to move this forward

brancz commented 4 years ago

I also realized this won’t work anymore with multitsdb, we way want to implement that right away as I think the strategy might be a bit different.

squat commented 4 years ago

I also realized this won’t work anymore with multitsdb, we way want to implement that right away as I think the strategy might be a bit different.

Yeah, that sounds like a plan

tekicode commented 9 months ago

This should be closed. Kubernetes now supports discarding PVCs on StatefulSet scale down Here's the 2021 blog on this feature as alpha in Kubernetes 1.23.

In 2023, most of us running Kuberentes should have access to this functionality. https://kubernetes.io/blog/2021/12/16/kubernetes-1-23-statefulset-pvc-auto-deletion/