search

garden-io / garden

Automation for Kubernetes development and testing. Spin up production-like environments for development, testing, and CI on demand. Use the same configuration and workflows at every step of the process. Speed up your builds and test runs via shared result caching
https://garden.io
Mozilla Public License 2.0
3.36k stars 272 forks source link

[FEATURE]: timeouts for cleanup commands #6076

Open salotz opened 4 months ago

salotz commented 4 months ago

Feature Request

Background / Motivation

I need to run cleanup namespace in a workflow and I'm hitting the hard-coded timeouts for this as it can take a while. When this runs in CI this becomes a problem as it then causes CI to fail as the task exits as failed.

What should the user be able to do?

Specify a timeout when running cleanup ... commands with garden.

Suggested Implementation(s)

garden cleanup --timeout=1000 [...]

How important is this feature for you/your team?

Somewhere between:

🥀 Crucial, Garden is unusable for us without it

🌵 Not having this feature makes using Garden painful

As I don't really know a workaround.

eysi09 commented 4 months ago

Hi @salotz

Thanks for flagging this. Your suggestion makes sense and we'll add support for this.

eysi09 commented 4 months ago

Actually, it might make more sense to just bump the default timeout.

Do you have a rough feeling for what a sensible value could be for your use case?

salotz commented 4 months ago

@eysi09 Usually increasing timeouts to 15min does it for me, but I don't have a good intuition about which operations actually take so long. Especially when cleaning up namespaces I find sometimes k8s gets stuck.

eysi09 commented 4 months ago

Ok gotcha. The current default timeout is 300 seconds so that would be quite a large bump.

In that case I think the quickest win would be to set this is an environment variable. The cleanup command can technically delete things beyond K8s namespaces so the --timeout flag wouldn't really be well defined and you might need different values for different "kinds" of namespaces in the future.

Do you think the env var approach would work for your use case?

Something like:

GARDEN_K8S_DELETE_NS_TIMEOUT_MS=900 garden cleanup namespace

(A little verbose but the quickest win AFIACT since this blocking you)

salotz commented 4 months ago

That should work yes.

salotz commented 4 months ago

I do have a workaround implemented in CI to just use kubectl for deleting the namespace which I can set timeouts for btw.