search

buildkite / agent-stack-k8s

Spin up an autoscaling stack of Buildkite Agents on Kubernetes
MIT License
77 stars 27 forks source link

Cache plugins not working #308

Open calvinbui opened 3 months ago

calvinbui commented 3 months ago

I am unable to get either the buildkite-plugins/cache-buildkite-plugin or nienbo/cache-buildkite-plugin working.

I have tried placing the cache before and after the kubernetes plugin.

with nienbo/cache, the pre-command hook fails:

2024-04-18 09:28:34 AEST Running plugin cache pre-command hook
2024-04-18 09:28:34 AEST $ /workspace/plugins/buildkite/github-com-nienbo-cache-buildkite-plugin-v2-4-15/hooks/pre-command
2024-04-18 09:28:36 AEST 🚨 Error: The plugin cache pre-command hook failed to run because it was already open. We couldn't find out what process had the hook open

with buildkite-plugins/cache-buildkite-plugin, the post-command hook fails:

2024-04-18 09:50:20 AEST Running plugin cache post-command hook
2024-04-18 09:50:20 AEST $ /workspace/plugins/buildkite/github-com-buildkite-plugins-cache-buildkite-plugin-v1-0-0/hooks/post-command
2024-04-18 09:50:22 AEST 🚨 Error: The plugin cache post-command hook failed to run because it was already open. We couldn't find out what process had the hook open
2024-04-18 09:50:22 AEST 🚨 Error: error running "/tmp/buildkite-agent-hook-wrapper/hook-script-wrapper-2475512101": error starting pty: fork/exec /tmp/buildkite-agent-hook-wrapper/hook-script-wrapper-2475512101: no such file or directory

I am open to other viable solutions as well. We are on EKS, so EFS/FSX volumes would work (ReadWriteMany), but they are expensive to be fast enough and are complicated to work with.

DrJosh9000 commented 3 months ago

Hi @calvinbui, thanks for raising the issue. The plugins appear to rely on Bash. Are you using a container image that has Bash installed (at /bin/bash)?

While the stack itself, and the agent, can run in Bashless images like Alpine, most plugins are written assuming Bash is present (since that was a requirement for most of the agent's history).

This should probably be called out in the docs - I suspect it's not clear enough already!

Also there is currently a bug with the agent such that the error message is a bit cryptic when the interpreter specified for a hook isn't present (https://github.com/buildkite/agent/issues/2734).