search

buildkite / agent

The Buildkite Agent is an open-source toolkit written in Go for securely running build jobs on any device or network
https://buildkite.com/
MIT License
812 stars 300 forks source link

Error: "argument list too long" (because of large BULIDKITE_PLUGINS env var) #2019

Open andpol opened 1 year ago

andpol commented 1 year ago

Is your feature request related to a problem? Please describe. We use this monorepo plugin which seems to pass it's configurations via encoded JSON through the BUILDKITE_PLUGINS environment variable. We recently found the limits of that, and have the following error when running the step:

fork/exec /buildkite-agent: argument list too long

This seems closely related to the existing (resolved) issues regarding long Git commit messages and the BUILDKITE_MESSAGE env var: https://github.com/buildkite/agent/pull/1302 and https://github.com/buildkite/agent/pull/1307.

Describe the solution you'd like It would be nice if the BUILDKITE_PLUGINS env var could accept longer strings (and maybe generalized to other env vars?) Not being super familiar to Buildkite and the source code, I'm not sure if this is actually possible, but hoping someone here can advise if this is one potential solution. I would argue this is not an unreasonable feature to add, since it seems not unreasonable that plugins could have more extensive configurations.

Describe alternatives you've considered As a short-term workaround, we've split up the Buildkite Steps that run the Monorepo plugin. This works for now, but scales based on the number of applications/files we've configured with the plugin, and will likely break for us in the future once again.

Another alternative is changing the Monorepo plugin, but I'm unfamiliar with how Buildkite plugins work, so maybe there isn't a great path forwards from that perspective?

Additional context As an attempted fix, we increased the stack size limit (ulimit -s) on our Buildkite agents, with no effect.

pda commented 1 year ago

Interesting!

I've done some digging in the past re. environment variable limits on different operating systems, and recently tested/updated it for macOS. Unfortunately the citations are lost to the sands of time. Here's the notes I made at the time:

macOS: 1 MiB (previously 256 KiB?) shared across all env and argv Linux: 128 KiB per env key=value, (2 MiB total? depends on… things) Windows: 16 MiB shared by all env (And argv? Does Windows have argv?) POSIX bare minimum: 4 KiB shared by all env and argv

andpol commented 1 year ago

We're operating our Agents on Linux (in a K8s cluster). The BUILDKITE_PLUGINS environment variable for the build in question is 168KiB in size, so hitting the error makes sense in the context of the limits you've found.

Do you think there is a reasonable way to pass along the plugin data to the agent in a way that doesn't use environment variables?