search

cue-lang / cue

The home of the CUE language! Validate and define text-based and dynamic configuration
https://cuelang.org
Apache License 2.0
5.14k stars 294 forks source link

cuelang.org: preprocessor: resource use is unconstrained #3014

Open jpluscplusm opened 7 months ago

jpluscplusm commented 7 months ago

What page were you looking at?

https://cuelang.org/

What version of the site were you looking at?

https://github.com/cue-lang/cuelang.org/commit/0f80600d1d2083d1e94276c13b532b71171ae12f

What did you do?

I bumped the version of CUE used by the site, and regenerated all the site's cache files.

What did you expect?

That my machine would remain usable.

What did you see instead?

$ ps axu | grep -c podman; uptime
743
 13:21:35 up 4 days,  6:19,  5 users,  load average: 254.19, 174.65, 82.23

Please could the preprocessor implement /some/ kind of load-based backoff strategy or max-in-flight mechanism?

FWIW I quite like the filesystem-based simplicity of https://github.com/leahneukirchen/nq, but I'm sure there are native Go solutions in the same area.

jpluscplusm commented 2 months ago

As a data-point, I implemented a max-containers-in-flight strategy using the task-spooler utility.

This reduced the wall clock time for a full regeneration of the site's cache files by 66%. e.g. 1 minute instead of 3 minutes (although these aren't the actual times).

For posterity, this was achieved simply by installing task-spooler, and then inserting a shim between the preprocessor's docker invocations and the docker binary, with task-spooler handling the queue size via its TS_SLOTS envvar: