Open cueckoo opened 3 years ago
Original reply by @myitcv in https://github.com/cuelang/cue/issues/709#issuecomment-771712539
Quoting myself from https://github.com/cuelang/cue/discussions/640#discussioncomment-331122:
I've just hit a similar use case where I need to effectively hold a mutex whilst running a specific task (that writes to a part of a file), a number of instances of which are created, much like above, using a comprehension based on a
file.Glob
.How about a solution that uses the equivalent of a length
n
buffered channel? Not sure what that would look like in terms of a declaration (using atool/sync
builtin of sorts?), but a task would effectively be able to depend on such a limiter.Initialising the value of such a limiter (via a builtin) would take a positive integer value. If we were to support the injection of special values like
os
, we might extend that to include the CUE equivalent of Go'sruntime.NumCPU()
.
Original reply by @myitcv in https://github.com/cuelang/cue/issues/709#issuecomment-772808008
Noting a discussion from this afternoon regarding this:
cue cmd
/cuerun
tools/flow
(we could create and use an internal API to start with)command: x: exec.Run & {
$semaphore: s1: 3
cmd: "........"
}
where a $semaphore
field on a command is essentially a [string]: int
type, allowing a command to depend on multiple semaphores
Originally opened by @myitcv in https://github.com/cuelang/cue/issues/709
cue version:
cue version 0.3.0-beta.1 darwin/amd64
I using a custom cue command to start a large number (100s) of independent tasks using exec.Run. The number of process tasks is too large to run them all in parallel, so I am trying to find a way to limit how many tasks are run at a time.
My current attempt involves grouping tasks, and making each group depend on the previous group completing. However, adding the
$after
dependencies makes the cue command very slow.Here's an simplfied example of what I am trying:
test_tool.cue
This should start four groups of tasks serially. The tasks in a group run in parallel. I would expect the tasks to take about four seconds, but it takes over a minute on my computer, and causes high CPU.
time cue test
shows:If I comment out the
$after: task["\( i-1 )"]
,time cue test
shows:Is there a reason adding these dependencies is so expensive? Is there a better way to control how many tasks run at once?
Originally posted by @svend in https://github.com/cuelang/cue/discussions/640