magefile / mage

a Make/rake-like dev tool using Go
https://magefile.org
Apache License 2.0
4.15k stars 257 forks source link

Add helper for parallelisation limits #38

Open kortschak opened 7 years ago

kortschak commented 7 years ago

The code is trivial, but it might be nice to add a helper to provide parallelisation limits.

natefinch commented 7 years ago

I'm curious what problem this solves? I try to keep the ux as simple as possible.

kortschak commented 7 years ago

It is common in make-like tools to be able to specify a parallelisation limit/level, e.g. make -j N. mage already allows concurrent processing purely by virtue of being Go, but say for example the user's rules involved processing thousands of files, the naive approach would suffer significantly unless there were a limit on the number of concurrent goroutines running.

The user can write

var n = flag.Int("j", 0, "specify the number of jobs to run simultaneously")

type limiter struct {
    token chan struct{}
    wg    sync.WaitGroup
}

var limit = limiter{token: make(chan struct{}, n) }

func acquire() {
    limit.wg.Add(1)
    limit.token <- struct{}{}
}

func release() {
    <-limit.token
    limit.wg.Done
}

for each set of mage files they want to use concurrent processing in which this is an issue, but then you can say that about any specific problem/solution.

natefinch commented 7 years ago

ahh, thanks for the more detailed explanation.