While the parser de-duplicates any references, the current resolution occurs in serial. This is fine for a small number of references, but it can take unnecessarily long for many. We should parallelize the reference lookup using the semaphore pattern.
Design
Add new flag -concurrency to update and pin commands. Default to max(numCPUs()-1, 1), plumb through.
Create a semaphore of concurrency and background the go routines:
for _, ref := range references {
if err :- sem.Acquire(1); err != nil {
return err
}
go func() {
if err := ref.Resolve(); err != nil {
// handle error
}
}()
}
if err := sem.Acquire(max); err != nil {
return err
}
// handle goroutine errors
While the parser de-duplicates any references, the current resolution occurs in serial. This is fine for a small number of references, but it can take unnecessarily long for many. We should parallelize the reference lookup using the semaphore pattern.
Design
Add new flag
-concurrency
toupdate
andpin
commands. Default tomax(numCPUs()-1, 1)
, plumb through.Create a semaphore of
concurrency
and background the go routines: