sethvargo / ratchet

A tool for securing CI/CD workflows with version pinning.
Apache License 2.0
772 stars 32 forks source link

Parallelize reference resolution #3

Closed sethvargo closed 2 years ago

sethvargo commented 2 years ago

While the parser de-duplicates any references, the current resolution occurs in serial. This is fine for a small number of references, but it can take unnecessarily long for many. We should parallelize the reference lookup using the semaphore pattern.

Design

  1. Add new flag -concurrency to update and pin commands. Default to max(numCPUs()-1, 1), plumb through.

  2. Create a semaphore of concurrency and background the go routines:

    for _, ref := range references {
      if err :- sem.Acquire(1); err != nil {
        return err
      }
    
      go func() {
        if err := ref.Resolve(); err != nil {
          // handle error
        }
      }()
    }
    
    if err := sem.Acquire(max); err != nil {
      return err
    }
    
    // handle goroutine errors