alitto / pond

🔘 Minimalistic and High-performance goroutine worker pool written in Go
MIT License
1.45k stars 64 forks source link

How to pass variable value to pool.Submit func #1

Closed brandon099 closed 4 years ago

brandon099 commented 4 years ago

This is a question rather than a bug, and should preface this with saying I am new to Go. Also, should add: this library has been awesome to quickly add worker pools to my code!

I am iterating over a map and I think what I need is to be able to pass the current value of the key for the current iteration. Because as soon as I submit the func to the pool (pool.Submit(func(){})), since a map doesn't guarantee order I'm running into issues.

Example code that prints out the same item several times, rather than going through each one.

pool := pond.New(10, 0)

for _, item := range element {
    fmt.Printf("%v",item)
    pool.Submit(func() {
        // Do something with **item**, but it's no longer the original value,
        // so I'd like to pass it in as part of the loop, to this anonymous function to use
        fmt.Printf("%v",item)
    })
pool.StopAndWait()

This process can be done Async (it takes item and uses it as part of a web request), it's just it needs to match the current iteration instance of item, when it does not.

alitto commented 4 years ago

Hey Brandon! thanks man, I'm glad this library is useful for you :slightly_smiling_face:

That's right, that happens because Go reuses the same variable when iterating through an array or slice. In your example, the item variable is reused in every iteration, so its memory address never changes, only its value is updated. That's why all the goroutines (which very likely end up running after the iteration is completed) see the same value at the moment of printing it. There are several ways to prevent this, but here are 2 of the most common ones:

pool := pond.New(10, 0)

// Solution 1: creating a local copy of the iteration variable and using the copy inside the function
for _, item := range element {
    itemCopy := item
    pool.Submit(func() {
        fmt.Printf("%v", itemCopy)
    })
}

// Solution 2: creating an annonymous function that's immediately invoked with the iterator value
for _, item := range element {
    // Here you'll need to replace interface{} by the actual type of item
    pool.Submit(func(item interface{}) func() {
        return func() {
            fmt.Printf("%v", item)
        }
    }(item))
}

pool.StopAndWait()

The second solution doesn't look very nice tbh, but it should work :slightly_smiling_face: Here's an article that explains this situation with greater detail: https://github.com/golang/go/wiki/CommonMistakes#using-goroutines-on-loop-iterator-variables

Have a nice day! and please let me know if that worked for you.

brandon099 commented 4 years ago

Wow, thank you so much -- that worked perfectly. Have a nice day as well!