moleculer-go / moleculer

🚀 Progressive microservices framework for Go - based and compatible with https://github.com/moleculerjs/moleculer
MIT License
144 stars 27 forks source link

Broker hang if multiple goroutines make calls #88

Closed casskir closed 3 years ago

casskir commented 4 years ago

Describe the bug Sometimes when we make calls from multiple goroutines broker is hanged. Also, a timeout error is not fired.

To Reproduce Make calls from multiple goroutines. Often reproduced on a slow machine.

broker := broker.New(config)

for t := 0; t < 10000; t++ {
        wg := sync.WaitGroup{}
        wg.Add(3)

        go func() {
            defer wg.Done()

            <-broker.Call(action, params)
            log.Info("first done")
        }()

        go func() {
            defer wg.Done()

            <-broker.Call(action, params)
            log.Info("second done")
        }()

        go func() {
            defer wg.Done()

            <-broker.Call(action, params)
            log.Info("third done")
        }()

        wg.Wait()
    }

Expected behavior All requests successfully finished or execution timeout error returned.

Additional context My investigation showed that multiple async calls generate requests with the same ID: image

And the main problem in: https://github.com/moleculer-go/moleculer/blob/b092546776be0c54a7466ba1161bc3fe1777b343/context/contextFactory.go#L51

This function using random generator that unsafe for multiple goroutines:

// NewSource returns a new pseudo-random Source seeded with the given value.
// Unlike the default Source used by top-level functions, this source is not
// safe for concurrent use by multiple goroutines.

Another way how reproduce:

t := make([]string, 1000)

    wg := sync.WaitGroup{}
    wg.Add(1000)

    for i := 0; i < 1000; i++ {
        go func(idx int) {
            defer wg.Done()
            t[idx] = util.RandomString(12)
        }(i)
    }

    wg.Wait()

    v := map[string]interface{}{}
    for i := 0; i < len(t); i++ {
        v[t[i]] = nil
    }

    fmt.Printf("Uniq: %d", len(v))

generates less than 1000 unique strings.