circuit may open even though ErrorPercentThreshold is bigger than 100

[Heisenberg-Y]

Bug description

In theory, when ErrorPercentThreshold is bigger than 100, the circuit should be always closed. But there are some exception. If you run the code below, you will find out that the circuit may possibly be open.

package main

import (
    "fmt"
    "github.com/afex/hystrix-go/hystrix"
    "log"
    "math/rand"
    "sync"
    "time"
)

func main() {
    f := time.Millisecond * 200
    cun := 1000
    num := cun * 3
    hystrix.SetLogger(log.Default())
    hystrix.ConfigureCommand("my_command", hystrix.CommandConfig{
        Timeout:               int(f.Milliseconds()),
        MaxConcurrentRequests: cun,
        RequestVolumeThreshold: 30,
        SleepWindow: 30,
        ErrorPercentThreshold: 120,
    })

    var lock sync.Mutex
    var wg sync.WaitGroup
    rand.Seed(int64(time.Now().Nanosecond()))
    mm := make(map[string]int)
    mark := 0
    now := time.Now()
    for i := 0; i < num; i++ {
        wg.Add(1)
        go func() {
            defer wg.Done()
            err := hystrix.Do("my_command", func() error {
                a := rand.Intn(5)
                if a < 1 {
                    time.Sleep(time.Millisecond * 10)
                    return fmt.Errorf("internal error")
                } else if a >= 4 {
                    time.Sleep(time.Millisecond * 300)
                    return nil
                } else {
                    time.Sleep(time.Millisecond * 9)
                    lock.Lock()
                    if _, found := mm["success"]; found {
                        mm["success"]++
                    } else {
                        mm["success"] = 1
                    }
                    lock.Unlock()
                    return nil
                }
            }, func(err error) error {
                if err != nil {
                    lock.Lock()
                    if err.Error() == "hystrix: circuit open" && mark == 0 {
                        log.Println("circuit on at: ", time.Now().Sub(now).Milliseconds())
                        mark = 1
                    }
                    if _, found := mm[err.Error()]; found {
                        mm[err.Error()]++
                    } else {
                        mm[err.Error()] = 1
                    }
                    lock.Unlock()
                } else {
                    panic("err is nil")
                }

                return nil
            })

            if err != nil {
                fmt.Println("err: ", err)
            }
        }()
    }
    wg.Wait()
    log.Println("end at: ", time.Now().Sub(now).Milliseconds())

    count := 0
    for key, value := range mm {
        count += value
        fmt.Printf("%s: %f\n", key, float32(value) / float32(num))
    }
    if count != num {
        panic("don't match")
    }
}

In https://github.com/afex/hystrix-go/blob/fa1af6a1f4f56e0e50d427fe901cd604d8c6fb8a/hystrix/metrics.go#L138, the value of errs may be larger than that of reqs because the code calculate the errs later than reqs.

Resolution

One simple way to fix the problem is to make the circuit always heathy when ErrorPercentThreshold is bigger than 100 like:

func (m *metricExchange) IsHealthy(now time.Time) bool {
        errRate := getSettings(m.Name).ErrorPercentThreshold
    if errRate > 100 {
                return true
        }
        return m.ErrorPercent(now) < errRate
}