We are seeing a pattern which indicates a likely goroutine leak. When there are downstream errors and circuit is opened for any given command, then after a while lot of goroutines get created and never closed ( pprof output attached ). A quick glance at the code suggests that they are created for every command's CircuitBreaker. But we ran few test where only single command is used, still the goroutine count created by metricExchange.Monitor and poolMetrics.Monitor seemed to be increasing according to the pprof output.
Test environment:
We ran around 100 RPM, no failure ( i.e. circuit was never opened ), the expectation was that it will have constant number of go-routines created by hystrix-go lib metricExchange.Monitor and poolMetrics.Monitor. But as it can be seen in the output, the number increased.
We are seeing a pattern which indicates a likely goroutine leak. When there are downstream errors and circuit is opened for any given command, then after a while lot of goroutines get created and never closed ( pprof output attached ). A quick glance at the code suggests that they are created for every command's
CircuitBreaker. But we ran few test where only single command is used, still the goroutine count created bymetricExchange.MonitorandpoolMetrics.Monitorseemed to be increasing according to the pprof output.Test environment:
We ran around 100 RPM, no failure ( i.e. circuit was never opened ), the expectation was that it will have constant number of go-routines created by
hystrix-golibmetricExchange.MonitorandpoolMetrics.Monitor. But as it can be seen in the output, the number increased.pprof output before running the test:
pprof output after running the test: