search

mozilla-services / heka

DEPRECATED: Data collection and processing made easy.
http://hekad.readthedocs.org/
Other
3.39k stars 531 forks source link

High CPU usage on SSL TCPOutput #1890

Open cristi- opened 8 years ago

cristi- commented 8 years ago

Hi, We have a setup with heka clients connection to a central heka master. The heka master is configured with SSL. Clients send logs to the master using TcpOutput with use_tls = true. The problem is that when the master is down, heka starts consuming 100% of one CPU on the clients. I suspect this is due to SSL retry.\ Disabling SSL **on the agents (with the master down) stops heka from fully using a CPU.

Here is a cpuprof from a client when the master was down:

Dropped 65 nodes (cum <= 0.70s) flat flat% sum% cum cum% 36.97s 26.44% 26.44% 36.97s 26.44% math/big.mulAddVWW 28.38s 20.30% 46.74% 28.38s 20.30% math/big.addMulVVW 26.36s 18.85% 65.60% 98.56s 70.50% math/big.nat.divLarge 20.94s 14.98% 80.57% 20.94s 14.98% math/big.subVV 6.78s 4.85% 85.42% 36.82s 26.34% math/big.basicMul 3.41s 2.44% 87.86% 3.41s 2.44% math/big.mulWW 3.36s 2.40% 90.27% 3.36s 2.40% math/big.nat.clear 2.89s 2.07% 92.33% 2.89s 2.07% math/big.shrVU 1.49s 1.07% 93.40% 1.49s 1.07% math/big.shlVU 1.38s 0.99% 94.39% 1.38s 0.99% math/big.nat.norm 1.23s 0.88% 95.27% 1.23s 0.88% math/big.divWW 0.93s 0.67% 95.93% 0.93s 0.67% runtime.memclr 0.81s 0.58% 96.51% 1.75s 1.25% runtime.makeslice 0.44s 0.31% 96.82% 0.94s 0.67% runtime.newarray 0.08s 0.057% 96.88% 98.66s 70.57% math/big.nat.div 0.05s 0.036% 96.92% 36.87s 26.37% math/big.nat.mul 0.03s 0.021% 96.94% 135.55s 96.95% math/big.nat.expNNWindowed 0 0% 96.94% 4.23s 3.03% System 0 0% 96.94% 135.55s 96.95% crypto/rsa.(_PrivateKey).Validate 0 0% 96.94% 135.56s 96.96% crypto/tls.LoadX509KeyPair 0 0% 96.94% 135.56s 96.96% crypto/tls.X509KeyPair 0 0% 96.94% 135.56s 96.96% crypto/tls.parsePrivateKey 0 0% 96.94% 135.56s 96.96% crypto/x509.ParsePKCS1PrivateKey 0 0% 96.94% 135.56s 96.96% github.com/mozilla-services/heka/pipeline.(_BufferReader).NewStreamOutput 0 0% 96.94% 135.56s 96.96% github.com/mozilla-services/heka/pipeline.(_foRunner).Starter 0 0% 96.94% 135.56s 96.96% github.com/mozilla-services/heka/pipeline.(_foRunner).bufferLoop 0 0% 96.94% 135.56s 96.96% github.com/mozilla-services/heka/plugins/tcp.(_TcpOutput).ProcessMessage 0 0% 96.94% 135.56s 96.96% github.com/mozilla-services/heka/plugins/tcp.(_TcpOutput).connect 0 0% 96.94% 135.56s 96.96% github.com/mozilla-services/heka/plugins/tcp.CreateGoTlsConfig 0 0% 96.94% 135.55s 96.95% math/big.(*Int).ProbablyPrime 0 0% 96.94% 135.55s 96.95% math/big.nat.expNN 0 0% 96.94% 135.55s 96.95% math/big.nat.probablyPrime 0 0% 96.94% 135.56s 96.96% runtime.goexit

Would building heka with go 1.6 help ? I think there are some improvements to the SSL connection in the newer version.