search

DataDog / datadog-agent

Main repository for Datadog Agent
https://docs.datadoghq.com/
Apache License 2.0
2.89k stars 1.21k forks source link

Race condition on datadog/agent:6.2.1-jmx #1812

Open igorvpcleao opened 6 years ago

igorvpcleao commented 6 years ago

Hi there,

It seems agent:6.2.1 has a race condition.

[...]
[ AGENT ] 2018-06-09 15:48:22 UTC | INFO | (check_sampler.go:50 in commit) | No value returned for check metric 'kubernetes.network.tx_bytes' on host '' and tags '[kube_deployment:instant-bot-2-processor kube_namespace:instant-bot-2 kube_replica_set:instant-bot-2-processor-69fdfcbc6 pod_name:instant-bot-2-processor-69fdfcbc6-skvgc]': Rate value is negative, discarding it (the underlying counter may have been reset)
[ AGENT ] fatal error: concurrent map read and map write
[ AGENT ]
[ AGENT ] goroutine 138 [running]:
[ AGENT ] runtime.throw(0x1f78805, 0x21)
[ AGENT ]   /root/.gimme/versions/go1.9.4.linux.amd64/src/runtime/panic.go:605 +0x95 fp=0xc426c15d70 sp=0xc426c15d50 pc=0x44bc05
[ AGENT ] runtime.mapaccess2_faststr(0x1c79c80, 0xc4203705d0, 0xc42940d4e0, 0x19, 0xd, 0xc423e899e0)
[ AGENT ]   /root/.gimme/versions/go1.9.4.linux.amd64/src/runtime/hashmap_fast.go:324 +0x47a fp=0xc426c15dc8 sp=0xc426c15d70 pc=0x42c0aa
[ AGENT ] github.com/DataDog/datadog-agent/pkg/collector/runner.shouldLog(0xc42940d4e0, 0x19, 0x0)
[ AGENT ]   /go/src/github.com/DataDog/datadog-agent/.omnibus/src/datadog-agent/src/github.com/DataDog/datadog-agent/pkg/collector/runner/runner.go:322 +0xd5 fp=0xc426c15e10 sp=0xc426c15dc8 pc=0xc0ca65
[ AGENT ] github.com/DataDog/datadog-agent/pkg/collector/runner.(*Runner).work(0xc422ed5a10)
[ AGENT ]   /go/src/github.com/DataDog/datadog-agent/.omnibus/src/datadog-agent/src/github.com/DataDog/datadog-agent/pkg/collector/runner/runner.go:243 +0x3bb fp=0xc426c15fd8 sp=0xc426c15e10 pc=0xc0bbeb
[ AGENT ] runtime.goexit()
[ AGENT ]   /root/.gimme/versions/go1.9.4.linux.amd64/src/runtime/asm_amd64.s:2337 +0x1 fp=0xc426c15fe0 sp=0xc426c15fd8 pc=0x47ce01
[ AGENT ] created by github.com/DataDog/datadog-agent/pkg/collector/runner.(*Runner).AddWorker
[ AGENT ]   /go/src/github.com/DataDog/datadog-agent/.omnibus/src/datadog-agent/src/github.com/DataDog/datadog-agent/pkg/collector/runner/runner.go:100 +0x8a
[ AGENT ]
[ AGENT ] goroutine 1 [chan receive, 2760 minutes]:
[ AGENT ] github.com/DataDog/datadog-agent/cmd/agent/app.start(0x2fe8ca0, 0x30f9e30, 0x0, 0x0, 0x0, 0x0)
[ AGENT ]   /go/src/github.com/DataDog/datadog-agent/.omnibus/src/datadog-agent/src/github.com/DataDog/datadog-agent/cmd/agent/app/start.go:116 +0x17c
[ AGENT ] github.com/DataDog/datadog-agent/vendor/github.com/spf13/cobra.(*Command).execute(0x2fe8ca0, 0x30f9e30, 0x0, 0x0, 0x2fe8ca0, 0x30f9e30)
[ AGENT ]   /go/src/github.com/DataDog/datadog-agent/.omnibus/src/datadog-agent/src/github.com/DataDog/datadog-agent/vendor/github.com/spf13/cobra/command.go:698 +0x47a
[ AGENT ] github.com/DataDog/datadog-agent/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0x2fe6880, 0xc4204e3f48, 0x1, 0x1)
[ AGENT ]   /go/src/github.com/DataDog/datadog-agent/.omnibus/src/datadog-agent/src/github.com/DataDog/datadog-agent/vendor/github.com/spf13/cobra/command.go:783 +0x30e
[ AGENT ] github.com/DataDog/datadog-agent/vendor/github.com/spf13/cobra.(*Command).Execute(0x2fe6880, 0xc4204e3f70, 0x191b139)
[ AGENT ]   /go/src/github.com/DataDog/datadog-agent/.omnibus/src/datadog-agent/src/github.com/DataDog/datadog-agent/vendor/github.com/spf13/cobra/command.go:736 +0x2b
[ AGENT ] main.main()
[ AGENT ]   /go/src/github.com/DataDog/datadog-agent/.omnibus/src/datadog-agent/src/github.com/DataDog/datadog-agent/cmd/agent/main.go:20 +0x2d
[ AGENT ]
[ AGENT ] goroutine 5 [semacquire, 2761 minutes]:
[ AGENT ] sync.runtime_notifyListWait(0xc42005c710, 0x0)
[ AGENT ]   /root/.gimme/versions/go1.9.4.linux.amd64/src/runtime/sema.go:507 +0x110
[ AGENT ] sync.(*Cond).Wait(0xc42005c700)
[ AGENT ]   /root/.gimme/versions/go1.9.4.linux.amd64/src/sync/cond.go:56 +0x80
[ AGENT ] github.com/DataDog/datadog-agent/vendor/github.com/cihub/seelog.(*asyncLoopLogger).processItem(0xc4200ce000, 0x0)
[ AGENT ]   /go/src/github.com/DataDog/datadog-agent/.omnibus/src/datadog-agent/src/github.com/DataDog/datadog-agent/vendor/github.com/cihub/seelog/behavior_asynclooplogger.go:50 +0x91
[ AGENT ] github.com/DataDog/datadog-agent/vendor/github.com/cihub/seelog.(*asyncLoopLogger).processQueue(0xc4200ce000)
[ AGENT ]   /go/src/github.com/DataDog/datadog-agent/.omnibus/src/datadog-agent/src/github.com/DataDog/datadog-agent/vendor/github.com/cihub/seelog/behavior_asynclooplogger.go:63 +0x44
[ AGENT ] created by github.com/DataDog/datadog-agent/vendor/github.com/cihub/seelog.NewAsyncLoopLogger
[ AGENT ]   /go/src/github.com/DataDog/datadog-agent/.omnibus/src/datadog-agent/src/github.com/DataDog/datadog-agent/vendor/github.com/cihub/seelog/behavior_asynclooplogger.go:40 +0x9f
[ AGENT ]
[ AGENT ] goroutine 6 [semacquire, 2761 minutes]:
[ AGENT ] sync.runtime_notifyListWait(0xc42005c890, 0x0)
[ AGENT ]   /root/.gimme/versions/go1.9.4.linux.amd64/src/runtime/sema.go:507 +0x110
[ AGENT ] sync.(*Cond).Wait(0xc42005c880)
[ AGENT ]   /root/.gimme/versions/go1.9.4.linux.amd64/src/sync/cond.go:56 +0x80
[ AGENT ] github.com/DataDog/datadog-agent/vendor/github.com/cihub/seelog.(*asyncLoopLogger).processItem(0xc4200ce120, 0x0)
[ AGENT ]   /go/src/github.com/DataDog/datadog-agent/.omnibus/src/datadog-agent/src/github.com/DataDog/datadog-agent/vendor/github.com/cihub/seelog/behavior_asynclooplogger.go:50 +0x91
[ AGENT ] github.com/DataDog/datadog-agent/vendor/github.com/cihub/seelog.(*asyncLoopLogger).processQueue(0xc4200ce120)
[ AGENT ]   /go/src/github.com/DataDog/datadog-agent/.omnibus/src/datadog-agent/src/github.com/DataDog/datadog-agent/vendor/github.com/cihub/seelog/behavior_asynclooplogger.go:63 +0x44
[ AGENT ] created by github.com/DataDog/datadog-agent/vendor/github.com/cihub/seelog.NewAsyncLoopLogger
[ AGENT ]   /go/src/github.com/DataDog/datadog-agent/.omnibus/src/datadog-agent/src/github.com/DataDog/datadog-agent/vendor/github.com/cihub/seelog/behavior_asynclooplogger.go:40 +0x9f
[ AGENT ]
[ AGENT ] goroutine 7 [select]:
[ AGENT ] github.com/DataDog/datadog-agent/vendor/github.com/patrickmn/go-cache.(*janitor).Run(0xc4202f4250, 0xc420297a80)
[ AGENT ]   /go/src/github.com/DataDog/datadog-agent/.omnibus/src/datadog-agent/src/github.com/DataDog/datadog-agent/vendor/github.com/patrickmn/go-cache/cache.go:1079 +0x11a
[ AGENT ] created by github.com/DataDog/datadog-agent/vendor/github.com/patrickmn/go-cache.runJanitor
[ AGENT ]   /go/src/github.com/DataDog/datadog-agent/.omnibus/src/datadog-agent/src/github.com/DataDog/datadog-agent/vendor/github.com/patrickmn/go-cache/cache.go:1099 +0xba
[ AGENT ]
[ AGENT ] goroutine 8 [chan receive]:
[ AGENT ] github.com/DataDog/datadog-agent/vendor/github.com/golang/glog.(*loggingT).flushDaemon(0x30b5260)
[ AGENT ]   /go/src/github.com/DataDog/datadog-agent/.omnibus/src/datadog-agent/src/github.com/DataDog/datadog-agent/vendor/github.com/golang/glog/glog.go:882 +0x9f
[ AGENT ] created by github.com/DataDog/datadog-agent/vendor/github.com/golang/glog.init.0
[ AGENT ]   /go/src/github.com/DataDog/datadog-agent/.omnibus/src/datadog-agent/src/github.com/DataDog/datadog-agent/vendor/github.com/golang/glog/glog.go:410 +0x203
[ AGENT ]
[ AGENT ] goroutine 9 [chan receive]:
[ AGENT ] github.com/DataDog/datadog-agent/pkg/status/health.(*catalog).run(0xc420364700)
[ AGENT ]   /go/src/github.com/DataDog/datadog-agent/.omnibus/src/datadog-agent/src/github.com/DataDog/datadog-agent/pkg/status/health/health.go:80 +0x6e
[ AGENT ] created by github.com/DataDog/datadog-agent/pkg/status/health.(*catalog).register
[ AGENT ]   /go/src/github.com/DataDog/datadog-agent/.omnibus/src/datadog-agent/src/github.com/DataDog/datadog-agent/pkg/status/health/health.go:52 +0x254
[ AGENT ]
[ AGENT ] goroutine 116 [select]:
[ AGENT ] github.com/DataDog/datadog-agent/pkg/collector/autodiscovery.(*AutoConfig).pollConfigs.func1(0xc42317c000)
[ AGENT ]   /go/src/github.com/DataDog/datadog-agent/.omnibus/src/datadog-agent/src/github.com/DataDog/datadog-agent/pkg/collector/autodiscovery/autoconfig.go:358 +0x17d
[ AGENT ] created by github.com/DataDog/datadog-agent/pkg/collector/autodiscovery.(*AutoConfig).pollConfigs
[ AGENT ]   /go/src/github.com/DataDog/datadog-agent/.omnibus/src/datadog-agent/src/github.com/DataDog/datadog-agent/pkg/collector/autodiscovery/autoconfig.go:356 +0x3f
[ AGENT ]
[ AGENT ] goroutine 131 [select]:
[ AGENT ] github.com/DataDog/datadog-agent/pkg/tagger.(*Tagger).run(0xc42047c1c0, 0x17, 0xc4203615c8)
[ AGENT ]   /go/src/github.com/DataDog/datadog-agent/.omnibus/src/datadog-agent/src/github.com/DataDog/datadog-agent/pkg/tagger/tagger.go:87 +0x225
[ AGENT ] created by github.com/DataDog/datadog-agent/pkg/tagger.(*Tagger).Init
[ AGENT ]   /go/src/github.com/DataDog/datadog-agent/.omnibus/src/datadog-agent/src/github.com/DataDog/datadog-agent/pkg/tagger/tagger.go:81 +0x1b4
[ AGENT ]
[ AGENT ] goroutine 14 [syscall, 2761 minutes]:
[ AGENT ] os/signal.signal_recv(0x0)
[ AGENT ]   /root/.gimme/versions/go1.9.4.linux.amd64/src/runtime/sigqueue.go:131 +0xa6
[ AGENT ] os/signal.loop()
[ AGENT ]   /root/.gimme/versions/go1.9.4.linux.amd64/src/os/signal/signal_unix.go:22 +0x22
[ AGENT ] created by os/signal.init.0
[ AGENT ]   /root/.gimme/versions/go1.9.4.linux.amd64/src/os/signal/signal_unix.go:28 +0x41
[ AGENT ]
[ AGENT ] goroutine 15 [select, 2761 minutes, locked to thread]:
[ AGENT ] runtime.gopark(0x1ff8a78, 0x0, 0x1f4d8fe, 0x6, 0x18, 0x1)
[ AGENT ]   /root/.gimme/versions/go1.9.4.linux.amd64/src/runtime/proc.go:287 +0x12c
[ AGENT ] runtime.selectgo(0xc42002a750, 0xc42006a5a0)
[ AGENT ]   /root/.gimme/versions/go1.9.4.linux.amd64/src/runtime/select.go:395 +0x1149
[ AGENT ] runtime.ensureSigM.func1()
[ AGENT ]   /root/.gimme/versions/go1.9.4.linux.amd64/src/runtime/signal_unix.go:511 +0x220
[ AGENT ] runtime.goexit()
[ AGENT ]   /root/.gimme/versions/go1.9.4.linux.amd64/src/runtime/asm_amd64.s:2337 +0x1
[ AGENT ]
[ AGENT ] goroutine 16 [select, 2761 minutes]:
[ AGENT ] github.com/DataDog/datadog-agent/cmd/agent/app.start.func2(0xc42006a660, 0xc420524780)
[ AGENT ]   /go/src/github.com/DataDog/datadog-agent/.omnibus/src/datadog-agent/src/github.com/DataDog/datadog-agent/cmd/agent/app/start.go:98 +0x134
[ AGENT ] created by github.com/DataDog/datadog-agent/cmd/agent/app.start
[ AGENT ]   /go/src/github.com/DataDog/datadog-agent/.omnibus/src/datadog-agent/src/github.com/DataDog/datadog-agent/cmd/agent/app/start.go:96 +0x13a
[ AGENT ]
[ AGENT ] goroutine 50 [semacquire]:
[ AGENT ] sync.runtime_notifyListWait(0xc420514f10, 0x2dedd)
[ AGENT ]   /root/.gimme/versions/go1.9.4.linux.amd64/src/runtime/sema.go:507 +0x110
[ AGENT ] sync.(*Cond).Wait(0xc420514f00)
[ AGENT ]   /root/.gimme/versions/go1.9.4.linux.amd64/src/sync/cond.go:56 +0x80
[ AGENT ] github.com/DataDog/datadog-agent/vendor/github.com/cihub/seelog.(*asyncLoopLogger).processItem(0xc4204561b0, 0x0)
[ AGENT ]   /go/src/github.com/DataDog/datadog-agent/.omnibus/src/datadog-agent/src/github.com/DataDog/datadog-agent/vendor/github.com/cihub/seelog/behavior_asynclooplogger.go:50 +0x91
[ AGENT ] github.com/DataDog/datadog-agent/vendor/github.com/cihub/seelog.(*asyncLoopLogger).processQueue(0xc4204561b0)
[ AGENT ]   /go/src/github.com/DataDog/datadog-agent/.omnibus/src/datadog-agent/src/github.com/DataDog/datadog-agent/vendor/github.com/cihub/seelog/behavior_asynclooplogger.go:63 +0x44
[ AGENT ] created by github.com/DataDog/datadog-agent/vendor/github.com/cihub/seelog.NewAsyncLoopLogger
[ AGENT ]   /go/src/github.com/DataDog/datadog-agent/.omnibus/src/datadog-agent/src/github.com/DataDog/datadog-agent/vendor/github.com/cihub/seelog/behavior_asynclooplogger.go:40 +0x9f
[ AGENT ]
[ AGENT ] goroutine 51 [IO wait, 2761 minutes]:
[ AGENT ] internal/poll.runtime_pollWait(0x7fa2034a2f70, 0x72, 0xffffffffffffffff)
[ AGENT ]   /root/.gimme/versions/go1.9.4.linux.amd64/src/runtime/netpoll.go:173 +0x57
[ AGENT ] internal/poll.(*pollDesc).wait(0xc42049c398, 0x72, 0xc42003cc00, 0x0, 0x0)
[ AGENT ]   /root/.gimme/versions/go1.9.4.linux.amd64/src/internal/poll/fd_poll_runtime.go:85 +0xae
[ AGENT ] internal/poll.(*pollDesc).waitRead(0xc42049c398, 0xffffffffffffff00, 0x0, 0x0)
[ AGENT ]   /root/.gimme/versions/go1.9.4.linux.amd64/src/internal/poll/fd_poll_runtime.go:90 +0x3d
[ AGENT ] internal/poll.(*FD).Accept(0xc42049c380, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
[ AGENT ]   /root/.gimme/versions/go1.9.4.linux.amd64/src/internal/poll/fd_unix.go:335 +0x1e2
[ AGENT ] net.(*netFD).accept(0xc42049c380, 0x7fa2034fe6c8, 0x0, 0x1ff8ba0)
[ AGENT ]   /root/.gimme/versions/go1.9.4.linux.amd64/src/net/fd_unix.go:238 +0x42
[ AGENT ] net.(*TCPListener).accept(0xc4204980a8, 0xc42003ce30, 0x431328, 0x30)
[ AGENT ]   /root/.gimme/versions/go1.9.4.linux.amd64/src/net/tcpsock_posix.go:136 +0x2e
[ AGENT ] net.(*TCPListener).AcceptTCP(0xc4204980a8, 0xc42014b260, 0xc42014b260, 0x1c66340)
[ AGENT ]   /root/.gimme/versions/go1.9.4.linux.amd64/src/net/tcpsock.go:234 +0x49
[ AGENT ] net/http.tcpKeepAliveListener.Accept(0xc4204980a8, 0xc420012060, 0x1c66340, 0x2fd9340, 0x1f02700)
[ AGENT ]   /root/.gimme/versions/go1.9.4.linux.amd64/src/net/http/server.go:3120 +0x2f
[ AGENT ] net/http.(*Server).Serve(0xc420087ad0, 0x3051a80, 0xc4204980a8, 0x0, 0x0)
[ AGENT ]   /root/.gimme/versions/go1.9.4.linux.amd64/src/net/http/server.go:2695 +0x1b2
[ AGENT ] net/http.(*Server).ListenAndServe(0xc420087ad0, 0xc420087ad0, 0x0)
[ AGENT ]   /root/.gimme/versions/go1.9.4.linux.amd64/src/net/http/server.go:2636 +0xa9
[ AGENT ] net/http.ListenAndServe(0xc420528b90, 0xe, 0x3032780, 0x30b4600, 0xe, 0x0)
[ AGENT ]   /root/.gimme/versions/go1.9.4.linux.amd64/src/net/http/server.go:2882 +0x7f
[ AGENT ] created by github.com/DataDog/datadog-agent/cmd/agent/app.StartAgent
[ AGENT ]   /go/src/github.com/DataDog/datadog-agent/.omnibus/src/datadog-agent/src/github.com/DataDog/datadog-agent/cmd/agent/app/start.go:160 +0x560
[ AGENT ]
[ AGENT ] goroutine 53 [IO wait]:
[ AGENT ] internal/poll.runtime_pollWait(0x7fa2034a2eb0, 0x72, 0xffffffffffffffff)
[ AGENT ]   /root/.gimme/versions/go1.9.4.linux.amd64/src/runtime/netpoll.go:173 +0x57
[ AGENT ] internal/poll.(*pollDesc).wait(0xc4203ea918, 0x72, 0xc42059cc00, 0x0, 0x0)
[ AGENT ]   /root/.gimme/versions/go1.9.4.linux.amd64/src/internal/poll/fd_poll_runtime.go:85 +0xae
[ AGENT ] internal/poll.(*pollDesc).waitRead(0xc4203ea918, 0xffffffffffffff00, 0x0, 0x0)
[ AGENT ]   /root/.gimme/versions/go1.9.4.linux.amd64/src/internal/poll/fd_poll_runtime.go:90 +0x3d
[ AGENT ] internal/poll.(*FD).Accept(0xc4203ea900, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
[ AGENT ]   /root/.gimme/versions/go1.9.4.linux.amd64/src/internal/poll/fd_unix.go:335 +0x1e2
[ AGENT ] net.(*netFD).accept(0xc4203ea900, 0x1ff8ba0, 0xc42059ce88, 0x530000c4226c3e80)
[ AGENT ]   /root/.gimme/versions/go1.9.4.linux.amd64/src/net/fd_unix.go:238 +0x42
[ AGENT ] net.(*TCPListener).accept(0xc42051c308, 0x42111b, 0xc42059ce90, 0x4210c1)
[ AGENT ]   /root/.gimme/versions/go1.9.4.linux.amd64/src/net/tcpsock_posix.go:136 +0x2e
[ AGENT ] net.(*TCPListener).Accept(0xc42051c308, 0x454eeb, 0xc42059cec8, 0x7bc1e8, 0x479540)
[ AGENT ]   /root/.gimme/versions/go1.9.4.linux.amd64/src/net/tcpsock.go:247 +0x49
[ AGENT ] crypto/tls.(*listener).Accept(0xc420225280, 0x1ff83a0, 0xc42b418780, 0x3052280, 0xc42014a570)
[ AGENT ]   /root/.gimme/versions/go1.9.4.linux.amd64/src/crypto/tls/tls.go:52 +0x37
[ AGENT ] net/http.(*Server).Serve(0xc420361450, 0x303dac0, 0xc420225280, 0x0, 0x0)
[ AGENT ]   /root/.gimme/versions/go1.9.4.linux.amd64/src/net/http/server.go:2695 +0x1b2
[ AGENT ] created by github.com/DataDog/datadog-agent/cmd/agent/api.StartServer
[ AGENT ]   /go/src/github.com/DataDog/datadog-agent/.omnibus/src/datadog-agent/src/github.com/DataDog/datadog-agent/cmd/agent/api/server.go:87 +0x7ec
[ AGENT ]
[ AGENT ] goroutine 54 [select]:
[ AGENT ] github.com/DataDog/datadog-agent/pkg/forwarder.(*Worker).Start.func1(0xc42014ac30)
[ AGENT ]   /go/src/github.com/DataDog/datadog-agent/.omnibus/src/datadog-agent/src/github.com/DataDog/datadog-agent/pkg/forwarder/worker.go:78 +0x269
[ AGENT ] created by github.com/DataDog/datadog-agent/pkg/forwarder.(*Worker).Start
[ AGENT ]   /go/src/github.com/DataDog/datadog-agent/.omnibus/src/datadog-agent/src/github.com/DataDog/datadog-agent/pkg/forwarder/worker.go:64 +0x3f
[ AGENT ]
[ AGENT ] goroutine 55 [select]:
[ AGENT ] github.com/DataDog/datadog-agent/pkg/forwarder.(*DefaultForwarder).handleFailedTransactions(0xc420456120)
[ AGENT ]   /go/src/github.com/DataDog/datadog-agent/.omnibus/src/datadog-agent/src/github.com/DataDog/datadog-agent/pkg/forwarder/forwarder.go:186 +0x177
[ AGENT ] created by github.com/DataDog/datadog-agent/pkg/forwarder.(*DefaultForwarder).Start
[ AGENT ]   /go/src/github.com/DataDog/datadog-agent/.omnibus/src/datadog-agent/src/github.com/DataDog/datadog-agent/pkg/forwarder/forwarder.go:227 +0x248
[ AGENT ]
[ AGENT ] goroutine 56 [select]:
[ AGENT ] github.com/DataDog/datadog-agent/pkg/forwarder.(*DefaultForwarder).healthCheckLoop(0xc420456120)
[ AGENT ]   /go/src/github.com/DataDog/datadog-agent/.omnibus/src/datadog-agent/src/github.com/DataDog/datadog-agent/pkg/forwarder/forwarder.go:328 +0x1fc
[ AGENT ] created by github.com/DataDog/datadog-agent/pkg/forwarder.(*DefaultForwarder).Start
[ AGENT ]   /go/src/github.com/DataDog/datadog-agent/.omnibus/src/datadog-agent/src/github.com/DataDog/datadog-agent/pkg/forwarder/forwarder.go:238 +0x6d4
[ AGENT ]
[ AGENT ] goroutine 57 [select]:
[ AGENT ] github.com/DataDog/datadog-agent/pkg/aggregator.(*BufferedAggregator).run(0xc42011c000)
[ AGENT ]   /go/src/github.com/DataDog/datadog-agent/.omnibus/src/datadog-agent/src/github.com/DataDog/datadog-agent/pkg/aggregator/aggregator.go:443 +0x322
[ AGENT ] created by github.com/DataDog/datadog-agent/pkg/aggregator.InitAggregatorWithFlushInterval.func1
[ AGENT ]   /go/src/github.com/DataDog/datadog-agent/.omnibus/src/datadog-agent/src/github.com/DataDog/datadog-agent/pkg/aggregator/aggregator.go:105 +0x7c
[ AGENT ]
[ AGENT ] goroutine 59 [IO wait]:
[ AGENT ] internal/poll.runtime_pollWait(0x7fa2034a2df0, 0x72, 0x0)
[ AGENT ]   /root/.gimme/versions/go1.9.4.linux.amd64/src/runtime/netpoll.go:173 +0x57
[ AGENT ] internal/poll.(*pollDesc).wait(0xc42049c298, 0x72, 0x0, 0x0, 0x0)
[ AGENT ]   /root/.gimme/versions/go1.9.4.linux.amd64/src/internal/poll/fd_poll_runtime.go:85 +0xae
[ AGENT ] internal/poll.(*pollDesc).waitRead(0xc42049c298, 0xc42b360000, 0x2000, 0x2000)
[ AGENT ]   /root/.gimme/versions/go1.9.4.linux.amd64/src/internal/poll/fd_poll_runtime.go:90 +0x3d
[ AGENT ] internal/poll.(*FD).ReadFrom(0xc42049c280, 0xc42b360000, 0x2000, 0x2000, 0x0, 0x0, 0x0, 0x0, 0x0)
[ AGENT ]   /root/.gimme/versions/go1.9.4.linux.amd64/src/internal/poll/fd_unix.go:170 +0x19a
[ AGENT ] net.(*netFD).readFrom(0xc42049c280, 0xc42b360000, 0x2000, 0x2000, 0x8, 0x30331c0, 0xc429e8bb40, 0xc420040ef8, 0x423db0)
[ AGENT ]   /root/.gimme/versions/go1.9.4.linux.amd64/src/net/fd_unix.go:208 +0x5e
[ AGENT ] net.(*UDPConn).readFrom(0xc42000e0b8, 0xc42b360000, 0x2000, 0x2000, 0x0, 0x0, 0xc42221cf80, 0xc42221cfa8)
[ AGENT ]   /root/.gimme/versions/go1.9.4.linux.amd64/src/net/udpsock_posix.go:47 +0x6a
[ AGENT ] net.(*UDPConn).ReadFrom(0xc42000e0b8, 0xc42b360000, 0x2000, 0x2000, 0xb4, 0x303b580, 0xc421300720, 0x0, 0x0)
[ AGENT ]   /root/.gimme/versions/go1.9.4.linux.amd64/src/net/udpsock.go:118 +0x72
[ AGENT ] github.com/DataDog/datadog-agent/pkg/dogstatsd/listeners.(*UDPListener).Listen(0xc420225a00)
[ AGENT ]   /go/src/github.com/DataDog/datadog-agent/.omnibus/src/datadog-agent/src/github.com/DataDog/datadog-agent/pkg/dogstatsd/listeners/udp.go:66 +0x122
[ AGENT ] created by github.com/DataDog/datadog-agent/pkg/dogstatsd.(*Server).handleMessages
[ AGENT ]   /go/src/github.com/DataDog/datadog-agent/.omnibus/src/datadog-agent/src/github.com/DataDog/datadog-agent/pkg/dogstatsd/server.go:125 +0x70
[ AGENT ]
[ AGENT ] goroutine 60 [select]:
[ AGENT ] github.com/DataDog/datadog-agent/pkg/dogstatsd.(*Server).worker(0xc420236de0, 0xc4202366c0, 0xc4202367e0, 0xc420236780)
[ AGENT ]   /go/src/github.com/DataDog/datadog-agent/.omnibus/src/datadog-agent/src/github.com/DataDog/datadog-agent/pkg/dogstatsd/server.go:159 +0x162
[ AGENT ] created by github.com/DataDog/datadog-agent/pkg/dogstatsd.(*Server).handleMessages
[ AGENT ]   /go/src/github.com/DataDog/datadog-agent/.omnibus/src/datadog-agent/src/github.com/DataDog/datadog-agent/pkg/dogstatsd/server.go:136 +0x103
[ AGENT ]
[ AGENT ] goroutine 61 [select]:
[ AGENT ] github.com/DataDog/datadog-agent/pkg/dogstatsd.(*Server).worker(0xc420236de0, 0xc4202366c0, 0xc4202367e0, 0xc420236780)
[ AGENT ]   /go/src/github.com/DataDog/datadog-agent/.omnibus/src/datadog-agent/src/github.com/DataDog/datadog-agent/pkg/dogstatsd/server.go:159 +0x162
[ AGENT ] created by github.com/DataDog/datadog-agent/pkg/dogstatsd.(*Server).handleMessages
[ AGENT ]   /go/src/github.com/DataDog/datadog-agent/.omnibus/src/datadog-agent/src/github.com/DataDog/datadog-agent/pkg/dogstatsd/server.go:136 +0x103
[ AGENT ]
[ AGENT ] goroutine 62 [select]:
[ AGENT ] github.com/DataDog/datadog-agent/pkg/dogstatsd.(*Server).worker(0xc420236de0, 0xc4202366c0, 0xc4202367e0, 0xc420236780)
[ AGENT ]   /go/src/github.com/DataDog/datadog-agent/.omnibus/src/datadog-agent/src/github.com/DataDog/datadog-agent/pkg/dogstatsd/server.go:159 +0x162
[ AGENT ] created by github.com/DataDog/datadog-agent/pkg/dogstatsd.(*Server).handleMessages
[ AGENT ]   /go/src/github.com/DataDog/datadog-agent/.omnibus/src/datadog-agent/src/github.com/DataDog/datadog-agent/pkg/dogstatsd/server.go:136 +0x103
[ AGENT ]
[ AGENT ] goroutine 63 [select]:
[ AGENT ] github.com/DataDog/datadog-agent/pkg/dogstatsd.(*Server).worker(0xc420236de0, 0xc4202366c0, 0xc4202367e0, 0xc420236780)
[ AGENT ]   /go/src/github.com/DataDog/datadog-agent/.omnibus/src/datadog-agent/src/github.com/DataDog/datadog-agent/pkg/dogstatsd/server.go:159 +0x162
[ AGENT ] created by github.com/DataDog/datadog-agent/pkg/dogstatsd.(*Server).handleMessages
[ AGENT ]   /go/src/github.com/DataDog/datadog-agent/.omnibus/src/datadog-agent/src/github.com/DataDog/datadog-agent/pkg/dogstatsd/server.go:136 +0x103
[ AGENT ]
[ AGENT ] goroutine 64 [select]:
[ AGENT ] github.com/DataDog/datadog-agent/pkg/dogstatsd.(*Server).worker(0xc420236de0, 0xc4202366c0, 0xc4202367e0, 0xc420236780)
[ AGENT ]   /go/src/github.com/DataDog/datadog-agent/.omnibus/src/datadog-agent/src/github.com/DataDog/datadog-agent/pkg/dogstatsd/server.go:159 +0x162
[ AGENT ] created by github.com/DataDog/datadog-agent/pkg/dogstatsd.(*Server).handleMessages
[ AGENT ]   /go/src/github.com/DataDog/datadog-agent/.omnibus/src/datadog-agent/src/github.com/DataDog/datadog-agent/pkg/dogstatsd/server.go:136 +0x103
[ AGENT ]
[ AGENT ] goroutine 65 [select]:
[ AGENT ] github.com/DataDog/datadog-agent/pkg/dogstatsd.(*Server).worker(0xc420236de0, 0xc4202366c0, 0xc4202367e0, 0xc420236780)
[ AGENT ]   /go/src/github.com/DataDog/datadog-agent/.omnibus/src/datadog-agent/src/github.com/DataDog/datadog-agent/pkg/dogstatsd/server.go:159 +0x162
[ AGENT ] created by github.com/DataDog/datadog-agent/pkg/dogstatsd.(*Server).handleMessages
[ AGENT ]   /go/src/github.com/DataDog/datadog-agent/.omnibus/src/datadog-agent/src/github.com/DataDog/datadog-agent/pkg/dogstatsd/server.go:136 +0x103
[ AGENT ]
[ AGENT ] goroutine 104 [runnable]:
[ AGENT ] sync.(*RWMutex).RUnlock(0xc4203469c8)
[ AGENT ]   /root/.gimme/versions/go1.9.4.linux.amd64/src/sync/rwmutex.go:62 +0x91
[ AGENT ] github.com/DataDog/datadog-agent/pkg/collector/runner.RemoveCheckStats(0xc42940d4e0, 0x19)
[ AGENT ]   /go/src/github.com/DataDog/datadog-agent/.omnibus/src/datadog-agent/src/github.com/DataDog/datadog-agent/pkg/collector/runner/runner.go:373 +0x99
[ AGENT ] github.com/DataDog/datadog-agent/pkg/collector.(*Collector).StopCheck(0xc422ee5540, 0xc42940d4e0, 0x19, 0x0, 0xc420fa93e0)
[ AGENT ]   /go/src/github.com/DataDog/datadog-agent/.omnibus/src/datadog-agent/src/github.com/DataDog/datadog-agent/pkg/collector/collector.go:178 +0x1fd
[ AGENT ] github.com/DataDog/datadog-agent/pkg/collector/autodiscovery.(*ConfigResolver).processDelService(0xc423177620, 0x3055260, 0xc429351590)
[ AGENT ]   /go/src/github.com/DataDog/datadog-agent/.omnibus/src/datadog-agent/src/github.com/DataDog/datadog-agent/pkg/collector/autodiscovery/configresolver.go:249 +0x21e
[ AGENT ] github.com/DataDog/datadog-agent/pkg/collector/autodiscovery.(*ConfigResolver).listen.func1(0xc423177620)
[ AGENT ]   /go/src/github.com/DataDog/datadog-agent/.omnibus/src/datadog-agent/src/github.com/DataDog/datadog-agent/pkg/collector/autodiscovery/configresolver.go:89 +0x1ca
[ AGENT ] created by github.com/DataDog/datadog-agent/pkg/collector/autodiscovery.(*ConfigResolver).listen
[ AGENT ]   /go/src/github.com/DataDog/datadog-agent/.omnibus/src/datadog-agent/src/github.com/DataDog/datadog-agent/pkg/collector/autodiscovery/configresolver.go:79 +0x3f
[ AGENT ]
[ AGENT ] goroutine 115 [chan send]:
[ AGENT ] github.com/DataDog/datadog-agent/pkg/collector/listeners.(*KubeletListener).removeService(0xc422d011d0, 0xc425c214a0, 0x49)
[ AGENT ]   /go/src/github.com/DataDog/datadog-agent/.omnibus/src/datadog-agent/src/github.com/DataDog/datadog-agent/pkg/collector/listeners/kubelet.go:208 +0x168
[ AGENT ] github.com/DataDog/datadog-agent/pkg/collector/listeners.(*KubeletListener).Listen.func1(0xc422d011d0)
[ AGENT ]   /go/src/github.com/DataDog/datadog-agent/.omnibus/src/datadog-agent/src/github.com/DataDog/datadog-agent/pkg/collector/listeners/kubelet.go:99 +0x27a
[ AGENT ] created by github.com/DataDog/datadog-agent/pkg/collector/listeners.(*KubeletListener).Listen
[ AGENT ]   /go/src/github.com/DataDog/datadog-agent/.omnibus/src/datadog-agent/src/github.com/DataDog/datadog-agent/pkg/collector/listeners/kubelet.go:72 +0x69
[ AGENT ]
[ AGENT ] goroutine 26 [IO wait, 3 minutes]:
[ AGENT ] internal/poll.runtime_pollWait(0x7fa2034a2bb0, 0x72, 0x1)
[ AGENT ]   /root/.gimme/versions/go1.9.4.linux.amd64/src/runtime/netpoll.go:173 +0x57
[ AGENT ] internal/poll.(*pollDesc).wait(0xc4242502e8, 0x72, 0xffffffffffffff01, 0x3035d80, 0x302bfd0)
[ AGENT ]   /root/.gimme/versions/go1.9.4.linux.amd64/src/internal/poll/fd_poll_runtime.go:85 +0xae
[ AGENT ] internal/poll.(*pollDesc).waitRead(0xc4242502e8, 0xc42441a401, 0xba4, 0xba4)
[ AGENT ]   /root/.gimme/versions/go1.9.4.linux.amd64/src/internal/poll/fd_poll_runtime.go:90 +0x3d
[ AGENT ] internal/poll.(*FD).Read(0xc4242502d0, 0xc42441a45c, 0xba4, 0xba4, 0x0, 0x0, 0x0)
[ AGENT ]   /root/.gimme/versions/go1.9.4.linux.amd64/src/internal/poll/fd_unix.go:126 +0x18a
[ AGENT ] os.(*File).read(0xc424225e80, 0xc42441a45c, 0xba4, 0xba4, 0x0, 0x2, 0x3)
[ AGENT ]   /root/.gimme/versions/go1.9.4.linux.amd64/src/os/file_unix.go:216 +0x4e
[ AGENT ] os.(*File).Read(0xc424225e80, 0xc42441a45c, 0xba4, 0xba4, 0x1ff8ba0, 0xc4299b3200, 0xc4204a1828)
[ AGENT ]   /root/.gimme/versions/go1.9.4.linux.amd64/src/os/file.go:103 +0x6d
[ AGENT ] bufio.(*Scanner).Scan(0xc424478f40, 0x1)
[ AGENT ]   /root/.gimme/versions/go1.9.4.linux.amd64/src/bufio/scan.go:207 +0xaf
[ AGENT ] github.com/DataDog/datadog-agent/pkg/jmxfetch.(*JMXFetch).Start.func1(0x303c0c0, 0xc424225e80)
[ AGENT ]   /go/src/github.com/DataDog/datadog-agent/.omnibus/src/datadog-agent/src/github.com/DataDog/datadog-agent/pkg/jmxfetch/jmxfetch.go:207 +0xc8
[ AGENT ] created by github.com/DataDog/datadog-agent/pkg/jmxfetch.(*JMXFetch).Start
[ AGENT ]   /go/src/github.com/DataDog/datadog-agent/.omnibus/src/datadog-agent/src/github.com/DataDog/datadog-agent/pkg/jmxfetch/jmxfetch.go:205 +0xd4a
[ AGENT ]
[ AGENT ] goroutine 27 [IO wait, 2760 minutes]:
[ AGENT ] internal/poll.runtime_pollWait(0x7fa2034a2970, 0x72, 0x1)
[ AGENT ]   /root/.gimme/versions/go1.9.4.linux.amd64/src/runtime/netpoll.go:173 +0x57
[ AGENT ] internal/poll.(*pollDesc).wait(0xc424250388, 0x72, 0xffffffffffffff01, 0x3035d80, 0x302bfd0)
[ AGENT ]   /root/.gimme/versions/go1.9.4.linux.amd64/src/internal/poll/fd_poll_runtime.go:85 +0xae
[ AGENT ] internal/poll.(*pollDesc).waitRead(0xc424250388, 0xc422fcb001, 0x1000, 0x1000)
[ AGENT ]   /root/.gimme/versions/go1.9.4.linux.amd64/src/internal/poll/fd_poll_runtime.go:90 +0x3d
[ AGENT ] internal/poll.(*FD).Read(0xc424250370, 0xc422fcb000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
[ AGENT ]   /root/.gimme/versions/go1.9.4.linux.amd64/src/internal/poll/fd_unix.go:126 +0x18a
[ AGENT ] os.(*File).read(0xc424225e90, 0xc422fcb000, 0x1000, 0x1000, 0x3, 0x3, 0xc42002ef76)
[ AGENT ]   /root/.gimme/versions/go1.9.4.linux.amd64/src/os/file_unix.go:216 +0x4e
[ AGENT ] os.(*File).Read(0xc424225e90, 0xc422fcb000, 0x1000, 0x1000, 0x1000, 0x1000, 0xc420022000)
[ AGENT ]   /root/.gimme/versions/go1.9.4.linux.amd64/src/os/file.go:103 +0x6d
[ AGENT ] bufio.(*Scanner).Scan(0xc42002ef40, 0x303c0c0)
[ AGENT ]   /root/.gimme/versions/go1.9.4.linux.amd64/src/bufio/scan.go:207 +0xaf
[ AGENT ] github.com/DataDog/datadog-agent/pkg/jmxfetch.(*JMXFetch).Start.func2(0x303c0c0, 0xc424225e90)
[ AGENT ]   /go/src/github.com/DataDog/datadog-agent/.omnibus/src/datadog-agent/src/github.com/DataDog/datadog-agent/pkg/jmxfetch/jmxfetch.go:219 +0xc8
[ AGENT ] created by github.com/DataDog/datadog-agent/pkg/jmxfetch.(*JMXFetch).Start
[ AGENT ]   /go/src/github.com/DataDog/datadog-agent/.omnibus/src/datadog-agent/src/github.com/DataDog/datadog-agent/pkg/jmxfetch/jmxfetch.go:217 +0xda1
[ AGENT ]
[ AGENT ] goroutine 24 [IO wait]:
[ AGENT ] internal/poll.runtime_pollWait(0x7fa2034a2af0, 0x72, 0x0)
[ AGENT ]   /root/.gimme/versions/go1.9.4.linux.amd64/src/runtime/netpoll.go:173 +0x57
[ AGENT ] internal/poll.(*pollDesc).wait(0xc42047e298, 0x72, 0xffffffffffffff00, 0x3035d80, 0x302bfd0)
[ AGENT ]   /root/.gimme/versions/go1.9.4.linux.amd64/src/internal/poll/fd_poll_runtime.go:85 +0xae
[ AGENT ] internal/poll.(*pollDesc).waitRead(0xc42047e298, 0xc4205ba000, 0x8000, 0x8000)
[ AGENT ]   /root/.gimme/versions/go1.9.4.linux.amd64/src/internal/poll/fd_poll_runtime.go:90 +0x3d
[ AGENT ] internal/poll.(*FD).Read(0xc42047e280, 0xc4205ba000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
[ AGENT ]   /root/.gimme/versions/go1.9.4.linux.amd64/src/internal/poll/fd_unix.go:126 +0x18a
[ AGENT ] net.(*netFD).Read(0xc42047e280, 0xc4205ba000, 0x8000, 0x8000, 0x185e, 0x8, 0x0)
[ AGENT ]   /root/.gimme/versions/go1.9.4.linux.amd64/src/net/fd_unix.go:202 +0x52
[ AGENT ] net.(*conn).Read(0xc420498070, 0xc4205ba000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
[ AGENT ]   /root/.gimme/versions/go1.9.4.linux.amd64/src/net/net.go:176 +0x6d
[ AGENT ] crypto/tls.(*block).readFromUntil(0xc420668870, 0x7fa200082028, 0xc420498070, 0x5, 0xc420498070, 0x0)
[ AGENT ]   /root/.gimme/versions/go1.9.4.linux.amd64/src/crypto/tls/conn.go:488 +0x95
[ AGENT ] crypto/tls.(*Conn).readRecord(0xc42034d500, 0x1ff8b17, 0xc42034d620, 0x0)
[ AGENT ]   /root/.gimme/versions/go1.9.4.linux.amd64/src/crypto/tls/conn.go:590 +0xe0
[ AGENT ] crypto/tls.(*Conn).Read(0xc42034d500, 0xc420124000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
[ AGENT ]   /root/.gimme/versions/go1.9.4.linux.amd64/src/crypto/tls/conn.go:1134 +0x110
[ AGENT ] bufio.(*Reader).Read(0xc4203047e0, 0xc4203131b8, 0x9, 0x9, 0xc421626580, 0xc4204f0f28, 0xc420639c50)
[ AGENT ]   /root/.gimme/versions/go1.9.4.linux.amd64/src/bufio/bufio.go:213 +0x30b
[ AGENT ] io.ReadAtLeast(0x302db00, 0xc4203047e0, 0xc4203131b8, 0x9, 0x9, 0x9, 0xc4256df401, 0xc420639cf8, 0xc4c70b)
[ AGENT ]   /root/.gimme/versions/go1.9.4.linux.amd64/src/io/io.go:309 +0x86
[ AGENT ] io.ReadFull(0x302db00, 0xc4203047e0, 0xc4203131b8, 0x9, 0x9, 0x302e340, 0xc420042040, 0xc420639d28)
[ AGENT ]   /root/.gimme/versions/go1.9.4.linux.amd64/src/io/io.go:327 +0x58
[ AGENT ] github.com/DataDog/datadog-agent/vendor/golang.org/x/net/http2.readFrameHeader(0xc4203131b8, 0x9, 0x9, 0x302db00, 0xc4203047e0, 0x0, 0x0, 0xc4210d5980, 0x0)
[ AGENT ]   /go/src/github.com/DataDog/datadog-agent/.omnibus/src/datadog-agent/src/github.com/DataDog/datadog-agent/vendor/golang.org/x/net/http2/frame.go:237 +0x7b
[ AGENT ] github.com/DataDog/datadog-agent/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc420313180, 0xc4210d5980, 0x0, 0x0, 0x0)
[ AGENT ]   /go/src/github.com/DataDog/datadog-agent/.omnibus/src/datadog-agent/src/github.com/DataDog/datadog-agent/vendor/golang.org/x/net/http2/frame.go:492 +0xa4
[ AGENT ] github.com/DataDog/datadog-agent/vendor/golang.org/x/net/http2.(*clientConnReadLoop).run(0xc420639fb8, 0x1ff7a58, 0xc4202df7b8)
[ AGENT ]   /go/src/github.com/DataDog/datadog-agent/.omnibus/src/datadog-agent/src/github.com/DataDog/datadog-agent/vendor/golang.org/x/net/http2/transport.go:1477 +0x90
[ AGENT ] github.com/DataDog/datadog-agent/vendor/golang.org/x/net/http2.(*ClientConn).readLoop(0xc420010540)
[ AGENT ]   /go/src/github.com/DataDog/datadog-agent/.omnibus/src/datadog-agent/src/github.com/DataDog/datadog-agent/vendor/golang.org/x/net/http2/transport.go:1405 +0x68
[ AGENT ] created by github.com/DataDog/datadog-agent/vendor/golang.org/x/net/http2.(*Transport).newClientConn
[ AGENT ]   /go/src/github.com/DataDog/datadog-agent/.omnibus/src/datadog-agent/src/github.com/DataDog/datadog-agent/vendor/golang.org/x/net/http2/transport.go:592 +0x6c4
[ AGENT ]
[ AGENT ] goroutine 130 [chan receive]:
[ AGENT ] github.com/DataDog/datadog-agent/pkg/util/kubernetes/apiserver.(*APIClient).StartMetadataMapping.func1(0xc422ee54c0, 0xc42047e000)
[ AGENT ]   /go/src/github.com/DataDog/datadog-agent/.omnibus/src/datadog-agent/src/github.com/DataDog/datadog-agent/pkg/util/kubernetes/apiserver/apiserver.go:248 +0x31
[ AGENT ] created by github.com/DataDog/datadog-agent/pkg/util/kubernetes/apiserver.(*APIClient).StartMetadataMapping
[ AGENT ]   /go/src/github.com/DataDog/datadog-agent/.omnibus/src/datadog-agent/src/github.com/DataDog/datadog-agent/pkg/util/kubernetes/apiserver/apiserver.go:245 +0x5c
[ AGENT ]
[ AGENT ] goroutine 133 [select, 2760 minutes]:
[ AGENT ] github.com/DataDog/datadog-agent/pkg/collector/corechecks/embed.(*JMXCheck).run(0x2fdd600, 0x20ac3041, 0x165bb9b81)
[ AGENT ]   /go/src/github.com/DataDog/datadog-agent/.omnibus/src/datadog-agent/src/github.com/DataDog/datadog-agent/pkg/collector/corechecks/embed/jmx.go:101 +0x2cc
[ AGENT ] github.com/DataDog/datadog-agent/pkg/collector/corechecks/embed.(*JMXCheck).(github.com/DataDog/datadog-agent/pkg/collector/corechecks/embed.run)-fm(0xbebe79cbe0ac3041, 0x165bb9b81)
[ AGENT ]   /go/src/github.com/DataDog/datadog-agent/.omnibus/src/datadog-agent/src/github.com/DataDog/datadog-agent/pkg/collector/corechecks/embed/jmx.go:71 +0x2a
[ AGENT ] github.com/DataDog/datadog-agent/pkg/collector/check.Retry(0x12a05f200, 0x3, 0xc42440fdf0, 0x1f50f13, 0x9, 0x4a1d86, 0x5b196faf)
[ AGENT ]   /go/src/github.com/DataDog/datadog-agent/.omnibus/src/datadog-agent/src/github.com/DataDog/datadog-agent/pkg/collector/check/retry.go:31 +0x9f
[ AGENT ] github.com/DataDog/datadog-agent/pkg/collector/corechecks/embed.(*JMXCheck).Run(0x2fdd600, 0x165bb8f94, 0x30b4f60)
[ AGENT ]   /go/src/github.com/DataDog/datadog-agent/.omnibus/src/datadog-agent/src/github.com/DataDog/datadog-agent/pkg/collector/corechecks/embed/jmx.go:71 +0xe1
[ AGENT ] github.com/DataDog/datadog-agent/pkg/collector/runner.(*Runner).work(0xc422ed5a10)
[ AGENT ]   /go/src/github.com/DataDog/datadog-agent/.omnibus/src/datadog-agent/src/github.com/DataDog/datadog-agent/pkg/collector/runner/runner.go:255 +0x4a1
[ AGENT ] created by github.com/DataDog/datadog-agent/pkg/collector/runner.(*Runner).AddWorker
[ AGENT ]   /go/src/github.com/DataDog/datadog-agent/.omnibus/src/datadog-agent/src/github.com/DataDog/datadog-agent/pkg/collector/runner/runner.go:100 +0x8a
[ AGENT ]
[ AGENT ] goroutine 134 [chan receive, 2760 minutes]:
[ AGENT ] github.com/DataDog/datadog-agent/pkg/collector/scheduler.(*Scheduler).Run.func1(0xc422ebdc20)
[ AGENT ]   /go/src/github.com/DataDog/datadog-agent/.omnibus/src/datadog-agent/src/github.com/DataDog/datadog-agent/pkg/collector/scheduler/scheduler.go:136 +0xc9
[ AGENT ] created by github.com/DataDog/datadog-agent/pkg/collector/scheduler.(*Scheduler).Run
[ AGENT ]   /go/src/github.com/DataDog/datadog-agent/.omnibus/src/datadog-agent/src/github.com/DataDog/datadog-agent/pkg/collector/scheduler/scheduler.go:124 +0xce
[ AGENT ]
[ AGENT ] goroutine 136 [select]:
[ AGENT ] github.com/DataDog/datadog-agent/pkg/collector/scheduler.(*jobQueue).waitForTick(0xc4243a72c0, 0xc422e46180, 0x47ce01)
[ AGENT ]   /go/src/github.com/DataDog/datadog-agent/.omnibus/src/datadog-agent/src/github.com/DataDog/datadog-agent/pkg/collector/scheduler/job.go:93 +0x2e1
[ AGENT ] github.com/DataDog/datadog-agent/pkg/collector/scheduler.(*jobQueue).run.func1(0xc4243a72c0, 0xc422e46180)
[ AGENT ]   /go/src/github.com/DataDog/datadog-agent/.omnibus/src/datadog-agent/src/github.com/DataDog/datadog-agent/pkg/collector/scheduler/job.go:71 +0x35
[ AGENT ] created by github.com/DataDog/datadog-agent/pkg/collector/scheduler.(*jobQueue).run
[ AGENT ]   /go/src/github.com/DataDog/datadog-agent/.omnibus/src/datadog-agent/src/github.com/DataDog/datadog-agent/pkg/collector/scheduler/job.go:70 +0x49
[ AGENT ]
[ AGENT ] goroutine 28 [syscall, 2760 minutes]:
[ AGENT ] syscall.Syscall6(0xf7, 0x1, 0x1a9, 0xc4202da580, 0x1000004, 0x0, 0x0, 0xd5bd34f6ed4d78a6, 0x658956957b521954, 0x5ad00a946bd28df8)
[ AGENT ]   /root/.gimme/versions/go1.9.4.linux.amd64/src/syscall/asm_linux_amd64.s:44 +0x5
[ AGENT ] os.(*Process).blockUntilWaitable(0xc42421ab70, 0xc2b3547f8f2e7480, 0xae8fd27db192735d, 0x4096b42d7901f37b)
[ AGENT ]   /root/.gimme/versions/go1.9.4.linux.amd64/src/os/wait_waitid.go:31 +0xa5
[ AGENT ] os.(*Process).wait(0xc42421ab70, 0x1f5bee7c46d21c1, 0x833951f98027f678, 0xe14ccded3036cce7)
[ AGENT ]   /root/.gimme/versions/go1.9.4.linux.amd64/src/os/exec_unix.go:22 +0x42
[ AGENT ] os.(*Process).Wait(0xc42421ab70, 0x48d82c9f5c49a440, 0xcb95294a5f6147b7, 0x1b1c5223b22a3534)
[ AGENT ]   /root/.gimme/versions/go1.9.4.linux.amd64/src/os/exec.go:115 +0x2b
[ AGENT ] os/exec.(*Cmd).Wait(0xc424412000, 0xb65ef2d72cad5d6d, 0xf6950db45032ce3)
[ AGENT ]   /root/.gimme/versions/go1.9.4.linux.amd64/src/os/exec/exec.go:446 +0x62
[ AGENT ] github.com/DataDog/datadog-agent/pkg/jmxfetch.(*JMXFetch).Wait(0x30b5320, 0xab3c9afc4b734fc7, 0xa948bcf57c1af779)
[ AGENT ]   /go/src/github.com/DataDog/datadog-agent/.omnibus/src/datadog-agent/src/github.com/DataDog/datadog-agent/pkg/jmxfetch/jmxfetch.go:248 +0x32
[ AGENT ] github.com/DataDog/datadog-agent/pkg/collector/corechecks/embed.(*JMXCheck).run.func1(0xc4243f8480, 0x2fdd600)
[ AGENT ]   /go/src/github.com/DataDog/datadog-agent/.omnibus/src/datadog-agent/src/github.com/DataDog/datadog-agent/pkg/collector/corechecks/embed/jmx.go:98 +0x2e
[ AGENT ] created by github.com/DataDog/datadog-agent/pkg/collector/corechecks/embed.(*JMXCheck).run
[ AGENT ]   /go/src/github.com/DataDog/datadog-agent/.omnibus/src/datadog-agent/src/github.com/DataDog/datadog-agent/pkg/collector/corechecks/embed/jmx.go:97 +0x1fd
[ AGENT ]
[ AGENT ] goroutine 163 [select]:
[ AGENT ] github.com/DataDog/datadog-agent/pkg/metadata.(*Scheduler).AddCollector.func1(0xc42287e6c0, 0xc4227758c0, 0x302ebc0, 0x30f9e30, 0xc422d01220, 0x1f4b127, 0x4)
[ AGENT ]   /go/src/github.com/DataDog/datadog-agent/.omnibus/src/datadog-agent/src/github.com/DataDog/datadog-agent/pkg/metadata/scheduler.go:68 +0xeb
[ AGENT ] created by github.com/DataDog/datadog-agent/pkg/metadata.(*Scheduler).AddCollector
[ AGENT ]   /go/src/github.com/DataDog/datadog-agent/.omnibus/src/datadog-agent/src/github.com/DataDog/datadog-agent/pkg/metadata/scheduler.go:66 +0x15e
[ AGENT ]
[ AGENT ] goroutine 164 [select]:
[ AGENT ] github.com/DataDog/datadog-agent/pkg/metadata.(*Scheduler).AddCollector.func1(0xc42287e6d8, 0xc422775900, 0x302e540, 0x30f9e30, 0xc422d01220, 0x1f55621, 0xc)
[ AGENT ]   /go/src/github.com/DataDog/datadog-agent/.omnibus/src/datadog-agent/src/github.com/DataDog/datadog-agent/pkg/metadata/scheduler.go:68 +0xeb
[ AGENT ] created by github.com/DataDog/datadog-agent/pkg/metadata.(*Scheduler).AddCollector
[ AGENT ]   /go/src/github.com/DataDog/datadog-agent/.omnibus/src/datadog-agent/src/github.com/DataDog/datadog-agent/pkg/metadata/scheduler.go:66 +0x15e
[ AGENT ]
[ AGENT ] goroutine 165 [select]:
[ AGENT ] github.com/DataDog/datadog-agent/pkg/metadata.(*Scheduler).AddCollector.func1(0xc42287e6e0, 0xc422775940, 0x302ec00, 0x30f9e30, 0xc422d01220, 0x1f51c6f, 0x9)
[ AGENT ]   /go/src/github.com/DataDog/datadog-agent/.omnibus/src/datadog-agent/src/github.com/DataDog/datadog-agent/pkg/metadata/scheduler.go:68 +0xeb
[ AGENT ] created by github.com/DataDog/datadog-agent/pkg/metadata.(*Scheduler).AddCollector
[ AGENT ]   /go/src/github.com/DataDog/datadog-agent/.omnibus/src/datadog-agent/src/github.com/DataDog/datadog-agent/pkg/metadata/scheduler.go:66 +0x15e
[ AGENT ]
[ AGENT ] goroutine 202770 [IO wait]:
[ AGENT ] internal/poll.runtime_pollWait(0x7fa2034a2730, 0x72, 0x0)
[ AGENT ]   /root/.gimme/versions/go1.9.4.linux.amd64/src/runtime/netpoll.go:173 +0x57
[ AGENT ] internal/poll.(*pollDesc).wait(0xc426a55718, 0x72, 0xffffffffffffff00, 0x3035d80, 0x302bfd0)
[ AGENT ]   /root/.gimme/versions/go1.9.4.linux.amd64/src/internal/poll/fd_poll_runtime.go:85 +0xae
[ AGENT ] internal/poll.(*pollDesc).waitRead(0xc426a55718, 0xc426b64000, 0x1000, 0x1000)
[ AGENT ]   /root/.gimme/versions/go1.9.4.linux.amd64/src/internal/poll/fd_poll_runtime.go:90 +0x3d
[ AGENT ] internal/poll.(*FD).Read(0xc426a55700, 0xc426b64000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
[ AGENT ]   /root/.gimme/versions/go1.9.4.linux.amd64/src/internal/poll/fd_unix.go:126 +0x18a
[ AGENT ] net.(*netFD).Read(0xc426a55700, 0xc426b64000, 0x1000, 0x1000, 0x44d8fb, 0xc42063ca40, 0x478e70)
[ AGENT ]   /root/.gimme/versions/go1.9.4.linux.amd64/src/net/fd_unix.go:202 +0x52
[ AGENT ] net.(*conn).Read(0xc422373a40, 0xc426b64000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
[ AGENT ]   /root/.gimme/versions/go1.9.4.linux.amd64/src/net/net.go:176 +0x6d
[ AGENT ] net/http.(*persistConn).Read(0xc426614480, 0xc426b64000, 0x1000, 0x1000, 0xc426a971a0, 0xc421b7d5b8, 0x476f10)
[ AGENT ]   /root/.gimme/versions/go1.9.4.linux.amd64/src/net/http/transport.go:1391 +0x140
[ AGENT ] bufio.(*Reader).fill(0xc424504c00)
[ AGENT ]   /root/.gimme/versions/go1.9.4.linux.amd64/src/bufio/bufio.go:97 +0x11a
[ AGENT ] bufio.(*Reader).Peek(0xc424504c00, 0x1, 0x0, 0x0, 0x0, 0xc4213224e0, 0x0)
[ AGENT ]   /root/.gimme/versions/go1.9.4.linux.amd64/src/bufio/bufio.go:129 +0x3a
[ AGENT ] net/http.(*persistConn).readLoop(0xc426614480)
[ AGENT ]   /root/.gimme/versions/go1.9.4.linux.amd64/src/net/http/transport.go:1539 +0x185
[ AGENT ] created by net/http.(*Transport).dialConn
[ AGENT ]   /root/.gimme/versions/go1.9.4.linux.amd64/src/net/http/transport.go:1186 +0xa2e
[ AGENT ]
[ AGENT ] goroutine 223163 [select]:
[ AGENT ] net/http.(*persistConn).writeLoop(0xc428d319e0)
[ AGENT ]   /root/.gimme/versions/go1.9.4.linux.amd64/src/net/http/transport.go:1759 +0x165
[ AGENT ] created by net/http.(*Transport).dialConn
[ AGENT ]   /root/.gimme/versions/go1.9.4.linux.amd64/src/net/http/transport.go:1187 +0xa53
[ AGENT ]
[ AGENT ] goroutine 121407 [select, 327 minutes]:
[ AGENT ] net/http.(*persistConn).writeLoop(0xc42166c000)
[ AGENT ]   /root/.gimme/versions/go1.9.4.linux.amd64/src/net/http/transport.go:1759 +0x165
[ AGENT ] created by net/http.(*Transport).dialConn
[ AGENT ]   /root/.gimme/versions/go1.9.4.linux.amd64/src/net/http/transport.go:1187 +0xa53
[ AGENT ]
[ AGENT ] goroutine 121406 [IO wait, 327 minutes]:
[ AGENT ] internal/poll.runtime_pollWait(0x7fa2034a2d30, 0x72, 0x0)
[ AGENT ]   /root/.gimme/versions/go1.9.4.linux.amd64/src/runtime/netpoll.go:173 +0x57
[ AGENT ] internal/poll.(*pollDesc).wait(0xc4212d3118, 0x72, 0xffffffffffffff00, 0x3035d80, 0x302bfd0)
[ AGENT ]   /root/.gimme/versions/go1.9.4.linux.amd64/src/internal/poll/fd_poll_runtime.go:85 +0xae
[ AGENT ] internal/poll.(*pollDesc).waitRead(0xc4212d3118, 0xc4226ae000, 0x1000, 0x1000)
[ AGENT ]   /root/.gimme/versions/go1.9.4.linux.amd64/src/internal/poll/fd_poll_runtime.go:90 +0x3d
[ AGENT ] internal/poll.(*FD).Read(0xc4212d3100, 0xc4226ae000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
[ AGENT ]   /root/.gimme/versions/go1.9.4.linux.amd64/src/internal/poll/fd_unix.go:126 +0x18a
[ AGENT ] net.(*netFD).Read(0xc4212d3100, 0xc4226ae000, 0x1000, 0x1000, 0x44d8fb, 0xc420cbca40, 0x478e70)
[ AGENT ]   /root/.gimme/versions/go1.9.4.linux.amd64/src/net/fd_unix.go:202 +0x52
[ AGENT ] net.(*conn).Read(0xc42051c070, 0xc4226ae000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
[ AGENT ]   /root/.gimme/versions/go1.9.4.linux.amd64/src/net/net.go:176 +0x6d
[ AGENT ] net/http.(*persistConn).Read(0xc42166c000, 0xc4226ae000, 0x1000, 0x1000, 0xc4209e4180, 0xc423fd9f18, 0x476f10)
[ AGENT ]   /root/.gimme/versions/go1.9.4.linux.amd64/src/net/http/transport.go:1391 +0x140
[ AGENT ] bufio.(*Reader).fill(0xc424ba21e0)
[ AGENT ]   /root/.gimme/versions/go1.9.4.linux.amd64/src/bufio/bufio.go:97 +0x11a
[ AGENT ] bufio.(*Reader).Peek(0xc424ba21e0, 0x1, 0x0, 0x0, 0x0, 0xc426a78c60, 0x0)
[ AGENT ]   /root/.gimme/versions/go1.9.4.linux.amd64/src/bufio/bufio.go:129 +0x3a
[ AGENT ] net/http.(*persistConn).readLoop(0xc42166c000)
[ AGENT ]   /root/.gimme/versions/go1.9.4.linux.amd64/src/net/http/transport.go:1539 +0x185
[ AGENT ] created by net/http.(*Transport).dialConn
[ AGENT ]   /root/.gimme/versions/go1.9.4.linux.amd64/src/net/http/transport.go:1186 +0xa2e
[ AGENT ]
[ AGENT ] goroutine 223162 [IO wait]:
[ AGENT ] internal/poll.runtime_pollWait(0x7fa2034a2430, 0x72, 0x0)
[ AGENT ]   /root/.gimme/versions/go1.9.4.linux.amd64/src/runtime/netpoll.go:173 +0x57
[ AGENT ] internal/poll.(*pollDesc).wait(0xc429d1bf98, 0x72, 0xffffffffffffff00, 0x3035d80, 0x302bfd0)
[ AGENT ]   /root/.gimme/versions/go1.9.4.linux.amd64/src/internal/poll/fd_poll_runtime.go:85 +0xae
[ AGENT ] internal/poll.(*pollDesc).waitRead(0xc429d1bf98, 0xc420589400, 0x400, 0x400)
[ AGENT ]   /root/.gimme/versions/go1.9.4.linux.amd64/src/internal/poll/fd_poll_runtime.go:90 +0x3d
[ AGENT ] internal/poll.(*FD).Read(0xc429d1bf80, 0xc420589400, 0x400, 0x400, 0x0, 0x0, 0x0)
[ AGENT ]   /root/.gimme/versions/go1.9.4.linux.amd64/src/internal/poll/fd_unix.go:126 +0x18a
[ AGENT ] net.(*netFD).Read(0xc429d1bf80, 0xc420589400, 0x400, 0x400, 0x1, 0x2, 0xc4204b6788)
[ AGENT ]   /root/.gimme/versions/go1.9.4.linux.amd64/src/net/fd_unix.go:202 +0x52
[ AGENT ] net.(*conn).Read(0xc4240f0000, 0xc420589400, 0x400, 0x400, 0x0, 0x0, 0x0)
[ AGENT ]   /root/.gimme/versions/go1.9.4.linux.amd64/src/net/net.go:176 +0x6d
[ AGENT ] crypto/tls.(*block).readFromUntil(0xc421842780, 0x7fa200082028, 0xc4240f0000, 0x5, 0xc4240f0000, 0xc4284ab8f0)
[ AGENT ]   /root/.gimme/versions/go1.9.4.linux.amd64/src/crypto/tls/conn.go:488 +0x95
[ AGENT ] crypto/tls.(*Conn).readRecord(0xc42008ea80, 0x1ff8b17, 0xc42008eba0, 0x44d8fb)
[ AGENT ]   /root/.gimme/versions/go1.9.4.linux.amd64/src/crypto/tls/conn.go:590 +0xe0
[ AGENT ] crypto/tls.(*Conn).Read(0xc42008ea80, 0xc4210e9000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
[ AGENT ]   /root/.gimme/versions/go1.9.4.linux.amd64/src/crypto/tls/conn.go:1134 +0x110
[ AGENT ] net/http.(*persistConn).Read(0xc428d319e0, 0xc4210e9000, 0x1000, 0x1000, 0xc426d38000, 0xc4255dc5f8, 0x476f10)
[ AGENT ]   /root/.gimme/versions/go1.9.4.linux.amd64/src/net/http/transport.go:1391 +0x140
[ AGENT ] bufio.(*Reader).fill(0xc4274403c0)
[ AGENT ]   /root/.gimme/versions/go1.9.4.linux.amd64/src/bufio/bufio.go:97 +0x11a
[ AGENT ] bufio.(*Reader).Peek(0xc4274403c0, 0x1, 0x0, 0x0, 0x0, 0xc421b7c060, 0x0)
[ AGENT ]   /root/.gimme/versions/go1.9.4.linux.amd64/src/bufio/bufio.go:129 +0x3a
[ AGENT ] net/http.(*persistConn).readLoop(0xc428d319e0)
[ AGENT ]   /root/.gimme/versions/go1.9.4.linux.amd64/src/net/http/transport.go:1539 +0x185
[ AGENT ] created by net/http.(*Transport).dialConn
[ AGENT ]   /root/.gimme/versions/go1.9.4.linux.amd64/src/net/http/transport.go:1186 +0xa2e
[ AGENT ]
[ AGENT ] goroutine 202771 [select]:
[ AGENT ] net/http.(*persistConn).writeLoop(0xc426614480)
[ AGENT ]   /root/.gimme/versions/go1.9.4.linux.amd64/src/net/http/transport.go:1759 +0x165
[ AGENT ] created by net/http.(*Transport).dialConn
[ AGENT ]   /root/.gimme/versions/go1.9.4.linux.amd64/src/net/http/transport.go:1187 +0xa53
[ AGENT ] AGENT EXITED WITH CODE 2, SIGNAL 0, KILLING CONTAINER
[ AGENT ] AGENT EXITED WITH CODE 256, SIGNAL 15, KILLING CONTAINER

In addition, for some reason, Datadog pod didn't die, despite its log saying 

[ AGENT ] AGENT EXITED WITH CODE 2, SIGNAL 0, KILLING CONTAINER
[ AGENT ] AGENT EXITED WITH CODE 256, SIGNAL 15, KILLING CONTAINER

It would be great to have the pod dead in such scenarios, so Kubernetes could create a new pod in case of failure.

deployment:

apiVersion: v1
items:
- apiVersion: extensions/v1beta1
  kind: Deployment
  metadata:
    labels:
      app: datadog-integration
    name: datadog-integration
    namespace: datadog-integration
  spec:
    replicas: 1
    selector:
      matchLabels:
        app: datadog-integration
    strategy:
      rollingUpdate:
        maxSurge: 1
        maxUnavailable: 1
      type: RollingUpdate
    template:
      metadata:
        creationTimestamp: null
        labels:
          app: datadog-integration
      spec:
        containers:
        - env:
          - name: DD_API_KEY
            value: xxxx
          - name: DD_COLLECT_KUBERNETES_EVENTS
            value: "false"
          - name: DD_HOSTNAME
            value: dd-integration-agent-v6
          - name: KUBERNETES
            value: "yes"
          - name: DD_DOGSTATSD_NON_LOCAL_TRAFFIC
            value: "true"
          - name: DD_CONF_HISTOGRAM_PERCENTILES
            value: 0.90, 0.95, 0.99
          - name: DD_KUBERNETES_KUBELET_HOST
            valueFrom:
              fieldRef:
                apiVersion: v1
                fieldPath: status.hostIP
          image: datadog/agent:6.2.1-jmx
          imagePullPolicy: IfNotPresent
          name: datadog-agent
          resources:
            limits:
              cpu: 500m
              memory: 1Gi
            requests:
              cpu: 70m
              memory: 750Mi
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
          [...]
        dnsPolicy: ClusterFirst
        restartPolicy: Always
        schedulerName: default-scheduler
        securityContext: {}
        serviceAccount: datadog-agent
        serviceAccountName: datadog-agent
        terminationGracePeriodSeconds: 30
        volumes:
        [...]
igorvpcleao commented 6 years ago

maybe related to https://github.com/DataDog/datadog-agent/issues/1641, but it seems https://github.com/DataDog/datadog-agent/issues/1641 has been already fixed https://github.com/DataDog/datadog-agent/commit/84d14c86d5ba220b95601a8866cc1a5ed566511f