search

alibaba / MongoShake

MongoShake is a universal data replication platform based on MongoDB's oplog. Redundant replication and active-active replication are two most important functions. 基于mongodb oplog的集群复制工具，可以满足迁移和同步的需求，进一步实现灾备和多活功能。
GNU General Public License v3.0
1.72k stars 441 forks source link

关于receiver的使用 #212

Closed zlklink closed 5 years ago

zlklink commented 5 years ago

按照FAQ中的叙述，./receiver -conf=../conf/receiver.conf -verbose 实际上直接下载的编译后的文件并没有receiver.conf。 我将源代码中的文件receiver.conf copy过来设置tennul与collector一致启动一直停留在 [16:38:12 CST 2019/08/06] [INFO] (main.startup:93) receiver is starting... 新进入kafka中的数据并没有任何不同

zlklink commented 5 years ago

到底如何才能从kafka得到正确的数据呢

vinllen commented 5 years ago

那应该是配置的问题了，或者kafka连接的问题

vinllen commented 5 years ago

贴下配置文件

zlklink commented 5 years ago

global log level. lower level message

will be filter

log_level = info

log file name. use logs/ prefix folder path

log_file = reeciver.log

log buffer or unbuffer. If set true, logs may not be print when exit. If

set false, performance will be decreased

log_buffer = true

profiling on net/http/profile

system_profile = 9500

tunnel pipeline type. now we support rpc,tcp,file,mock,kafka

tunnel = kafka

tunnel target resource url

for rpc. this is receiver socket address

for tcp. this is receiver socket address

for file. this is the file path, for instance "data"

for mock. this is useless. mongoshake will generate random data including "i", "d", "u", "n"

for kafka. this is the topic and brokers address which split by comma, for

instance: topic@brokers1,brokers2, default topic is "mongoshake"

tunnel.address = mongoshake@172.19.25.6:9092,172.19.25.7:9092,172.19.25.5:9092

replayer worker concurrency. must equal to the collector worker number

replayer = 8

zlklink commented 5 years ago

仅仅修改了tunnel 与 tunnel.address 与collector中的一致。 此外 kafka集群目前正常工作，能捕获mongodb的oplog增量变化，看起来就是receiver有问题

zlklink commented 5 years ago

receiver.conf与collector.conf中的tunnel 与 tunnel.address是不是一致的？

vinllen commented 5 years ago

是一致的，你先确定下kafka中能消费到数据吗？搞个client读一下是能读到的？

zlklink commented 5 years ago

确实是可以消费的，数据也是mongodb中修改的增量数据，只不过包含了你们的控制信息

vinllen commented 5 years ago

打一下进程堆栈看下： curl http://127.0.0.1:9500/debug/pprof/goroutine?debug=2

zlklink commented 5 years ago 

goroutine 28386 [running]:
runtime/pprof.writeGoroutineStacks(0x955980, 0xc4201e61c0, 0x411cd9, 0xc420292330)
        /data/u4/boyi.gw/tools/go1.10.3/src/runtime/pprof/pprof.go:650 +0xa7
runtime/pprof.writeGoroutine(0x955980, 0xc4201e61c0, 0x2, 0xc420042800, 0x955780)
        /data/u4/boyi.gw/tools/go1.10.3/src/runtime/pprof/pprof.go:639 +0x44
runtime/pprof.(*Profile).WriteTo(0xb8a8c0, 0x955980, 0xc4201e61c0, 0x2, 0xc4201e61c0, 0x8fc193)
        /data/u4/boyi.gw/tools/go1.10.3/src/runtime/pprof/pprof.go:310 +0x3e4
net/http/pprof.handler.ServeHTTP(0xc4202724c1, 0x9, 0x9595a0, 0xc4201e61c0, 0xc4200bac00)
        /data/u4/boyi.gw/tools/go1.10.3/src/net/http/pprof/pprof.go:243 +0x20d
net/http/pprof.Index(0x9595a0, 0xc4201e61c0, 0xc4200bac00)
        /data/u4/boyi.gw/tools/go1.10.3/src/net/http/pprof/pprof.go:254 +0x1d1
net/http.HandlerFunc.ServeHTTP(0x91a1c8, 0x9595a0, 0xc4201e61c0, 0xc4200bac00)
        /data/u4/boyi.gw/tools/go1.10.3/src/net/http/server.go:1947 +0x44
net/http.(*ServeMux).ServeHTTP(0xb98e00, 0x9595a0, 0xc4201e61c0, 0xc4200bac00)
        /data/u4/boyi.gw/tools/go1.10.3/src/net/http/server.go:2337 +0x130
net/http.serverHandler.ServeHTTP(0xc4200872b0, 0x9595a0, 0xc4201e61c0, 0xc4200bac00)
        /data/u4/boyi.gw/tools/go1.10.3/src/net/http/server.go:2694 +0xbc
net/http.(*conn).serve(0xc4201b8000, 0x959c60, 0xc420064300)
        /data/u4/boyi.gw/tools/go1.10.3/src/net/http/server.go:1830 +0x651
created by net/http.(*Server).Serve
        /data/u4/boyi.gw/tools/go1.10.3/src/net/http/server.go:2795 +0x27b

goroutine 1 [select (no cases), 188 minutes]:
main.main()
        /home/zhuzhao.cx/mongo-shake/src/mongoshake/receiver/main/receiver.go:62 +0x592

goroutine 5 [chan receive, 188 minutes]:
vendor/github.com/vinllen/log4go.(*ConsoleLogWriter).run(0xc42000c100, 0x955a80, 0xc42000e018)
        /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/log4go/termlog.go:33 +0x10c
created by vendor/github.com/vinllen/log4go.NewConsoleLogWriter
        /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/log4go/termlog.go:26 +0xbf

goroutine 6 [select, 1 minutes]:
vendor/github.com/vinllen/mgo.newcoarseTimeProvider.func1(0x17d7840, 0xc42001e960)
        /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/coarse_time.go:51 +0x157
created by vendor/github.com/vinllen/mgo.newcoarseTimeProvider
        /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/coarse_time.go:48 +0x107

goroutine 7 [syscall, 188 minutes]:
os/signal.signal_recv(0x0)
        /data/u4/boyi.gw/tools/go1.10.3/src/runtime/sigqueue.go:139 +0xa6
os/signal.loop()
        /data/u4/boyi.gw/tools/go1.10.3/src/os/signal/signal_unix.go:22 +0x22
created by os/signal.init.0
        /data/u4/boyi.gw/tools/go1.10.3/src/os/signal/signal_unix.go:28 +0x41

goroutine 20 [chan receive, 1 minutes]:
vendor/github.com/Shopify/sarama.(*Broker).responseReceiver(0xc4201de2c0)
        /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/Shopify/sarama/broker.go:682 +0xf7
vendor/github.com/Shopify/sarama.(*Broker).(vendor/github.com/Shopify/sarama.responseReceiver)-fm()
        /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/Shopify/sarama/broker.go:149 +0x2a
vendor/github.com/Shopify/sarama.withRecover(0xc42018c190)
        /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/Shopify/sarama/utils.go:45 +0x43
created by vendor/github.com/Shopify/sarama.(*Broker).Open.func1
        /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/Shopify/sarama/broker.go:149 +0x878

goroutine 36 [chan receive, 188 minutes]:
vendor/github.com/vinllen/log4go.(*ConsoleLogWriter).run(0xc42019ad00, 0x955a80, 0xc42000e018)
        /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/log4go/termlog.go:33 +0x10c
created by vendor/github.com/vinllen/log4go.NewConsoleLogWriter
        /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/log4go/termlog.go:26 +0xbf

goroutine 37 [select, 188 minutes]:
vendor/github.com/vinllen/log4go.NewFileLogWriter.func1(0xc4201b8500)
        /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/log4go/filelog.go:87 +0x136
created by vendor/github.com/vinllen/log4go.NewFileLogWriter
        /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/log4go/filelog.go:78 +0x23d

goroutine 38 [IO wait, 1 minutes]:
internal/poll.runtime_pollWait(0x7fccf2dfff00, 0x72, 0x0)
        /data/u4/boyi.gw/tools/go1.10.3/src/runtime/netpoll.go:173 +0x57
internal/poll.(*pollDesc).wait(0xc4200c2118, 0x72, 0xc42006fa00, 0x0, 0x0)
        /data/u4/boyi.gw/tools/go1.10.3/src/internal/poll/fd_poll_runtime.go:85 +0x9b
internal/poll.(*pollDesc).waitRead(0xc4200c2118, 0xffffffffffffff00, 0x0, 0x0)
        /data/u4/boyi.gw/tools/go1.10.3/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(*FD).Accept(0xc4200c2100, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
        /data/u4/boyi.gw/tools/go1.10.3/src/internal/poll/fd_unix.go:372 +0x1a8
net.(*netFD).accept(0xc4200c2100, 0xc4201b8080, 0xc420051db8, 0x402a68)
        /data/u4/boyi.gw/tools/go1.10.3/src/net/fd_unix.go:238 +0x42
net.(*TCPListener).accept(0xc42000e058, 0xc420051de8, 0x4019c7, 0xc4201b8080)
        /data/u4/boyi.gw/tools/go1.10.3/src/net/tcpsock_posix.go:136 +0x2e
net.(*TCPListener).AcceptTCP(0xc42000e058, 0xc420051e30, 0xc420051e38, 0x18)
        /data/u4/boyi.gw/tools/go1.10.3/src/net/tcpsock.go:246 +0x49
net/http.tcpKeepAliveListener.Accept(0xc42000e058, 0x919fc8, 0xc4201b8000, 0x959d20, 0xc42001f7d0)
        /data/u4/boyi.gw/tools/go1.10.3/src/net/http/server.go:3216 +0x2f
net/http.(*Server).Serve(0xc4200872b0, 0x959920, 0xc42000e058, 0x0, 0x0)
        /data/u4/boyi.gw/tools/go1.10.3/src/net/http/server.go:2770 +0x1a5
net/http.(*Server).ListenAndServe(0xc4200872b0, 0xc4200872b0, 0xc420170000)
        /data/u4/boyi.gw/tools/go1.10.3/src/net/http/server.go:2711 +0xa9
net/http.ListenAndServe(0xc4200245a8, 0x5, 0x0, 0x0, 0x1, 0xc4200245a8)
        /data/u4/boyi.gw/tools/go1.10.3/src/net/http/server.go:2969 +0x7a
vendor/github.com/gugemichael/nimo4go.Profiling.func1()
        /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/gugemichael/nimo4go/profile.go:22 +0xb5
vendor/github.com/gugemichael/nimo4go.GoRoutine.func1(0xc42025a2b0)
        /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/gugemichael/nimo4go/runtime.go:12 +0x25
created by vendor/github.com/gugemichael/nimo4go.GoRoutine
        /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/gugemichael/nimo4go/runtime.go:11 +0x3f

goroutine 39 [select, 188 minutes, locked to thread]:
runtime.gopark(0x91a5c0, 0x0, 0x8f8306, 0x6, 0x18, 0x1)
        /data/u4/boyi.gw/tools/go1.10.3/src/runtime/proc.go:291 +0x11a
runtime.selectgo(0xc4201ac750, 0xc4201a04e0)
        /data/u4/boyi.gw/tools/go1.10.3/src/runtime/select.go:392 +0xe50
runtime.ensureSigM.func1()
        /data/u4/boyi.gw/tools/go1.10.3/src/runtime/signal_unix.go:549 +0x1f4
runtime.goexit()
        /data/u4/boyi.gw/tools/go1.10.3/src/runtime/asm_amd64.s:2361 +0x1

goroutine 40 [chan receive, 188 minutes]:
vendor/github.com/gugemichael/nimo4go.RegisterSignalForProfiling.func1(0xc4201a0420, 0xc42025a2a6)
        /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/gugemichael/nimo4go/profile.go:46 +0x63
created by vendor/github.com/gugemichael/nimo4go.RegisterSignalForProfiling
        /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/gugemichael/nimo4go/profile.go:44 +0xb6

goroutine 41 [chan receive, 188 minutes]:
vendor/github.com/gugemichael/nimo4go.RegisterSignalForPrintStack.func1(0xc4201a05a0, 0x919dc0)
        /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/gugemichael/nimo4go/profile.go:31 +0xcd
created by vendor/github.com/gugemichael/nimo4go.RegisterSignalForPrintStack
        /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/gugemichael/nimo4go/profile.go:30 +0xb1

goroutine 42 [chan receive, 188 minutes]:
mongoshake/receiver.(*ExampleReplayer).handler(0xc420253440)
        /home/zhuzhao.cx/mongo-shake/src/mongoshake/receiver/replayer.go:113 +0x6f
created by mongoshake/receiver.NewExampleReplayer
        /home/zhuzhao.cx/mongo-shake/src/mongoshake/receiver/replayer.go:42 +0xf3

goroutine 43 [chan receive, 188 minutes]:
mongoshake/receiver.(*ExampleReplayer).handler(0xc420253470)
        /home/zhuzhao.cx/mongo-shake/src/mongoshake/receiver/replayer.go:113 +0x6f
created by mongoshake/receiver.NewExampleReplayer
        /home/zhuzhao.cx/mongo-shake/src/mongoshake/receiver/replayer.go:42 +0xf3

goroutine 44 [chan receive, 188 minutes]:
mongoshake/receiver.(*ExampleReplayer).handler(0xc4202534a0)
        /home/zhuzhao.cx/mongo-shake/src/mongoshake/receiver/replayer.go:113 +0x6f
created by mongoshake/receiver.NewExampleReplayer
        /home/zhuzhao.cx/mongo-shake/src/mongoshake/receiver/replayer.go:42 +0xf3

goroutine 45 [chan receive, 188 minutes]:
mongoshake/receiver.(*ExampleReplayer).handler(0xc4202534d0)
        /home/zhuzhao.cx/mongo-shake/src/mongoshake/receiver/replayer.go:113 +0x6f
created by mongoshake/receiver.NewExampleReplayer
        /home/zhuzhao.cx/mongo-shake/src/mongoshake/receiver/replayer.go:42 +0xf3

goroutine 46 [chan receive, 188 minutes]:
mongoshake/receiver.(*ExampleReplayer).handler(0xc420253500)
        /home/zhuzhao.cx/mongo-shake/src/mongoshake/receiver/replayer.go:113 +0x6f
created by mongoshake/receiver.NewExampleReplayer
        /home/zhuzhao.cx/mongo-shake/src/mongoshake/receiver/replayer.go:42 +0xf3

goroutine 47 [chan receive, 188 minutes]:
mongoshake/receiver.(*ExampleReplayer).handler(0xc420253530)
        /home/zhuzhao.cx/mongo-shake/src/mongoshake/receiver/replayer.go:113 +0x6f
created by mongoshake/receiver.NewExampleReplayer
        /home/zhuzhao.cx/mongo-shake/src/mongoshake/receiver/replayer.go:42 +0xf3

goroutine 48 [chan receive, 188 minutes]:
mongoshake/receiver.(*ExampleReplayer).handler(0xc420253560)
        /home/zhuzhao.cx/mongo-shake/src/mongoshake/receiver/replayer.go:113 +0x6f
created by mongoshake/receiver.NewExampleReplayer
        /home/zhuzhao.cx/mongo-shake/src/mongoshake/receiver/replayer.go:42 +0xf3

goroutine 49 [chan receive, 188 minutes]:
mongoshake/receiver.(*ExampleReplayer).handler(0xc420253590)
        /home/zhuzhao.cx/mongo-shake/src/mongoshake/receiver/replayer.go:113 +0x6f
created by mongoshake/receiver.NewExampleReplayer
        /home/zhuzhao.cx/mongo-shake/src/mongoshake/receiver/replayer.go:42 +0xf3

goroutine 21 [select, 10 minutes]:
vendor/github.com/Shopify/sarama.(*client).backgroundMetadataUpdater(0xc4201d4120)
        /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/Shopify/sarama/client.go:633 +0x156
vendor/github.com/Shopify/sarama.(*client).(vendor/github.com/Shopify/sarama.backgroundMetadataUpdater)-fm()
        /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/Shopify/sarama/client.go:163 +0x2a
vendor/github.com/Shopify/sarama.withRecover(0xc42018c200)
        /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/Shopify/sarama/utils.go:45 +0x43
created by vendor/github.com/Shopify/sarama.NewClient
        /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/Shopify/sarama/client.go:163 +0x52e

goroutine 19 [chan receive, 1 minutes]:
vendor/github.com/rcrowley/go-metrics.(*meterArbiter).tick(0xb99040)
        /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/rcrowley/go-metrics/meter.go:245 +0x31
created by vendor/github.com/rcrowley/go-metrics.NewMeter
        /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/rcrowley/go-metrics/meter.go:46 +0x10b

goroutine 24 [select, 1 minutes]:
vendor/github.com/Shopify/sarama.(*partitionConsumer).dispatcher(0xc4202d2000)
        /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/Shopify/sarama/consumer.go:337 +0x11e
vendor/github.com/Shopify/sarama.(*partitionConsumer).(vendor/github.com/Shopify/sarama.dispatcher)-fm()
        /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/Shopify/sarama/consumer.go:163 +0x2a
vendor/github.com/Shopify/sarama.withRecover(0xc42018c290)
        /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/Shopify/sarama/utils.go:45 +0x43
created by vendor/github.com/Shopify/sarama.(*consumer).ConsumePartition
        /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/Shopify/sarama/consumer.go:163 +0x2d7

goroutine 25 [chan receive, 188 minutes]:
vendor/github.com/Shopify/sarama.(*partitionConsumer).responseFeeder(0xc4202d2000)
        /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/Shopify/sarama/consumer.go:451 +0x2a0
vendor/github.com/Shopify/sarama.(*partitionConsumer).(vendor/github.com/Shopify/sarama.responseFeeder)-fm()
        /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/Shopify/sarama/consumer.go:164 +0x2a
vendor/github.com/Shopify/sarama.withRecover(0xc42018c2a0)
        /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/Shopify/sarama/utils.go:45 +0x43
created by vendor/github.com/Shopify/sarama.(*consumer).ConsumePartition
        /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/Shopify/sarama/consumer.go:164 +0x32e

goroutine 67 [chan receive, 188 minutes]:
mongoshake/tunnel/kafka.(*Reader).send(0xc42028c140)
        /home/zhuzhao.cx/mongo-shake/src/mongoshake/tunnel/kafka/reader.go:53 +0x86
created by mongoshake/tunnel/kafka.NewReader
        /home/zhuzhao.cx/mongo-shake/src/mongoshake/tunnel/kafka/reader.go:44 +0x20b

goroutine 68 [chan receive, 188 minutes]:
mongoshake/tunnel.(*KafkaReader).replay(0xc420253410)
        /home/zhuzhao.cx/mongo-shake/src/mongoshake/tunnel/kafka_reader.go:40 +0x6d3
created by mongoshake/tunnel.(*KafkaReader).Link
        /home/zhuzhao.cx/mongo-shake/src/mongoshake/tunnel/kafka_reader.go:27 +0x18e

goroutine 28387 [IO wait, 1 minutes]:
internal/poll.runtime_pollWait(0x7fccf2dffd60, 0x72, 0xc420041e58)
        /data/u4/boyi.gw/tools/go1.10.3/src/runtime/netpoll.go:173 +0x57
internal/poll.(*pollDesc).wait(0xc42019f218, 0x72, 0xffffffffffffff00, 0x956b40, 0xb5a688)
        /data/u4/boyi.gw/tools/go1.10.3/src/internal/poll/fd_poll_runtime.go:85 +0x9b
internal/poll.(*pollDesc).waitRead(0xc42019f218, 0xc420292100, 0x1, 0x1)
        /data/u4/boyi.gw/tools/go1.10.3/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(*FD).Read(0xc42019f200, 0xc420292191, 0x1, 0x1, 0x0, 0x0, 0x0)
        /data/u4/boyi.gw/tools/go1.10.3/src/internal/poll/fd_unix.go:157 +0x17d
net.(*netFD).Read(0xc42019f200, 0xc420292191, 0x1, 0x1, 0xc420372390, 0xb99380, 0x160d)
        /data/u4/boyi.gw/tools/go1.10.3/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc4200e21a0, 0xc420292191, 0x1, 0x1, 0x0, 0x0, 0x0)
        /data/u4/boyi.gw/tools/go1.10.3/src/net/net.go:176 +0x6a
net/http.(*connReader).backgroundRead(0xc420292180)
        /data/u4/boyi.gw/tools/go1.10.3/src/net/http/server.go:668 +0x5a
created by net/http.(*connReader).startBackgroundRead
        /data/u4/boyi.gw/tools/go1.10.3/src/net/http/server.go:664 +0xce

goroutine 28368 [chan receive, 1 minutes]:
vendor/github.com/Shopify/sarama.(*brokerConsumer).abort(0xc42024f680, 0x956c00, 0xc420324bf0)
        /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/Shopify/sarama/consumer.go:775 +0x201
vendor/github.com/Shopify/sarama.(*brokerConsumer).subscriptionConsumer(0xc42024f680)
        /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/Shopify/sarama/consumer.go:696 +0x342
vendor/github.com/Shopify/sarama.(*brokerConsumer).(vendor/github.com/Shopify/sarama.subscriptionConsumer)-fm()
        /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/Shopify/sarama/consumer.go:633 +0x2a
vendor/github.com/Shopify/sarama.withRecover(0xc420324a30)
        /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/Shopify/sarama/utils.go:45 +0x43
created by vendor/github.com/Shopify/sarama.(*consumer).newBrokerConsumer
        /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/Shopify/sarama/consumer.go:633 +0x1bb

goroutine 28367 [select, 1 minutes]:
vendor/github.com/Shopify/sarama.(*brokerConsumer).subscriptionManager(0xc42024f680)
        /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/Shopify/sarama/consumer.go:659 +0x3de
vendor/github.com/Shopify/sarama.(*brokerConsumer).(vendor/github.com/Shopify/sarama.subscriptionManager)-fm()
        /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/Shopify/sarama/consumer.go:632 +0x2a
vendor/github.com/Shopify/sarama.withRecover(0xc420324a20)
        /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/Shopify/sarama/utils.go:45 +0x43
created by vendor/github.com/Shopify/sarama.(*consumer).newBrokerConsumer
        /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/Shopify/sarama/consumer.go:632 +0x168```
vinllen commented 5 years ago

看堆栈是kafka客户端没有读到消息：

goroutine 67 [chan receive, 188 minutes]:
mongoshake/tunnel/kafka.(*Reader).send(0xc42028c140)
        /home/zhuzhao.cx/mongo-shake/src/mongoshake/tunnel/kafka/reader.go:53 +0x86
created by mongoshake/tunnel/kafka.NewReader
        /home/zhuzhao.cx/mongo-shake/src/mongoshake/tunnel/kafka/reader.go:44 +0x20b

goroutine 68 [chan receive, 188 minutes]:
mongoshake/tunnel.(*KafkaReader).replay(0xc420253410)
        /home/zhuzhao.cx/mongo-shake/src/mongoshake/tunnel/kafka_reader.go:40 +0x6d3
created by mongoshake/tunnel.(*KafkaReader).Link
        /home/zhuzhao.cx/mongo-shake/src/mongoshake/tunnel/kafka_reader.go:27 +0x18e

你再看看kafka的topic，parition都是ok的吗？ 再贴一下collector的配置文件。

zlklink commented 5 years ago

这样的，我用./kafka-console-consumer.sh --bootstrap-server 172.19.25.6:9092 --topic mongoshake --from-beginning 命令是能消费到mongodb的变化数据 以下是collector的配置文件

# 内部发送的worker数目，如果机器性能足够，可以提高worker个数。
worker = 8
# memory queue configuration, plz visit FAQ document to see more details.
# do not modify these variables if the performance and resource usage can
# meet your needs.
# 内部队列的配置参数，如果目前性能足够不建议修改，详细信息参考FAQ。
worker.batch_queue_size = 64
adaptive.batching_max_size = 16384
fetcher.buffer_capacity = 256

# batched oplogs have block level checksum value using 
# crc32 algorithm. and compressor for compressing content
# of oplog entry. 
# supported compressor are : gzip,zlib,deflate
# Do not enable this option when tunnel type is "direct"
# 是否启用发送，非direct模式发送可以选择压缩以减少网络带宽消耗。
worker.oplog_compressor = none

# tunnel pipeline type. now we support rpc,file,kafka,mock,direct
# 通道模式。
tunnel = kafka
# tunnel target resource url
# for rpc. this is remote receiver socket address
# for tcp. this is remote receiver socket address
# for file. this is the file path, for instance "data"
# for kafka. this is the topic and brokers address which split by comma, for
# instance: topic@brokers1,brokers2, default topic is "mongoshake"
# for mock. this is uesless
# for direct. this is target mongodb address which format is the same as `mongo_urls`. If
# the target is sharding, this should be the mongos address.
# direct模式用于直接写入MongoDB，其余模式用于一些分析，或者远距离传输场景，
# 注意，如果是非direct模式，需要通过receiver进行解析，具体参考FAQ文档。
# 此处配置通道的地址，格式与mongo_urls对齐。
tunnel.address = mongoshake@172.19.25.6:9092,172.19.25.7:9092,172.19.25.5:9092

# collector context storage mainly including store checkpoint.
# checkpoint存储信息，checkpoint本身是一个64位的时间戳表示本次开始拉取的地址。
# type include : database, api
# for api storage, address is http url
# for database storage, address is collection name while db name is "mongoshake" by default.
# checkpoint存储的地址，database表示存储到MongoDB中，api表示提供http的接口写入checkpoint。
context.storage = database
# context.storage.url is only used in to mark the checkpoint store database.
# If the source mongodb type is sharding, the address should be config server when MongoShake's
# version >= 1.5, otherwise, this is replicaSet address.
# When source mongodb type is replicaSet, checkpoint will write into source mongodb as default
# if `context.storage.url` is not set, otherwise, the checkpoint will be written into this
# mongodb. E.g., mongodb://127.0.0.1:20070
# checkpoint的具体写入的MongoDB地址，不配置对于副本集将写入源库(db=mongoshake)，对于分片集
# 将写入config-server（db=admin）
context.storage.url =
# checkpoint collection's name.
# checkpoint存储的表的名字，如果启动多条mongoshake拉取同一个源可以修改这个表名，
# 当然也可以修改上面的数据库名。
context.address = ckpt_default
# real checkpoint: the fetching oplog position.
# pay attention: this is UTC time which is 8 hours latter than CST time. this
# variable will only be used when checkpoint is not exist.
# 本次开始拉取的位置，如果checkpoint已经存在（参加上述存储位置）则该参数无效，
# 如果需要强制该位置开始拉取，需要先删除原来的checkpoint，详见FAQ。
context.start_position = 2000-01-01T00:00:01Z

# high availabity option.
# enable master election if set true. only one mongoshake can become master
# and do sync, the others will wait and at most one of them become master once 
# previous master die. The master information stores in the `mongoshake` db in the source 
# database by default.
# This option is useless when there is only one mongoshake running.
# 如果开启主备mongoshake拉取同一个源端，此参数需要开启。
master_quorum = false

# transform from source db or collection namespace to dest db or collection namespace.
# at most one of these two parameters can be given.
# transform: fromDbName1.fromCollectionName1:toDbName1.toCollectionName1;fromDbName2:toDbName2
# 转换命名空间，比如a.b同步后变成c.d，谨慎建议开启，比较耗性能。
transform.namespace =
# if use dbref in document rename, need to set it true, but it will decrease performance of replication
# 如果有dbref操作，这个需要置true，谨慎建议开启，比较耗性能。
dbref = false

# ----------------------splitter----------------------
# if tunnel type is direct, all the below variable should be set
# 写入MongoDB的参数配置。

# only transfer oplog commands for syncing. represent
# by oplog.op are "i","d","u".
# DDL will be transferred if disable like create index, drop databse,
# transaction in mongodb 4.0.
# 是否需要开启DDL同步，false表示开启，源是sharding暂时不支持开启。
# 如果目的端是sharding，暂时不支持applyOps命令，包括事务。
replayer.dml_only = true

# executors in single worker. do not modify it.
# 写入的线程个数，目前开源模式下请不要修改。
replayer.executor = 1

# oplog changes to Insert while Update found non-exist (_id or unique-index)
# 是否将update语句修改为insert语句，如果_id不存在在目的库。
replayer.executor.upsert = false
# oplog changes to Update while Insert found duplicated key (_id or unique-index)
# 是否将insert语句修改为update语句，如果_id存在在目的库。
replayer.executor.insert_on_dup_update = false
# db. write duplicated logs to mongoshake_conflict
# sdk. write duplicated logs to sdk.
# 如果写入存在冲突，记录冲突的文档。
replayer.conflict_write_to = none

# replayer duration mode. drop oplogs and take
# no any action(only for debugging enviroment) if 
# set to false. otherwise write to ${mongo_url} instance
# false表示取消写入，只用于调试。
replayer.durable = true

# ----------------------splitter----------------------
# full synchronization configuration
# 全量同步参数配置，详见wiki，如果目的端MongoDB压力过大，可以适当调低下列参数。

# the number of collection concurrence
# 并发最大拉取的表个数。
replayer.collection_parallel = 6

# the number of document concurrence in a collection concurrence
# 同一个表内部并发的线程数。
replayer.document_parallel = 8

# number of documents in a batch insert in a document concurrence
# 目的端写入的batch大小。
replayer.document_batch_size = 256

# drop the same name of collection in dest mongodb in full synchronization
# 同步时如果目的库存在，是否先删除目的库再进行同步。
replayer.collection_drop = false
vinllen commented 5 years ago

配置侧没问题，这个问题还是出在没从kafka中读到数据，driver的读取是oldest模式，你对比看看。

roggen-yang commented 5 years ago

我现在也遇到同样的问题了，有什么办法解么

zlklink commented 5 years ago

这是FAQ中的描述：The dataflow is mongoshake(collector)=>tunnel=>receiver=>user's platform 对于kafka方式，在我理解看来tunnel就是指的是kafka. receiver是处理程序，这部分会读取tunnel中的数据，推到user's platform。这个user's platform是什么？？？没发现任何可以配置这个user's platform的地方，还是又回写进之前的tunnel了？

vinllen commented 5 years ago

user platform是说的后面用户自己的处理平台，如果有的话。receiver解析后数据怎么用我们不管，user platform只是举个例子

On Wed, Aug 7, 2019 at 09:36 zlklink notifications@github.com wrote:

这是FAQ中的描述：The dataflow is mongoshake(collector)=>tunnel=>receiver=>user's platform 对于kafka方式，在我理解看来tunnel就是指的是kafka. receiver是处理程序，这部分会读取tunnel中的数据，推到user's platform。这个user's platform是什么？？？没发现任何可以配置这个user's platform的地方，还是又回写进之前的tunnel了？

— You are receiving this because you commented.

Reply to this email directly, view it on GitHub https://github.com/alibaba/MongoShake/issues/212?email_source=notifications&email_token=ABVJ5BX7HANE2IH7IJPWTIDQDIRK3A5CNFSM4IJUGE5KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3W5TQA#issuecomment-518904256, or mute the thread https://github.com/notifications/unsubscribe-auth/ABVJ5BR2HBNI4XCJ73LBYPDQDIRK3ANCNFSM4IJUGE5A .

-- Best Regards, Vinllen

zlklink commented 5 years ago

FAQ:这是因为写入的数据有控制信息在里面，需要用receiver进行接收，剥离控制信息，然后再进行后续的对接. receiver接受剥离控制信息，处理后的数据放在哪？？？

zlklink commented 5 years ago

感觉我的receiver一直没启动，大神能帮我看一下么：

这个是配置信息：

global log level. lower level message

will be filter

log_level = info

log file name. use logs/ prefix folder path

log_file = reeciver.log

log buffer or unbuffer. If set true, logs may not be print when exit. If

set false, performance will be decreased

log_buffer = true

profiling on net/http/profile

system_profile = 9500

tunnel pipeline type. now we support rpc,tcp,file,mock,kafka

tunnel = kafka

tunnel target resource url

for rpc. this is receiver socket address

for tcp. this is receiver socket address

for file. this is the file path, for instance "data"

for mock. this is useless. mongoshake will generate random data including "i", "d", "u", "n"

for kafka. this is the topic and brokers address which split by comma, for

instance: topic@brokers1,brokers2, default topic is "mongoshake"

tunnel.address = mongoshake@172.19.25.6:9092,172.19.25.7:9092,172.19.25.5:9092

replayer worker concurrency. must equal to the collector worker number

replayer = 8

这个是启动日志一直就卡在这边：

[10:18:09 CST 2019/08/07] [INFO] (mongoshake/receiver.NewExampleReplayer:37) ExampleReplayer start. pending queue capacity 256 [10:18:09 CST 2019/08/07] [INFO] (mongoshake/receiver.NewExampleReplayer:37) ExampleReplayer start. pending queue capacity 256 [10:18:09 CST 2019/08/07] [INFO] (mongoshake/receiver.NewExampleReplayer:37) ExampleReplayer start. pending queue capacity 256 [10:18:09 CST 2019/08/07] [INFO] (mongoshake/receiver.NewExampleReplayer:37) ExampleReplayer start. pending queue capacity 256 [10:18:09 CST 2019/08/07] [INFO] (mongoshake/receiver.NewExampleReplayer:37) ExampleReplayer start. pending queue capacity 256 [10:18:09 CST 2019/08/07] [INFO] (mongoshake/receiver.NewExampleReplayer:37) ExampleReplayer start. pending queue capacity 256 [10:18:09 CST 2019/08/07] [INFO] (mongoshake/receiver.NewExampleReplayer:37) ExampleReplayer start. pending queue capacity 256 [10:18:09 CST 2019/08/07] [INFO] (mongoshake/receiver.NewExampleReplayer:37) ExampleReplayer start. pending queue capacity 256 [10:18:09 CST 2019/08/07] [INFO] (main.startup:93) receiver is starting...

vinllen commented 5 years ago

receiver接受剥离控制信息，处理后的数据放在哪？？？

处理后的信息默认是打印到log里面，如果需要对接别的平台目前只能改源码。后续我们可能会提供别的比如http之类的接口进行对接，目前暂时没有人力搞这个。你们可以从log里面进行剥离。

读取不到这个问题不像是mongoshake的问题，大概率还是哪里配置不当，之前用户这么用也都是OK的。当然也有可能是driver的问题，目前看堆栈，shake用sarama的驱动的确是没拉到数据。

vinllen commented 5 years ago

@zlklink 问题是否已经解决，没有的话微信私聊我一起看下好了。

zlklink commented 5 years ago

因为时间的原因 ，这边迟迟搞不定，暂时放弃了使用该框架，改用了debezium。 但是依然会持续关注mongodb-shake。因为shake支持更多的target tunnel，而debezium只支持kafka，希望shake能做的更好

vinllen commented 5 years ago

@zlklink 感谢关注，这个问题我们会再看一下

yzhbest commented 5 years ago

最新的配置文件

global log level. lower level message

will be filter

log.level = info

log file name. use logs/ prefix folder path

log.file = reeciver.log

log buffer or unbuffer. If set true, logs may not be print when exit. If

set false, performance will be decreased

log.buffer = true

profiling on net/http/profile

system_profile = 9500

tunnel pipeline type. now we support rpc,tcp,file,mock,kafka

tunnel = kafka

tunnel target resource url

for rpc. this is receiver socket address

for tcp. this is receiver socket address

for file. this is the file path, for instance "data"

for mock. this is useless. mongoshake will generate random data including "i", "d", "u", "n"

for kafka. this is the topic and brokers address which split by comma, for

instance: topic@brokers1,brokers2, default topic is "mongoshake"

tunnel.address = mongoshake@10.0.0.35:9092

replayer worker concurrency. must equal to the collector worker number

replayer = 8

有部分下划线需要改为点

@vinllen ./receiver -conf=./receiver.conf -verbose 这样输出内容在控制台上面，我们如何调用呢？ 有通过api获取数据的方式？

[17:04:27 CST 2019/08/16] [INFO] (mongoshake/receiver.(ExampleReplayer).handler:130) &{6725671133426745374 d clue_dw.a [{_id ObjectIdHex("5d564f08a3980674c891e1ea")}] map[] map[] false map[] 138 0} [17:04:27 CST 2019/08/16] [INFO] (mongoshake/receiver.(ExampleReplayer).handler:130) &{6725671249390862337 i clue_dw.a [{_id ObjectIdHex("5d56625ef0094137c1f9697d")} {a 1}] map[] map[] false map[] 145 0} [17:04:27 CST 2019/08/16] [INFO] (mongoshake/receiver.(ExampleReplayer).handler:130) &{6725674208623329281 i clue_dw.a [{_id ObjectIdHex("5d56650ff0094137c1f96aa8")} {c 1}] map[] map[] false map[] 145 0} [17:05:00 CST 2019/08/16] [INFO] (mongoshake/receiver.(ExampleReplayer).handler:130) &{6725688141497237505 i clue_dw.a [{_id ObjectIdHex("5d5671bbf0094137c1f97041")} {yzh 1}] map[] map[] false map[] 147 0}

vinllen commented 5 years ago

@yzhbest 目前都是直接打印控制台的，没有提供api的方式，需要修改receiver的源码进行对接。 这里最好是提供一个api获取的方式，但是人力问题没有支持，你有精力可以进行api的支持，搞个类似sdk或者api方式，然后提个pull request。

liunianjie commented 5 years ago

感觉我的receiver一直没启动，大神能帮我看一下么：

这个是配置信息：

global log level. lower level message

will be filter

log_level = info

log file name. use logs/ prefix folder path

log_file = reeciver.log

log buffer or unbuffer. If set true, logs may not be print when exit. If

set false, performance will be decreased

log_buffer = true

profiling on net/http/profile

system_profile = 9500

tunnel pipeline type. now we support rpc,tcp,file,mock,kafka

tunnel = kafka

tunnel target resource url

for rpc. this is receiver socket address

for tcp. this is receiver socket address

for file. this is the file path, for instance "data"

for mock. this is useless. mongoshake will generate random data including "i", "d", "u", "n"

for kafka. this is the topic and brokers address which split by comma, for

instance: topic@brokers1,brokers2, default topic is "mongoshake"

tunnel.address = mongoshake@172.19.25.6:9092,172.19.25.7:9092,172.19.25.5:9092

replayer worker concurrency. must equal to the collector worker number

replayer = 8

这个是启动日志一直就卡在这边：

[10:18:09 CST 2019/08/07] [INFO] (mongoshake/receiver.NewExampleReplayer:37) ExampleReplayer start. pending queue capacity 256 [10:18:09 CST 2019/08/07] [INFO] (mongoshake/receiver.NewExampleReplayer:37) ExampleReplayer start. pending queue capacity 256 [10:18:09 CST 2019/08/07] [INFO] (mongoshake/receiver.NewExampleReplayer:37) ExampleReplayer start. pending queue capacity 256 [10:18:09 CST 2019/08/07] [INFO] (mongoshake/receiver.NewExampleReplayer:37) ExampleReplayer start. pending queue capacity 256 [10:18:09 CST 2019/08/07] [INFO] (mongoshake/receiver.NewExampleReplayer:37) ExampleReplayer start. pending queue capacity 256 [10:18:09 CST 2019/08/07] [INFO] (mongoshake/receiver.NewExampleReplayer:37) ExampleReplayer start. pending queue capacity 256 [10:18:09 CST 2019/08/07] [INFO] (mongoshake/receiver.NewExampleReplayer:37) ExampleReplayer start. pending queue capacity 256 [10:18:09 CST 2019/08/07] [INFO] (mongoshake/receiver.NewExampleReplayer:37) ExampleReplayer start. pending queue capacity 256 [10:18:09 CST 2019/08/07] [INFO] (main.startup:93) receiver is starting...

我这边也是遇到这种情况.现在有解决吗?

liunianjie commented 5 years ago

不用receiver的话,直接java消费kafka数据,怎么解析24字节头部后面的oplog数据(看起来不是json)?

vinllen commented 5 years ago

@liunianjie 打一下堆栈看一下： curl http://127.0.0.1:9500/debug/pprof/goroutine?debug=2 不用receiver无法解析

liunianjie commented 5 years ago

用java bson库是可以解析的

liunianjie commented 5 years ago

goroutine 118890 [running]: runtime/pprof.writeGoroutineStacks(0xa18cc0, 0xc4201fa000, 0x411cd9, 0xc4203599e0) /data/u4/boyi.gw/tools/go1.10.3/src/runtime/pprof/pprof.go:650 +0xa7 runtime/pprof.writeGoroutine(0xa18cc0, 0xc4201fa000, 0x2, 0xc4201a4400, 0xa18aa0) /data/u4/boyi.gw/tools/go1.10.3/src/runtime/pprof/pprof.go:639 +0x44 runtime/pprof.(Profile).WriteTo(0xc87ba0, 0xa18cc0, 0xc4201fa000, 0x2, 0xc4201fa000, 0x9b14bd) /data/u4/boyi.gw/tools/go1.10.3/src/runtime/pprof/pprof.go:310 +0x3e4 net/http/pprof.handler.ServeHTTP(0xc42294a041, 0x9, 0xa1cf20, 0xc4201fa000, 0xc4228ac300) /data/u4/boyi.gw/tools/go1.10.3/src/net/http/pprof/pprof.go:243 +0x20d net/http/pprof.Index(0xa1cf20, 0xc4201fa000, 0xc4228ac300) /data/u4/boyi.gw/tools/go1.10.3/src/net/http/pprof/pprof.go:254 +0x1d1 net/http.HandlerFunc.ServeHTTP(0x9d4140, 0xa1cf20, 0xc4201fa000, 0xc4228ac300) /data/u4/boyi.gw/tools/go1.10.3/src/net/http/server.go:1947 +0x44 net/http.(ServeMux).ServeHTTP(0xc962e0, 0xa1cf20, 0xc4201fa000, 0xc4228ac300) /data/u4/boyi.gw/tools/go1.10.3/src/net/http/server.go:2337 +0x130 net/http.serverHandler.ServeHTTP(0xc420273ad0, 0xa1cf20, 0xc4201fa000, 0xc4228ac300) /data/u4/boyi.gw/tools/go1.10.3/src/net/http/server.go:2694 +0xbc net/http.(conn).serve(0xc4201c2aa0, 0xa1d720, 0xc420312fc0) /data/u4/boyi.gw/tools/go1.10.3/src/net/http/server.go:1830 +0x651 created by net/http.(Server).Serve /data/u4/boyi.gw/tools/go1.10.3/src/net/http/server.go:2795 +0x27b

goroutine 1 [IO wait, 1343 minutes]: internal/poll.runtime_pollWait(0x7f41847f9610, 0x72, 0x0) /data/u4/boyi.gw/tools/go1.10.3/src/runtime/netpoll.go:173 +0x57 internal/poll.(pollDesc).wait(0xc4202b0398, 0x72, 0xc4200b8700, 0x0, 0x0) /data/u4/boyi.gw/tools/go1.10.3/src/internal/poll/fd_poll_runtime.go:85 +0x9b internal/poll.(pollDesc).waitRead(0xc4202b0398, 0xffffffffffffff00, 0x0, 0x0) /data/u4/boyi.gw/tools/go1.10.3/src/internal/poll/fd_poll_runtime.go:90 +0x3d internal/poll.(FD).Accept(0xc4202b0380, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /data/u4/boyi.gw/tools/go1.10.3/src/internal/poll/fd_unix.go:372 +0x1a8 net.(netFD).accept(0xc4202b0380, 0x7f4184845000, 0x0, 0x9d4670) /data/u4/boyi.gw/tools/go1.10.3/src/net/fd_unix.go:238 +0x42 net.(TCPListener).accept(0xc4202d00c0, 0xc420367aa0, 0x412528, 0x30) /data/u4/boyi.gw/tools/go1.10.3/src/net/tcpsock_posix.go:136 +0x2e net.(TCPListener).AcceptTCP(0xc4202d00c0, 0x95be40, 0xc42026a510, 0x8f7440) /data/u4/boyi.gw/tools/go1.10.3/src/net/tcpsock.go:246 +0x49 net/http.tcpKeepAliveListener.Accept(0xc4202d00c0, 0xc4200b0030, 0x8f7440, 0xc85430, 0x999380) /data/u4/boyi.gw/tools/go1.10.3/src/net/http/server.go:3216 +0x2f net/http.(Server).Serve(0xc4200aa820, 0xa1d320, 0xc4202d00c0, 0x0, 0x0) /data/u4/boyi.gw/tools/go1.10.3/src/net/http/server.go:2770 +0x1a5 net/http.(Server).ListenAndServe(0xc4200aa820, 0xc4200aa820, 0xc4202de0c0) /data/u4/boyi.gw/tools/go1.10.3/src/net/http/server.go:2711 +0xa9 net/http.ListenAndServe(0xc420022638, 0x5, 0x0, 0x0, 0x1, 0xc420022638) /data/u4/boyi.gw/tools/go1.10.3/src/net/http/server.go:2969 +0x7a vendor/github.com/gugemichael/nimo4go.(*HttpRestProvider).Listen(0xc4201b9250, 0x0, 0x0) /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/gugemichael/nimo4go/http_rest.go:46 +0x20e main.startup() /home/zhuzhao.cx/mongo-shake/src/mongoshake/collector/main/collector.go:110 +0x4f9 main.main() /home/zhuzhao.cx/mongo-shake/src/mongoshake/collector/main/collector.go:76 +0x62d

goroutine 19 [chan receive, 1343 minutes]: vendor/github.com/vinllen/log4go.(*ConsoleLogWriter).run(0xc4200ba0c0, 0xa18dc0, 0xc4200b4008) /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/log4go/termlog.go:33 +0x10c created by vendor/github.com/vinllen/log4go.NewConsoleLogWriter /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/log4go/termlog.go:26 +0xbf

goroutine 20 [select]: vendor/github.com/vinllen/mgo.newcoarseTimeProvider.func1(0x17d7840, 0xc4200a8900) /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/coarse_time.go:51 +0x157 created by vendor/github.com/vinllen/mgo.newcoarseTimeProvider /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/coarse_time.go:48 +0x107

goroutine 21 [syscall, 1343 minutes]: os/signal.signal_recv(0x0) /data/u4/boyi.gw/tools/go1.10.3/src/runtime/sigqueue.go:139 +0xa6 os/signal.loop() /data/u4/boyi.gw/tools/go1.10.3/src/os/signal/signal_unix.go:22 +0x22 created by os/signal.init.0 /data/u4/boyi.gw/tools/go1.10.3/src/os/signal/signal_unix.go:28 +0x41

goroutine 24 [IO wait]: internal/poll.runtime_pollWait(0x7f41847f9f00, 0x72, 0x0) /data/u4/boyi.gw/tools/go1.10.3/src/runtime/netpoll.go:173 +0x57 internal/poll.(pollDesc).wait(0xc4200f2318, 0x72, 0xc4201b0000, 0x0, 0x0) /data/u4/boyi.gw/tools/go1.10.3/src/internal/poll/fd_poll_runtime.go:85 +0x9b internal/poll.(pollDesc).waitRead(0xc4200f2318, 0xffffffffffffff00, 0x0, 0x0) /data/u4/boyi.gw/tools/go1.10.3/src/internal/poll/fd_poll_runtime.go:90 +0x3d internal/poll.(FD).Accept(0xc4200f2300, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /data/u4/boyi.gw/tools/go1.10.3/src/internal/poll/fd_unix.go:372 +0x1a8 net.(netFD).accept(0xc4200f2300, 0xc4201c2b20, 0xc42027edb8, 0x402a68) /data/u4/boyi.gw/tools/go1.10.3/src/net/fd_unix.go:238 +0x42 net.(TCPListener).accept(0xc4200b40a8, 0xc42027ede8, 0x4019c7, 0xc4201c2b20) /data/u4/boyi.gw/tools/go1.10.3/src/net/tcpsock_posix.go:136 +0x2e net.(TCPListener).AcceptTCP(0xc4200b40a8, 0xc42027ee30, 0xc42027ee38, 0x18) /data/u4/boyi.gw/tools/go1.10.3/src/net/tcpsock.go:246 +0x49 net/http.tcpKeepAliveListener.Accept(0xc4200b40a8, 0x9d3f20, 0xc4201c2aa0, 0xa1d7e0, 0xc42026ba40) /data/u4/boyi.gw/tools/go1.10.3/src/net/http/server.go:3216 +0x2f net/http.(Server).Serve(0xc420273ad0, 0xa1d320, 0xc4200b40a8, 0x0, 0x0) /data/u4/boyi.gw/tools/go1.10.3/src/net/http/server.go:2770 +0x1a5 net/http.(Server).ListenAndServe(0xc420273ad0, 0xc420273ad0, 0xc420198000) /data/u4/boyi.gw/tools/go1.10.3/src/net/http/server.go:2711 +0xa9 net/http.ListenAndServe(0xc420274270, 0x5, 0x0, 0x0, 0x1, 0xc420274270) /data/u4/boyi.gw/tools/go1.10.3/src/net/http/server.go:2969 +0x7a vendor/github.com/gugemichael/nimo4go.Profiling.func1() /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/gugemichael/nimo4go/profile.go:22 +0xb5 vendor/github.com/gugemichael/nimo4go.GoRoutine.func1(0xc420274240) /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/gugemichael/nimo4go/runtime.go:12 +0x25 created by vendor/github.com/gugemichael/nimo4go.GoRoutine /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/gugemichael/nimo4go/runtime.go:11 +0x3f

goroutine 23 [select]: vendor/github.com/vinllen/log4go.NewFileLogWriter.func1(0xc4200f0e60) /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/log4go/filelog.go:87 +0x136 created by vendor/github.com/vinllen/log4go.NewFileLogWriter /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/log4go/filelog.go:78 +0x23d

goroutine 25 [select, 1343 minutes, locked to thread]: runtime.gopark(0x9d4548, 0x0, 0x9ad30e, 0x6, 0x18, 0x1) /data/u4/boyi.gw/tools/go1.10.3/src/runtime/proc.go:291 +0x11a runtime.selectgo(0xc42004ef50, 0xc42009ac00) /data/u4/boyi.gw/tools/go1.10.3/src/runtime/select.go:392 +0xe50 runtime.ensureSigM.func1() /data/u4/boyi.gw/tools/go1.10.3/src/runtime/signal_unix.go:549 +0x1f4 runtime.goexit() /data/u4/boyi.gw/tools/go1.10.3/src/runtime/asm_amd64.s:2361 +0x1

goroutine 26 [chan receive, 1343 minutes]: vendor/github.com/gugemichael/nimo4go.RegisterSignalForProfiling.func1(0xc42009ab40, 0xc420274236) /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/gugemichael/nimo4go/profile.go:46 +0x63 created by vendor/github.com/gugemichael/nimo4go.RegisterSignalForProfiling /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/gugemichael/nimo4go/profile.go:44 +0xb6

goroutine 36 [chan receive, 1343 minutes]: vendor/github.com/gugemichael/nimo4go.RegisterSignalForPrintStack.func1(0xc42009acc0, 0x9d3ce0) /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/gugemichael/nimo4go/profile.go:31 +0xcd created by vendor/github.com/gugemichael/nimo4go.RegisterSignalForPrintStack /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/gugemichael/nimo4go/profile.go:30 +0xb1

goroutine 241 [IO wait]: internal/poll.runtime_pollWait(0x7f41847f9060, 0x72, 0xc42027bc88) /data/u4/boyi.gw/tools/go1.10.3/src/runtime/netpoll.go:173 +0x57 internal/poll.(pollDesc).wait(0xc4202f8298, 0x72, 0xffffffffffffff00, 0xa19f20, 0xc576a0) /data/u4/boyi.gw/tools/go1.10.3/src/internal/poll/fd_poll_runtime.go:85 +0x9b internal/poll.(pollDesc).waitRead(0xc4202f8298, 0xc42036a400, 0x10, 0x10) /data/u4/boyi.gw/tools/go1.10.3/src/internal/poll/fd_poll_runtime.go:90 +0x3d internal/poll.(FD).Read(0xc4202f8280, 0xc42036a440, 0x10, 0x10, 0x0, 0x0, 0x0) /data/u4/boyi.gw/tools/go1.10.3/src/internal/poll/fd_unix.go:157 +0x17d net.(netFD).Read(0xc4202f8280, 0xc42036a440, 0x10, 0x10, 0x0, 0x0, 0x0) /data/u4/boyi.gw/tools/go1.10.3/src/net/fd_unix.go:202 +0x4f net.(conn).Read(0xc4202d0070, 0xc42036a440, 0x10, 0x10, 0x0, 0x0, 0x0) /data/u4/boyi.gw/tools/go1.10.3/src/net/net.go:176 +0x6a vendor/github.com/vinllen/mgo.fill(0xa20200, 0xc4202d0070, 0xc42036a440, 0x10, 0x10, 0x0, 0x8e) /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/socket.go:601 +0x53 vendor/github.com/vinllen/mgo.(mongoSocket).readLoop(0xc4201eea20) /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/socket.go:619 +0x122 created by vendor/github.com/vinllen/mgo.newSocket /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/socket.go:211 +0x1f5

goroutine 13 [sleep]: time.Sleep(0x37e11d600) /data/u4/boyi.gw/tools/go1.10.3/src/runtime/time.go:102 +0x166 vendor/github.com/vinllen/mgo.(*mongoServer).pinger(0xc4202f03c0, 0xc420020501) /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/server.go:426 +0x4b3 created by vendor/github.com/vinllen/mgo.newServer /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/server.go:107 +0x186

goroutine 73 [select]: vendor/github.com/vinllen/mgo.(*mongoCluster).syncServersLoop(0xc42037e480) /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/cluster.go:431 +0x31a created by vendor/github.com/vinllen/mgo.newCluster /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/cluster.go:83 +0x1ae

goroutine 75 [sleep]: time.Sleep(0x37e11d600) /data/u4/boyi.gw/tools/go1.10.3/src/runtime/time.go:102 +0x166 vendor/github.com/vinllen/mgo.(*mongoServer).pinger(0xc4202bed20, 0xc420274e01) /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/server.go:426 +0x4b3 created by vendor/github.com/vinllen/mgo.newServer /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/server.go:107 +0x186

goroutine 77 [IO wait, 1343 minutes]: internal/poll.runtime_pollWait(0x7f41847f9e30, 0x72, 0xc420064c88) /data/u4/boyi.gw/tools/go1.10.3/src/runtime/netpoll.go:173 +0x57 internal/poll.(pollDesc).wait(0xc4200f2618, 0x72, 0xffffffffffffff00, 0xa19f20, 0xc576a0) /data/u4/boyi.gw/tools/go1.10.3/src/internal/poll/fd_poll_runtime.go:85 +0x9b internal/poll.(pollDesc).waitRead(0xc4200f2618, 0xc420274f00, 0x10, 0x10) /data/u4/boyi.gw/tools/go1.10.3/src/internal/poll/fd_poll_runtime.go:90 +0x3d internal/poll.(FD).Read(0xc4200f2600, 0xc420274fd0, 0x10, 0x10, 0x0, 0x0, 0x0) /data/u4/boyi.gw/tools/go1.10.3/src/internal/poll/fd_unix.go:157 +0x17d net.(netFD).Read(0xc4200f2600, 0xc420274fd0, 0x10, 0x10, 0x0, 0x0, 0x0) /data/u4/boyi.gw/tools/go1.10.3/src/net/fd_unix.go:202 +0x4f net.(conn).Read(0xc4200b4178, 0xc420274fd0, 0x10, 0x10, 0x0, 0x0, 0x0) /data/u4/boyi.gw/tools/go1.10.3/src/net/net.go:176 +0x6a vendor/github.com/vinllen/mgo.fill(0xa20200, 0xc4200b4178, 0xc420274fd0, 0x10, 0x10, 0x0, 0x10c) /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/socket.go:601 +0x53 vendor/github.com/vinllen/mgo.(mongoSocket).readLoop(0xc42037e5a0) /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/socket.go:619 +0x122 created by vendor/github.com/vinllen/mgo.newSocket /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/socket.go:211 +0x1f5

goroutine 103 [chan receive, 1343 minutes]: mongoshake/executor.(Executor).start(0xc4201b21e0) /home/zhuzhao.cx/mongo-shake/src/mongoshake/executor/executor.go:200 +0x10c created by mongoshake/executor.(BatchGroupExecutor).Start /home/zhuzhao.cx/mongo-shake/src/mongoshake/executor/executor.go:68 +0x9f

goroutine 101 [chan receive]: mongoshake/common.(ReplicationMetric).startup.func1(0xc4201fda00) /home/zhuzhao.cx/mongo-shake/src/mongoshake/common/metric.go:104 +0xaf created by mongoshake/common.(ReplicationMetric).startup /home/zhuzhao.cx/mongo-shake/src/mongoshake/common/metric.go:100 +0x97

goroutine 81 [select]: vendor/github.com/vinllen/mgo.(*mongoCluster).syncServersLoop(0xc42037c000) /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/cluster.go:431 +0x31a created by vendor/github.com/vinllen/mgo.newCluster /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/cluster.go:83 +0x1ae

goroutine 164 [chan receive, 1343 minutes]: mongoshake/executor.(Executor).start(0xc42036c230) /home/zhuzhao.cx/mongo-shake/src/mongoshake/executor/executor.go:200 +0x10c created by mongoshake/executor.(BatchGroupExecutor).Start /home/zhuzhao.cx/mongo-shake/src/mongoshake/executor/executor.go:68 +0x9f

goroutine 166 [select]: vendor/github.com/vinllen/mgo.(*mongoCluster).syncServersLoop(0xc4201ee480) /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/cluster.go:431 +0x31a created by vendor/github.com/vinllen/mgo.newCluster /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/cluster.go:83 +0x1ae

goroutine 165 [select]: mongoshake/collector.(Worker).findFirstAvailableBatch(0xc420326150, 0xa156c0, 0xc420377fa8, 0x2) /home/zhuzhao.cx/mongo-shake/src/mongoshake/collector/worker.go:102 +0xfb mongoshake/collector.(Worker).startWorker(0xc420326150) /home/zhuzhao.cx/mongo-shake/src/mongoshake/collector/worker.go:132 +0x129 created by mongoshake/collector.(*ReplicationCoordinator).startOplogReplication /home/zhuzhao.cx/mongo-shake/src/mongoshake/collector/replication.go:312 +0x1fe

goroutine 54 [chan receive, 1343 minutes]: mongoshake/executor.(Executor).start(0xc4200b60f0) /home/zhuzhao.cx/mongo-shake/src/mongoshake/executor/executor.go:200 +0x10c created by mongoshake/executor.(BatchGroupExecutor).Start /home/zhuzhao.cx/mongo-shake/src/mongoshake/executor/executor.go:68 +0x9f

goroutine 55 [select]: mongoshake/collector.(Worker).findFirstAvailableBatch(0xc4201cb880, 0xa156c0, 0xc420300fa8, 0x2) /home/zhuzhao.cx/mongo-shake/src/mongoshake/collector/worker.go:102 +0xfb mongoshake/collector.(Worker).startWorker(0xc4201cb880) /home/zhuzhao.cx/mongo-shake/src/mongoshake/collector/worker.go:132 +0x129 created by mongoshake/collector.(*ReplicationCoordinator).startOplogReplication /home/zhuzhao.cx/mongo-shake/src/mongoshake/collector/replication.go:312 +0x1fe

goroutine 56 [select]: vendor/github.com/vinllen/mgo.(*mongoCluster).syncServersLoop(0xc4201ee000) /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/cluster.go:431 +0x31a created by vendor/github.com/vinllen/mgo.newCluster /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/cluster.go:83 +0x1ae

goroutine 123 [chan receive, 1343 minutes]: mongoshake/executor.(Executor).start(0xc4201b2410) /home/zhuzhao.cx/mongo-shake/src/mongoshake/executor/executor.go:200 +0x10c created by mongoshake/executor.(BatchGroupExecutor).Start /home/zhuzhao.cx/mongo-shake/src/mongoshake/executor/executor.go:68 +0x9f

goroutine 104 [select]: mongoshake/collector.(Worker).findFirstAvailableBatch(0xc420172000, 0xa156c0, 0xc420375fa8, 0x2) /home/zhuzhao.cx/mongo-shake/src/mongoshake/collector/worker.go:102 +0xfb mongoshake/collector.(Worker).startWorker(0xc420172000) /home/zhuzhao.cx/mongo-shake/src/mongoshake/collector/worker.go:132 +0x129 created by mongoshake/collector.(*ReplicationCoordinator).startOplogReplication /home/zhuzhao.cx/mongo-shake/src/mongoshake/collector/replication.go:312 +0x1fe

goroutine 105 [select]: vendor/github.com/vinllen/mgo.(*mongoCluster).syncServersLoop(0xc4200d85a0) /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/cluster.go:431 +0x31a created by vendor/github.com/vinllen/mgo.newCluster /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/cluster.go:83 +0x1ae

goroutine 124 [select]: mongoshake/collector.(Worker).findFirstAvailableBatch(0xc420172070, 0xa156c0, 0xc42027cfa8, 0x2) /home/zhuzhao.cx/mongo-shake/src/mongoshake/collector/worker.go:102 +0xfb mongoshake/collector.(Worker).startWorker(0xc420172070) /home/zhuzhao.cx/mongo-shake/src/mongoshake/collector/worker.go:132 +0x129 created by mongoshake/collector.(*ReplicationCoordinator).startOplogReplication /home/zhuzhao.cx/mongo-shake/src/mongoshake/collector/replication.go:312 +0x1fe

goroutine 125 [select]: vendor/github.com/vinllen/mgo.(*mongoCluster).syncServersLoop(0xc4201ee360) /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/cluster.go:431 +0x31a created by vendor/github.com/vinllen/mgo.newCluster /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/cluster.go:83 +0x1ae

goroutine 127 [sleep]: time.Sleep(0x37e11d600) /data/u4/boyi.gw/tools/go1.10.3/src/runtime/time.go:102 +0x166 vendor/github.com/vinllen/mgo.(*mongoServer).pinger(0xc4202ae3c0, 0xc42026e201) /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/server.go:426 +0x4b3 created by vendor/github.com/vinllen/mgo.newServer /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/server.go:107 +0x186

goroutine 184 [IO wait]: internal/poll.runtime_pollWait(0x7f41847f92d0, 0x72, 0xc4200ebc88) /data/u4/boyi.gw/tools/go1.10.3/src/runtime/netpoll.go:173 +0x57 internal/poll.(pollDesc).wait(0xc4202f8098, 0x72, 0xffffffffffffff00, 0xa19f20, 0xc576a0) /data/u4/boyi.gw/tools/go1.10.3/src/internal/poll/fd_poll_runtime.go:85 +0x9b internal/poll.(pollDesc).waitRead(0xc4202f8098, 0xc42036a400, 0x10, 0x10) /data/u4/boyi.gw/tools/go1.10.3/src/internal/poll/fd_poll_runtime.go:90 +0x3d internal/poll.(FD).Read(0xc4202f8080, 0xc42036a400, 0x10, 0x10, 0x0, 0x0, 0x0) /data/u4/boyi.gw/tools/go1.10.3/src/internal/poll/fd_unix.go:157 +0x17d net.(netFD).Read(0xc4202f8080, 0xc42036a400, 0x10, 0x10, 0x0, 0x0, 0x0) /data/u4/boyi.gw/tools/go1.10.3/src/net/fd_unix.go:202 +0x4f net.(conn).Read(0xc4200b40e8, 0xc42036a400, 0x10, 0x10, 0x0, 0x0, 0x0) /data/u4/boyi.gw/tools/go1.10.3/src/net/net.go:176 +0x6a vendor/github.com/vinllen/mgo.fill(0xa20200, 0xc4200b40e8, 0xc42036a400, 0x10, 0x10, 0x0, 0x8e) /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/socket.go:601 +0x53 vendor/github.com/vinllen/mgo.(mongoSocket).readLoop(0xc4200d9320) /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/socket.go:619 +0x122 created by vendor/github.com/vinllen/mgo.newSocket /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/socket.go:211 +0x1f5

goroutine 110 [chan receive, 1343 minutes]: mongoshake/executor.(Executor).start(0xc42036c0a0) /home/zhuzhao.cx/mongo-shake/src/mongoshake/executor/executor.go:200 +0x10c created by mongoshake/executor.(BatchGroupExecutor).Start /home/zhuzhao.cx/mongo-shake/src/mongoshake/executor/executor.go:68 +0x9f

goroutine 111 [select]: mongoshake/collector.(Worker).findFirstAvailableBatch(0xc420294000, 0xa156c0, 0xc4202fafa8, 0x2) /home/zhuzhao.cx/mongo-shake/src/mongoshake/collector/worker.go:102 +0xfb mongoshake/collector.(Worker).startWorker(0xc420294000) /home/zhuzhao.cx/mongo-shake/src/mongoshake/collector/worker.go:132 +0x129 created by mongoshake/collector.(*ReplicationCoordinator).startOplogReplication /home/zhuzhao.cx/mongo-shake/src/mongoshake/collector/replication.go:312 +0x1fe

goroutine 112 [select]: vendor/github.com/vinllen/mgo.(*mongoCluster).syncServersLoop(0xc42037c5a0) /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/cluster.go:431 +0x31a created by vendor/github.com/vinllen/mgo.newCluster /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/cluster.go:83 +0x1ae

goroutine 202 [select]: vendor/github.com/vinllen/mgo.(*mongoCluster).syncServersLoop(0xc42037cc60) /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/cluster.go:431 +0x31a created by vendor/github.com/vinllen/mgo.newCluster /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/cluster.go:83 +0x1ae

goroutine 63033 [IO wait]: internal/poll.runtime_pollWait(0x7f418477c408, 0x72, 0xc4229d2c88) /data/u4/boyi.gw/tools/go1.10.3/src/runtime/netpoll.go:173 +0x57 internal/poll.(pollDesc).wait(0xc422964698, 0x72, 0xffffffffffffff00, 0xa19f20, 0xc576a0) /data/u4/boyi.gw/tools/go1.10.3/src/internal/poll/fd_poll_runtime.go:85 +0x9b internal/poll.(pollDesc).waitRead(0xc422964698, 0xc4202da500, 0x10, 0x10) /data/u4/boyi.gw/tools/go1.10.3/src/internal/poll/fd_poll_runtime.go:90 +0x3d internal/poll.(FD).Read(0xc422964680, 0xc4202da540, 0x10, 0x10, 0x0, 0x0, 0x0) /data/u4/boyi.gw/tools/go1.10.3/src/internal/poll/fd_unix.go:157 +0x17d net.(netFD).Read(0xc422964680, 0xc4202da540, 0x10, 0x10, 0x0, 0x0, 0x0) /data/u4/boyi.gw/tools/go1.10.3/src/net/fd_unix.go:202 +0x4f net.(conn).Read(0xc4201380f0, 0xc4202da540, 0x10, 0x10, 0x0, 0x0, 0x0) /data/u4/boyi.gw/tools/go1.10.3/src/net/net.go:176 +0x6a vendor/github.com/vinllen/mgo.fill(0xa20200, 0xc4201380f0, 0xc4202da540, 0x10, 0x10, 0x0, 0x8e) /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/socket.go:601 +0x53 vendor/github.com/vinllen/mgo.(mongoSocket).readLoop(0xc42030e120) /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/socket.go:619 +0x122 created by vendor/github.com/vinllen/mgo.newSocket /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/socket.go:211 +0x1f5

goroutine 200 [chan receive, 1343 minutes]: mongoshake/executor.(Executor).start(0xc4201b2370) /home/zhuzhao.cx/mongo-shake/src/mongoshake/executor/executor.go:200 +0x10c created by mongoshake/executor.(BatchGroupExecutor).Start /home/zhuzhao.cx/mongo-shake/src/mongoshake/executor/executor.go:68 +0x9f

goroutine 201 [select]: mongoshake/collector.(Worker).findFirstAvailableBatch(0xc420294150, 0xa156c0, 0xc4202fdfa8, 0x2) /home/zhuzhao.cx/mongo-shake/src/mongoshake/collector/worker.go:102 +0xfb mongoshake/collector.(Worker).startWorker(0xc420294150) /home/zhuzhao.cx/mongo-shake/src/mongoshake/collector/worker.go:132 +0x129 created by mongoshake/collector.(*ReplicationCoordinator).startOplogReplication /home/zhuzhao.cx/mongo-shake/src/mongoshake/collector/replication.go:312 +0x1fe

goroutine 173 [select]: mongoshake/collector.(Worker).findFirstAvailableBatch(0xc420326310, 0xa156c0, 0xc42032afa8, 0x2) /home/zhuzhao.cx/mongo-shake/src/mongoshake/collector/worker.go:102 +0xfb mongoshake/collector.(Worker).startWorker(0xc420326310) /home/zhuzhao.cx/mongo-shake/src/mongoshake/collector/worker.go:132 +0x129 created by mongoshake/collector.(*ReplicationCoordinator).startOplogReplication /home/zhuzhao.cx/mongo-shake/src/mongoshake/collector/replication.go:312 +0x1fe

goroutine 62941 [IO wait]: internal/poll.runtime_pollWait(0x7f41847f97b0, 0x72, 0xc4229d3c88) /data/u4/boyi.gw/tools/go1.10.3/src/runtime/netpoll.go:173 +0x57 internal/poll.(pollDesc).wait(0xc422980598, 0x72, 0xffffffffffffff00, 0xa19f20, 0xc576a0) /data/u4/boyi.gw/tools/go1.10.3/src/internal/poll/fd_poll_runtime.go:85 +0x9b internal/poll.(pollDesc).waitRead(0xc422980598, 0xc42036bb00, 0x10, 0x10) /data/u4/boyi.gw/tools/go1.10.3/src/internal/poll/fd_poll_runtime.go:90 +0x3d internal/poll.(FD).Read(0xc422980580, 0xc42036bbe0, 0x10, 0x10, 0x0, 0x0, 0x0) /data/u4/boyi.gw/tools/go1.10.3/src/internal/poll/fd_unix.go:157 +0x17d net.(netFD).Read(0xc422980580, 0xc42036bbe0, 0x10, 0x10, 0x0, 0x0, 0x0) /data/u4/boyi.gw/tools/go1.10.3/src/net/fd_unix.go:202 +0x4f net.(conn).Read(0xc422830050, 0xc42036bbe0, 0x10, 0x10, 0x0, 0x0, 0x0) /data/u4/boyi.gw/tools/go1.10.3/src/net/net.go:176 +0x6a vendor/github.com/vinllen/mgo.fill(0xa20200, 0xc422830050, 0xc42036bbe0, 0x10, 0x10, 0x0, 0x8e) /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/socket.go:601 +0x53 vendor/github.com/vinllen/mgo.(mongoSocket).readLoop(0xc4201efc20) /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/socket.go:619 +0x122 created by vendor/github.com/vinllen/mgo.newSocket /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/socket.go:211 +0x1f5

goroutine 172 [chan receive, 201 minutes]: mongoshake/executor.(Executor).start(0xc42036c410) /home/zhuzhao.cx/mongo-shake/src/mongoshake/executor/executor.go:200 +0x10c created by mongoshake/executor.(BatchGroupExecutor).Start /home/zhuzhao.cx/mongo-shake/src/mongoshake/executor/executor.go:68 +0x9f

goroutine 182 [sleep]: time.Sleep(0x37e11d600) /data/u4/boyi.gw/tools/go1.10.3/src/runtime/time.go:102 +0x166 vendor/github.com/vinllen/mgo.(*mongoServer).pinger(0xc42031e4b0, 0xc4200dc401) /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/server.go:426 +0x4b3 created by vendor/github.com/vinllen/mgo.newServer /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/server.go:107 +0x186

goroutine 204 [sleep]: time.Sleep(0x37e11d600) /data/u4/boyi.gw/tools/go1.10.3/src/runtime/time.go:102 +0x166 vendor/github.com/vinllen/mgo.(*mongoServer).pinger(0xc420354f00, 0xc4201cc401) /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/server.go:426 +0x4b3 created by vendor/github.com/vinllen/mgo.newServer /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/server.go:107 +0x186

goroutine 218 [semacquire]: sync.runtime_notifyListWait(0xc4202be3d8, 0xc400016082) /data/u4/boyi.gw/tools/go1.10.3/src/runtime/sema.go:510 +0x10b sync.(Cond).Wait(0xc4202be3c8) /data/u4/boyi.gw/tools/go1.10.3/src/sync/cond.go:56 +0x80 vendor/github.com/vinllen/mgo.(Iter).Next(0xc4202be3c0, 0x8fa380, 0xc4229ec4c0, 0xc4201cca00) /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/session.go:4269 +0x9a mongoshake/collector.(OplogReader).fetcher(0xc42033e880) /home/zhuzhao.cx/mongo-shake/src/mongoshake/collector/oplog_reader.go:144 +0x109 created by mongoshake/collector.(OplogReader).StartFetcher /home/zhuzhao.cx/mongo-shake/src/mongoshake/collector/oplog_reader.go:129 +0x82

goroutine 174 [select]: vendor/github.com/vinllen/mgo.(*mongoCluster).syncServersLoop(0xc4201ee5a0) /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/cluster.go:431 +0x31a created by vendor/github.com/vinllen/mgo.newCluster /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/cluster.go:83 +0x1ae

goroutine 226 [chan receive, 1343 minutes]: mongoshake/executor.(Executor).start(0xc42036c5a0) /home/zhuzhao.cx/mongo-shake/src/mongoshake/executor/executor.go:200 +0x10c created by mongoshake/executor.(BatchGroupExecutor).Start /home/zhuzhao.cx/mongo-shake/src/mongoshake/executor/executor.go:68 +0x9f

goroutine 227 [select]: mongoshake/collector.(Worker).findFirstAvailableBatch(0xc420294380, 0xa156c0, 0xc4200ecfa8, 0x2) /home/zhuzhao.cx/mongo-shake/src/mongoshake/collector/worker.go:102 +0xfb mongoshake/collector.(Worker).startWorker(0xc420294380) /home/zhuzhao.cx/mongo-shake/src/mongoshake/collector/worker.go:132 +0x129 created by mongoshake/collector.(*ReplicationCoordinator).startOplogReplication /home/zhuzhao.cx/mongo-shake/src/mongoshake/collector/replication.go:312 +0x1fe

goroutine 228 [select]: mongoshake/collector.(OplogReader).get(0xc42033e880, 0x0, 0xa186e0, 0xc4200a4dd0) /home/zhuzhao.cx/mongo-shake/src/mongoshake/collector/oplog_reader.go:112 +0xfd mongoshake/collector.(OplogReader).Next(0xc42033e880, 0xc4200a4dd0, 0xc4200a4dd0, 0xc4200a4d01) /home/zhuzhao.cx/mongo-shake/src/mongoshake/collector/oplog_reader.go:94 +0x2b mongoshake/collector.(OplogSyncer).next(0xc420355590, 0xc420358e00) /home/zhuzhao.cx/mongo-shake/src/mongoshake/collector/syncer.go:367 +0x47 mongoshake/collector.(OplogSyncer).poll(0xc420355590) /home/zhuzhao.cx/mongo-shake/src/mongoshake/collector/syncer.go:359 +0x106 mongoshake/collector.(OplogSyncer).start(0xc420355590) /home/zhuzhao.cx/mongo-shake/src/mongoshake/collector/syncer.go:166 +0x1a8 created by mongoshake/collector.(ReplicationCoordinator).startOplogReplication /home/zhuzhao.cx/mongo-shake/src/mongoshake/collector/replication.go:316 +0x4d2

goroutine 229 [chan receive]: mongoshake/collector.(OplogSyncer).deserializer(0xc420355590, 0x0) /home/zhuzhao.cx/mongo-shake/src/mongoshake/collector/syncer.go:305 +0x290 created by mongoshake/collector.(OplogSyncer).startDeserializer /home/zhuzhao.cx/mongo-shake/src/mongoshake/collector/syncer.go:299 +0x10b

goroutine 230 [chan receive]: mongoshake/collector.(OplogSyncer).deserializer(0xc420355590, 0x1) /home/zhuzhao.cx/mongo-shake/src/mongoshake/collector/syncer.go:305 +0x290 created by mongoshake/collector.(OplogSyncer).startDeserializer /home/zhuzhao.cx/mongo-shake/src/mongoshake/collector/syncer.go:299 +0x10b

goroutine 231 [chan receive]: mongoshake/collector.(OplogSyncer).deserializer(0xc420355590, 0x2) /home/zhuzhao.cx/mongo-shake/src/mongoshake/collector/syncer.go:305 +0x290 created by mongoshake/collector.(OplogSyncer).startDeserializer /home/zhuzhao.cx/mongo-shake/src/mongoshake/collector/syncer.go:299 +0x10b

goroutine 232 [chan receive]: mongoshake/collector.(OplogSyncer).deserializer(0xc420355590, 0x3) /home/zhuzhao.cx/mongo-shake/src/mongoshake/collector/syncer.go:305 +0x290 created by mongoshake/collector.(OplogSyncer).startDeserializer /home/zhuzhao.cx/mongo-shake/src/mongoshake/collector/syncer.go:299 +0x10b

goroutine 233 [chan receive]: mongoshake/collector.(Batcher).batchMore(0xc4202b0a80, 0x494ed23cd8d6, 0xc96880, 0xbf549d528f8facb8, 0x4971f8d91be5) /home/zhuzhao.cx/mongo-shake/src/mongoshake/collector/batcher.go:112 +0xf6 mongoshake/collector.(OplogSyncer).startBatcher.func1() /home/zhuzhao.cx/mongo-shake/src/mongoshake/collector/syncer.go:184 +0x6b vendor/github.com/gugemichael/nimo4go.GoRoutineInLoop.func1(0xc42026a570) /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/gugemichael/nimo4go/runtime.go:19 +0x25 created by vendor/github.com/gugemichael/nimo4go.GoRoutineInLoop /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/gugemichael/nimo4go/runtime.go:17 +0x3f

goroutine 234 [select]: vendor/github.com/vinllen/mgo.(*mongoCluster).syncServersLoop(0xc4201ee6c0) /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/cluster.go:431 +0x31a created by vendor/github.com/vinllen/mgo.newCluster /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/cluster.go:83 +0x1ae

goroutine 213 [sleep]: time.Sleep(0x37e11d600) /data/u4/boyi.gw/tools/go1.10.3/src/runtime/time.go:102 +0x166 vendor/github.com/vinllen/mgo.(*mongoServer).pinger(0xc420355680, 0xc4201cc901) /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/server.go:426 +0x4b3 created by vendor/github.com/vinllen/mgo.newServer /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/server.go:107 +0x186

goroutine 236 [sleep]: time.Sleep(0x37e11d600) /data/u4/boyi.gw/tools/go1.10.3/src/runtime/time.go:102 +0x166 vendor/github.com/vinllen/mgo.(*mongoServer).pinger(0xc4202aea50, 0xc420022601) /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/server.go:426 +0x4b3 created by vendor/github.com/vinllen/mgo.newServer /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/server.go:107 +0x186

goroutine 238 [IO wait]: internal/poll.runtime_pollWait(0x7f41847f9540, 0x72, 0xc42035fc88) /data/u4/boyi.gw/tools/go1.10.3/src/runtime/netpoll.go:173 +0x57 internal/poll.(pollDesc).wait(0xc4202b0498, 0x72, 0xffffffffffffff00, 0xa19f20, 0xc576a0) /data/u4/boyi.gw/tools/go1.10.3/src/internal/poll/fd_poll_runtime.go:85 +0x9b internal/poll.(pollDesc).waitRead(0xc4202b0498, 0xc420022800, 0x10, 0x10) /data/u4/boyi.gw/tools/go1.10.3/src/internal/poll/fd_poll_runtime.go:90 +0x3d internal/poll.(FD).Read(0xc4202b0480, 0xc420022840, 0x10, 0x10, 0x0, 0x0, 0x0) /data/u4/boyi.gw/tools/go1.10.3/src/internal/poll/fd_unix.go:157 +0x17d net.(netFD).Read(0xc4202b0480, 0xc420022840, 0x10, 0x10, 0x0, 0x0, 0x0) /data/u4/boyi.gw/tools/go1.10.3/src/net/fd_unix.go:202 +0x4f net.(conn).Read(0xc4202d00d8, 0xc420022840, 0x10, 0x10, 0x0, 0x0, 0x0) /data/u4/boyi.gw/tools/go1.10.3/src/net/net.go:176 +0x6a vendor/github.com/vinllen/mgo.fill(0xa20200, 0xc4202d00d8, 0xc420022840, 0x10, 0x10, 0x0, 0xe0) /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/socket.go:601 +0x53 vendor/github.com/vinllen/mgo.(mongoSocket).readLoop(0xc4201ee7e0) /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/socket.go:619 +0x122 created by vendor/github.com/vinllen/mgo.newSocket /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/socket.go:211 +0x1f5

goroutine 219 [select]: vendor/github.com/vinllen/mgo.(*mongoCluster).syncServersLoop(0xc42037db00) /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/cluster.go:431 +0x31a created by vendor/github.com/vinllen/mgo.newCluster /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/cluster.go:83 +0x1ae

goroutine 221 [sleep]: time.Sleep(0x37e11d600) /data/u4/boyi.gw/tools/go1.10.3/src/runtime/time.go:102 +0x166 vendor/github.com/vinllen/mgo.(*mongoServer).pinger(0xc420355d10, 0xc420275001) /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/server.go:426 +0x4b3 created by vendor/github.com/vinllen/mgo.newServer /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/server.go:107 +0x186

goroutine 223 [IO wait]: internal/poll.runtime_pollWait(0x7f41847f93a0, 0x72, 0xc42032dc88) /data/u4/boyi.gw/tools/go1.10.3/src/runtime/netpoll.go:173 +0x57 internal/poll.(pollDesc).wait(0xc4200f2698, 0x72, 0xffffffffffffff00, 0xa19f20, 0xc576a0) /data/u4/boyi.gw/tools/go1.10.3/src/internal/poll/fd_poll_runtime.go:85 +0x9b internal/poll.(pollDesc).waitRead(0xc4200f2698, 0xc420022800, 0x10, 0x10) /data/u4/boyi.gw/tools/go1.10.3/src/internal/poll/fd_poll_runtime.go:90 +0x3d internal/poll.(FD).Read(0xc4200f2680, 0xc4200228b0, 0x10, 0x10, 0x0, 0x0, 0x0) /data/u4/boyi.gw/tools/go1.10.3/src/internal/poll/fd_unix.go:157 +0x17d net.(netFD).Read(0xc4200f2680, 0xc4200228b0, 0x10, 0x10, 0x0, 0x0, 0x0) /data/u4/boyi.gw/tools/go1.10.3/src/net/fd_unix.go:202 +0x4f net.(conn).Read(0xc42033c138, 0xc4200228b0, 0x10, 0x10, 0x0, 0x0, 0x0) /data/u4/boyi.gw/tools/go1.10.3/src/net/net.go:176 +0x6a vendor/github.com/vinllen/mgo.fill(0xa20200, 0xc42033c138, 0xc4200228b0, 0x10, 0x10, 0x0, 0x0) /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/socket.go:601 +0x53 vendor/github.com/vinllen/mgo.(mongoSocket).readLoop(0xc42037dc20) /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/socket.go:619 +0x122 created by vendor/github.com/vinllen/mgo.newSocket /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/socket.go:211 +0x1f5

goroutine 2694 [IO wait]: internal/poll.runtime_pollWait(0x7f41847f9d60, 0x72, 0xc420373c88) /data/u4/boyi.gw/tools/go1.10.3/src/runtime/netpoll.go:173 +0x57 internal/poll.(pollDesc).wait(0xc4200f2418, 0x72, 0xffffffffffffff00, 0xa19f20, 0xc576a0) /data/u4/boyi.gw/tools/go1.10.3/src/internal/poll/fd_poll_runtime.go:85 +0x9b internal/poll.(pollDesc).waitRead(0xc4200f2418, 0xc4200b0600, 0x10, 0x10) /data/u4/boyi.gw/tools/go1.10.3/src/internal/poll/fd_poll_runtime.go:90 +0x3d internal/poll.(FD).Read(0xc4200f2400, 0xc4200b0650, 0x10, 0x10, 0x0, 0x0, 0x0) /data/u4/boyi.gw/tools/go1.10.3/src/internal/poll/fd_unix.go:157 +0x17d net.(netFD).Read(0xc4200f2400, 0xc4200b0650, 0x10, 0x10, 0x0, 0x0, 0x0) /data/u4/boyi.gw/tools/go1.10.3/src/net/fd_unix.go:202 +0x4f net.(conn).Read(0xc42000e008, 0xc4200b0650, 0x10, 0x10, 0x0, 0x0, 0x0) /data/u4/boyi.gw/tools/go1.10.3/src/net/net.go:176 +0x6a vendor/github.com/vinllen/mgo.fill(0xa20200, 0xc42000e008, 0xc4200b0650, 0x10, 0x10, 0x0, 0x8e) /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/socket.go:601 +0x53 vendor/github.com/vinllen/mgo.(mongoSocket).readLoop(0xc4202f2000) /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/socket.go:619 +0x122 created by vendor/github.com/vinllen/mgo.newSocket /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/socket.go:211 +0x1f5

goroutine 64391 [IO wait]: internal/poll.runtime_pollWait(0x7f41847f9880, 0x72, 0xc42035ec88) /data/u4/boyi.gw/tools/go1.10.3/src/runtime/netpoll.go:173 +0x57 internal/poll.(pollDesc).wait(0xc422980098, 0x72, 0xffffffffffffff00, 0xa19f20, 0xc576a0) /data/u4/boyi.gw/tools/go1.10.3/src/internal/poll/fd_poll_runtime.go:85 +0x9b internal/poll.(pollDesc).waitRead(0xc422980098, 0xc4229e9100, 0x10, 0x10) /data/u4/boyi.gw/tools/go1.10.3/src/internal/poll/fd_poll_runtime.go:90 +0x3d internal/poll.(FD).Read(0xc422980080, 0xc4229e91c0, 0x10, 0x10, 0x0, 0x0, 0x0) /data/u4/boyi.gw/tools/go1.10.3/src/internal/poll/fd_unix.go:157 +0x17d net.(netFD).Read(0xc422980080, 0xc4229e91c0, 0x10, 0x10, 0x0, 0x0, 0x0) /data/u4/boyi.gw/tools/go1.10.3/src/net/fd_unix.go:202 +0x4f net.(conn).Read(0xc4202d0128, 0xc4229e91c0, 0x10, 0x10, 0x0, 0x0, 0x0) /data/u4/boyi.gw/tools/go1.10.3/src/net/net.go:176 +0x6a vendor/github.com/vinllen/mgo.fill(0xa20200, 0xc4202d0128, 0xc4229e91c0, 0x10, 0x10, 0x0, 0x8e) /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/socket.go:601 +0x53 vendor/github.com/vinllen/mgo.(mongoSocket).readLoop(0xc42037e000) /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/socket.go:619 +0x122 created by vendor/github.com/vinllen/mgo.newSocket /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/socket.go:211 +0x1f5

goroutine 63046 [sleep]: time.Sleep(0x37e11d600) /data/u4/boyi.gw/tools/go1.10.3/src/runtime/time.go:102 +0x166 vendor/github.com/vinllen/mgo.(*mongoServer).pinger(0xc4202bfe00, 0xc4201cdc01) /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/server.go:426 +0x4b3 created by vendor/github.com/vinllen/mgo.newServer /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/server.go:107 +0x186

goroutine 62985 [sleep]: time.Sleep(0x37e11d600) /data/u4/boyi.gw/tools/go1.10.3/src/runtime/time.go:102 +0x166 vendor/github.com/vinllen/mgo.(*mongoServer).pinger(0xc4203223c0, 0xc422a0d701) /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/server.go:426 +0x4b3 created by vendor/github.com/vinllen/mgo.newServer /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/server.go:107 +0x186

goroutine 63038 [sleep]: time.Sleep(0x37e11d600) /data/u4/boyi.gw/tools/go1.10.3/src/runtime/time.go:102 +0x166 vendor/github.com/vinllen/mgo.(*mongoServer).pinger(0xc4228a0b40, 0xc4200dca01) /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/server.go:426 +0x4b3 created by vendor/github.com/vinllen/mgo.newServer /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/server.go:107 +0x186

goroutine 63044 [IO wait, 617 minutes]: internal/poll.runtime_pollWait(0x7f41847f96e0, 0x72, 0xc422929c88) /data/u4/boyi.gw/tools/go1.10.3/src/runtime/netpoll.go:173 +0x57 internal/poll.(pollDesc).wait(0xc42024c498, 0x72, 0xffffffffffffff00, 0xa19f20, 0xc576a0) /data/u4/boyi.gw/tools/go1.10.3/src/internal/poll/fd_poll_runtime.go:85 +0x9b internal/poll.(pollDesc).waitRead(0xc42024c498, 0xc42295f600, 0x10, 0x10) /data/u4/boyi.gw/tools/go1.10.3/src/internal/poll/fd_poll_runtime.go:90 +0x3d internal/poll.(FD).Read(0xc42024c480, 0xc42295f650, 0x10, 0x10, 0x0, 0x0, 0x0) /data/u4/boyi.gw/tools/go1.10.3/src/internal/poll/fd_unix.go:157 +0x17d net.(netFD).Read(0xc42024c480, 0xc42295f650, 0x10, 0x10, 0x0, 0x0, 0x0) /data/u4/boyi.gw/tools/go1.10.3/src/net/fd_unix.go:202 +0x4f net.(conn).Read(0xc42033c0f0, 0xc42295f650, 0x10, 0x10, 0x0, 0x0, 0x0) /data/u4/boyi.gw/tools/go1.10.3/src/net/net.go:176 +0x6a vendor/github.com/vinllen/mgo.fill(0xa20200, 0xc42033c0f0, 0xc42295f650, 0x10, 0x10, 0x0, 0x282) /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/socket.go:601 +0x53 vendor/github.com/vinllen/mgo.(mongoSocket).readLoop(0xc42295d320) /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/socket.go:619 +0x122 created by vendor/github.com/vinllen/mgo.newSocket /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/socket.go:211 +0x1f5

goroutine 2729 [IO wait, 1313 minutes]: internal/poll.runtime_pollWait(0x7f418477c9b8, 0x72, 0xc42032ec88) /data/u4/boyi.gw/tools/go1.10.3/src/runtime/netpoll.go:173 +0x57 internal/poll.(pollDesc).wait(0xc4202b0198, 0x72, 0xffffffffffffff00, 0xa19f20, 0xc576a0) /data/u4/boyi.gw/tools/go1.10.3/src/internal/poll/fd_poll_runtime.go:85 +0x9b internal/poll.(pollDesc).waitRead(0xc4202b0198, 0xc4201ba100, 0x10, 0x10) /data/u4/boyi.gw/tools/go1.10.3/src/internal/poll/fd_poll_runtime.go:90 +0x3d internal/poll.(FD).Read(0xc4202b0180, 0xc4201ba180, 0x10, 0x10, 0x0, 0x0, 0x0) /data/u4/boyi.gw/tools/go1.10.3/src/internal/poll/fd_unix.go:157 +0x17d net.(netFD).Read(0xc4202b0180, 0xc4201ba180, 0x10, 0x10, 0x0, 0x0, 0x0) /data/u4/boyi.gw/tools/go1.10.3/src/net/fd_unix.go:202 +0x4f net.(conn).Read(0xc42033c000, 0xc4201ba180, 0x10, 0x10, 0x0, 0x0, 0x0) /data/u4/boyi.gw/tools/go1.10.3/src/net/net.go:176 +0x6a vendor/github.com/vinllen/mgo.fill(0xa20200, 0xc42033c000, 0xc4201ba180, 0x10, 0x10, 0x0, 0x8e) /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/socket.go:601 +0x53 vendor/github.com/vinllen/mgo.(mongoSocket).readLoop(0xc42037e240) /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/socket.go:619 +0x122 created by vendor/github.com/vinllen/mgo.newSocket /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/socket.go:211 +0x1f5

goroutine 5884 [select]: vendor/github.com/vinllen/mgo.(*mongoCluster).syncServersLoop(0xc4200d9200) /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/cluster.go:431 +0x31a created by vendor/github.com/vinllen/mgo.newCluster /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/cluster.go:83 +0x1ae

goroutine 62976 [sleep]: time.Sleep(0x37e11d600) /data/u4/boyi.gw/tools/go1.10.3/src/runtime/time.go:102 +0x166 vendor/github.com/vinllen/mgo.(*mongoServer).pinger(0xc4228b4000, 0xc4229c6001) /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/server.go:426 +0x4b3 created by vendor/github.com/vinllen/mgo.newServer /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/server.go:107 +0x186

goroutine 62944 [sleep]: time.Sleep(0x37e11d600) /data/u4/boyi.gw/tools/go1.10.3/src/runtime/time.go:102 +0x166 vendor/github.com/vinllen/mgo.(*mongoServer).pinger(0xc4228685a0, 0xc4227cc201) /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/server.go:426 +0x4b3 created by vendor/github.com/vinllen/mgo.newServer /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/server.go:107 +0x186

goroutine 62993 [IO wait]: internal/poll.runtime_pollWait(0x7f418477c0c8, 0x72, 0xc4229e1c88) /data/u4/boyi.gw/tools/go1.10.3/src/runtime/netpoll.go:173 +0x57 internal/poll.(pollDesc).wait(0xc42024c398, 0x72, 0xffffffffffffff00, 0xa19f20, 0xc576a0) /data/u4/boyi.gw/tools/go1.10.3/src/internal/poll/fd_poll_runtime.go:85 +0x9b internal/poll.(pollDesc).waitRead(0xc42024c398, 0xc42295f100, 0x10, 0x10) /data/u4/boyi.gw/tools/go1.10.3/src/internal/poll/fd_poll_runtime.go:90 +0x3d internal/poll.(FD).Read(0xc42024c380, 0xc42295f120, 0x10, 0x10, 0x0, 0x0, 0x0) /data/u4/boyi.gw/tools/go1.10.3/src/internal/poll/fd_unix.go:157 +0x17d net.(netFD).Read(0xc42024c380, 0xc42295f120, 0x10, 0x10, 0x0, 0x0, 0x0) /data/u4/boyi.gw/tools/go1.10.3/src/net/fd_unix.go:202 +0x4f net.(conn).Read(0xc4202e2110, 0xc42295f120, 0x10, 0x10, 0x0, 0x0, 0x0) /data/u4/boyi.gw/tools/go1.10.3/src/net/net.go:176 +0x6a vendor/github.com/vinllen/mgo.fill(0xa20200, 0xc4202e2110, 0xc42295f120, 0x10, 0x10, 0x0, 0x8e) /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/socket.go:601 +0x53 vendor/github.com/vinllen/mgo.(mongoSocket).readLoop(0xc420310360) /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/socket.go:619 +0x122 created by vendor/github.com/vinllen/mgo.newSocket /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/socket.go:211 +0x1f5

goroutine 63058 [IO wait]: internal/poll.runtime_pollWait(0x7f41847f9950, 0x72, 0xc422996c88) /data/u4/boyi.gw/tools/go1.10.3/src/runtime/netpoll.go:173 +0x57 internal/poll.(pollDesc).wait(0xc422980698, 0x72, 0xffffffffffffff00, 0xa19f20, 0xc576a0) /data/u4/boyi.gw/tools/go1.10.3/src/internal/poll/fd_poll_runtime.go:85 +0x9b internal/poll.(pollDesc).waitRead(0xc422980698, 0xc420023c00, 0x10, 0x10) /data/u4/boyi.gw/tools/go1.10.3/src/internal/poll/fd_poll_runtime.go:90 +0x3d internal/poll.(FD).Read(0xc422980680, 0xc420023c60, 0x10, 0x10, 0x0, 0x0, 0x0) /data/u4/boyi.gw/tools/go1.10.3/src/internal/poll/fd_unix.go:157 +0x17d net.(netFD).Read(0xc422980680, 0xc420023c60, 0x10, 0x10, 0x0, 0x0, 0x0) /data/u4/boyi.gw/tools/go1.10.3/src/net/fd_unix.go:202 +0x4f net.(conn).Read(0xc4200b4158, 0xc420023c60, 0x10, 0x10, 0x0, 0x0, 0x0) /data/u4/boyi.gw/tools/go1.10.3/src/net/net.go:176 +0x6a vendor/github.com/vinllen/mgo.fill(0xa20200, 0xc4200b4158, 0xc420023c60, 0x10, 0x10, 0x0, 0x8e) /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/socket.go:601 +0x53 vendor/github.com/vinllen/mgo.(mongoSocket).readLoop(0xc4228b0360) /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/socket.go:619 +0x122 created by vendor/github.com/vinllen/mgo.newSocket /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/socket.go:211 +0x1f5

goroutine 63035 [sleep]: time.Sleep(0x37e11d600) /data/u4/boyi.gw/tools/go1.10.3/src/runtime/time.go:102 +0x166 vendor/github.com/vinllen/mgo.(*mongoServer).pinger(0xc4228a0a50, 0xc4200dc901) /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/server.go:426 +0x4b3 created by vendor/github.com/vinllen/mgo.newServer /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/server.go:107 +0x186

goroutine 42977 [IO wait]: internal/poll.runtime_pollWait(0x7f418477c268, 0x72, 0xc4200e7c88) /data/u4/boyi.gw/tools/go1.10.3/src/runtime/netpoll.go:173 +0x57 internal/poll.(pollDesc).wait(0xc4202f8498, 0x72, 0xffffffffffffff00, 0xa19f20, 0xc576a0) /data/u4/boyi.gw/tools/go1.10.3/src/internal/poll/fd_poll_runtime.go:85 +0x9b internal/poll.(pollDesc).waitRead(0xc4202f8498, 0xc4200b0a00, 0x10, 0x10) /data/u4/boyi.gw/tools/go1.10.3/src/internal/poll/fd_poll_runtime.go:90 +0x3d internal/poll.(FD).Read(0xc4202f8480, 0xc4200b0ad0, 0x10, 0x10, 0x0, 0x0, 0x0) /data/u4/boyi.gw/tools/go1.10.3/src/internal/poll/fd_unix.go:157 +0x17d net.(netFD).Read(0xc4202f8480, 0xc4200b0ad0, 0x10, 0x10, 0x0, 0x0, 0x0) /data/u4/boyi.gw/tools/go1.10.3/src/net/fd_unix.go:202 +0x4f net.(conn).Read(0xc42000e040, 0xc4200b0ad0, 0x10, 0x10, 0x0, 0x0, 0x0) /data/u4/boyi.gw/tools/go1.10.3/src/net/net.go:176 +0x6a vendor/github.com/vinllen/mgo.fill(0xa20200, 0xc42000e040, 0xc4200b0ad0, 0x10, 0x10, 0x0, 0x8e) /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/socket.go:601 +0x53 vendor/github.com/vinllen/mgo.(mongoSocket).readLoop(0xc42024e120) /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/socket.go:619 +0x122 created by vendor/github.com/vinllen/mgo.newSocket /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/socket.go:211 +0x1f5

goroutine 63043 [IO wait]: internal/poll.runtime_pollWait(0x7f418477c678, 0x72, 0xc42299bc88) /data/u4/boyi.gw/tools/go1.10.3/src/runtime/netpoll.go:173 +0x57 internal/poll.(pollDesc).wait(0xc422964798, 0x72, 0xffffffffffffff00, 0xa19f20, 0xc576a0) /data/u4/boyi.gw/tools/go1.10.3/src/internal/poll/fd_poll_runtime.go:85 +0x9b internal/poll.(pollDesc).waitRead(0xc422964798, 0xc42295f500, 0x10, 0x10) /data/u4/boyi.gw/tools/go1.10.3/src/internal/poll/fd_poll_runtime.go:90 +0x3d internal/poll.(FD).Read(0xc422964780, 0xc42295f590, 0x10, 0x10, 0x0, 0x0, 0x0) /data/u4/boyi.gw/tools/go1.10.3/src/internal/poll/fd_unix.go:157 +0x17d net.(netFD).Read(0xc422964780, 0xc42295f590, 0x10, 0x10, 0x0, 0x0, 0x0) /data/u4/boyi.gw/tools/go1.10.3/src/net/fd_unix.go:202 +0x4f net.(conn).Read(0xc42033c0d0, 0xc42295f590, 0x10, 0x10, 0x0, 0x0, 0x0) /data/u4/boyi.gw/tools/go1.10.3/src/net/net.go:176 +0x6a vendor/github.com/vinllen/mgo.fill(0xa20200, 0xc42033c0d0, 0xc42295f590, 0x10, 0x10, 0x0, 0x8e) /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/socket.go:601 +0x53 vendor/github.com/vinllen/mgo.(mongoSocket).readLoop(0xc42295cea0) /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/socket.go:619 +0x122 created by vendor/github.com/vinllen/mgo.newSocket /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/socket.go:211 +0x1f5

goroutine 103354 [IO wait]: internal/poll.runtime_pollWait(0x7f418477c8e8, 0x72, 0xc420372c88) /data/u4/boyi.gw/tools/go1.10.3/src/runtime/netpoll.go:173 +0x57 internal/poll.(pollDesc).wait(0xc422980218, 0x72, 0xffffffffffffff00, 0xa19f20, 0xc576a0) /data/u4/boyi.gw/tools/go1.10.3/src/internal/poll/fd_poll_runtime.go:85 +0x9b internal/poll.(pollDesc).waitRead(0xc422980218, 0xc4201ba600, 0x10, 0x10) /data/u4/boyi.gw/tools/go1.10.3/src/internal/poll/fd_poll_runtime.go:90 +0x3d internal/poll.(FD).Read(0xc422980200, 0xc4201ba610, 0x10, 0x10, 0x0, 0x0, 0x0) /data/u4/boyi.gw/tools/go1.10.3/src/internal/poll/fd_unix.go:157 +0x17d net.(netFD).Read(0xc422980200, 0xc4201ba610, 0x10, 0x10, 0x0, 0x0, 0x0) /data/u4/boyi.gw/tools/go1.10.3/src/net/fd_unix.go:202 +0x4f net.(conn).Read(0xc420138010, 0xc4201ba610, 0x10, 0x10, 0x0, 0x0, 0x0) /data/u4/boyi.gw/tools/go1.10.3/src/net/net.go:176 +0x6a vendor/github.com/vinllen/mgo.fill(0xa20200, 0xc420138010, 0xc4201ba610, 0x10, 0x10, 0x0, 0x8e) /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/socket.go:601 +0x53 vendor/github.com/vinllen/mgo.(mongoSocket).readLoop(0xc42024e000) /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/socket.go:619 +0x122 created by vendor/github.com/vinllen/mgo.newSocket /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/socket.go:211 +0x1f5

goroutine 5895 [IO wait, 201 minutes]: internal/poll.runtime_pollWait(0x7f418477c818, 0x72, 0xc422928c88) /data/u4/boyi.gw/tools/go1.10.3/src/runtime/netpoll.go:173 +0x57 internal/poll.(pollDesc).wait(0xc4202f8198, 0x72, 0xffffffffffffff00, 0xa19f20, 0xc576a0) /data/u4/boyi.gw/tools/go1.10.3/src/internal/poll/fd_poll_runtime.go:85 +0x9b internal/poll.(pollDesc).waitRead(0xc4202f8198, 0xc4201bb100, 0x10, 0x10) /data/u4/boyi.gw/tools/go1.10.3/src/internal/poll/fd_poll_runtime.go:90 +0x3d internal/poll.(FD).Read(0xc4202f8180, 0xc4201bb1b0, 0x10, 0x10, 0x0, 0x0, 0x0) /data/u4/boyi.gw/tools/go1.10.3/src/internal/poll/fd_unix.go:157 +0x17d net.(netFD).Read(0xc4202f8180, 0xc4201bb1b0, 0x10, 0x10, 0x0, 0x0, 0x0) /data/u4/boyi.gw/tools/go1.10.3/src/net/fd_unix.go:202 +0x4f net.(conn).Read(0xc4200b4108, 0xc4201bb1b0, 0x10, 0x10, 0x0, 0x0, 0x0) /data/u4/boyi.gw/tools/go1.10.3/src/net/net.go:176 +0x6a vendor/github.com/vinllen/mgo.fill(0xa20200, 0xc4200b4108, 0xc4201bb1b0, 0x10, 0x10, 0x0, 0x0) /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/socket.go:601 +0x53 vendor/github.com/vinllen/mgo.(mongoSocket).readLoop(0xc4202f2120) /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/socket.go:619 +0x122 created by vendor/github.com/vinllen/mgo.newSocket /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/socket.go:211 +0x1f5

goroutine 43070 [IO wait, 99 minutes]: internal/poll.runtime_pollWait(0x7f41847f9470, 0x72, 0xc42035cc88) /data/u4/boyi.gw/tools/go1.10.3/src/runtime/netpoll.go:173 +0x57 internal/poll.(pollDesc).wait(0xc4200f2198, 0x72, 0xffffffffffffff00, 0xa19f20, 0xc576a0) /data/u4/boyi.gw/tools/go1.10.3/src/internal/poll/fd_poll_runtime.go:85 +0x9b internal/poll.(pollDesc).waitRead(0xc4200f2198, 0xc420022300, 0x10, 0x10) /data/u4/boyi.gw/tools/go1.10.3/src/internal/poll/fd_poll_runtime.go:90 +0x3d internal/poll.(FD).Read(0xc4200f2180, 0xc420022300, 0x10, 0x10, 0x0, 0x0, 0x0) /data/u4/boyi.gw/tools/go1.10.3/src/internal/poll/fd_unix.go:157 +0x17d net.(netFD).Read(0xc4200f2180, 0xc420022300, 0x10, 0x10, 0x0, 0x0, 0x0) /data/u4/boyi.gw/tools/go1.10.3/src/net/fd_unix.go:202 +0x4f net.(conn).Read(0xc4201380c8, 0xc420022300, 0x10, 0x10, 0x0, 0x0, 0x0) /data/u4/boyi.gw/tools/go1.10.3/src/net/net.go:176 +0x6a vendor/github.com/vinllen/mgo.fill(0xa20200, 0xc4201380c8, 0xc420022300, 0x10, 0x10, 0x0, 0x8e) /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/socket.go:601 +0x53 vendor/github.com/vinllen/mgo.(mongoSocket).readLoop(0xc42037c6c0) /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/socket.go:619 +0x122 created by vendor/github.com/vinllen/mgo.newSocket /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/socket.go:211 +0x1f5

goroutine 99293 [IO wait]: internal/poll.runtime_pollWait(0x7f41847f9130, 0x72, 0xc42035dc88) /data/u4/boyi.gw/tools/go1.10.3/src/runtime/netpoll.go:173 +0x57 internal/poll.(pollDesc).wait(0xc4202f8218, 0x72, 0xffffffffffffff00, 0xa19f20, 0xc576a0) /data/u4/boyi.gw/tools/go1.10.3/src/internal/poll/fd_poll_runtime.go:85 +0x9b internal/poll.(pollDesc).waitRead(0xc4202f8218, 0xc42294c000, 0x10, 0x10) /data/u4/boyi.gw/tools/go1.10.3/src/internal/poll/fd_poll_runtime.go:90 +0x3d internal/poll.(FD).Read(0xc4202f8200, 0xc42294c0d0, 0x10, 0x10, 0x0, 0x0, 0x0) /data/u4/boyi.gw/tools/go1.10.3/src/internal/poll/fd_unix.go:157 +0x17d net.(netFD).Read(0xc4202f8200, 0xc42294c0d0, 0x10, 0x10, 0x0, 0x0, 0x0) /data/u4/boyi.gw/tools/go1.10.3/src/net/fd_unix.go:202 +0x4f net.(conn).Read(0xc42033c008, 0xc42294c0d0, 0x10, 0x10, 0x0, 0x0, 0x0) /data/u4/boyi.gw/tools/go1.10.3/src/net/net.go:176 +0x6a vendor/github.com/vinllen/mgo.fill(0xa20200, 0xc42033c008, 0xc42294c0d0, 0x10, 0x10, 0x0, 0x8e) /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/socket.go:601 +0x53 vendor/github.com/vinllen/mgo.(mongoSocket).readLoop(0xc42037e7e0) /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/socket.go:619 +0x122 created by vendor/github.com/vinllen/mgo.newSocket /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/socket.go:211 +0x1f5

goroutine 63059 [IO wait]: internal/poll.runtime_pollWait(0x7f41847f9bc0, 0x72, 0xc422926c88) /data/u4/boyi.gw/tools/go1.10.3/src/runtime/netpoll.go:173 +0x57 internal/poll.(pollDesc).wait(0xc422980798, 0x72, 0xffffffffffffff00, 0xa19f20, 0xc576a0) /data/u4/boyi.gw/tools/go1.10.3/src/internal/poll/fd_poll_runtime.go:85 +0x9b internal/poll.(pollDesc).waitRead(0xc422980798, 0xc420023d00, 0x10, 0x10) /data/u4/boyi.gw/tools/go1.10.3/src/internal/poll/fd_poll_runtime.go:90 +0x3d internal/poll.(FD).Read(0xc422980780, 0xc420023d20, 0x10, 0x10, 0x0, 0x0, 0x0) /data/u4/boyi.gw/tools/go1.10.3/src/internal/poll/fd_unix.go:157 +0x17d net.(netFD).Read(0xc422980780, 0xc420023d20, 0x10, 0x10, 0x0, 0x0, 0x0) /data/u4/boyi.gw/tools/go1.10.3/src/net/fd_unix.go:202 +0x4f net.(conn).Read(0xc4200b4168, 0xc420023d20, 0x10, 0x10, 0x0, 0x0, 0x0) /data/u4/boyi.gw/tools/go1.10.3/src/net/net.go:176 +0x6a vendor/github.com/vinllen/mgo.fill(0xa20200, 0xc4200b4168, 0xc420023d20, 0x10, 0x10, 0x0, 0x8e) /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/socket.go:601 +0x53 vendor/github.com/vinllen/mgo.(mongoSocket).readLoop(0xc4228b07e0) /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/socket.go:619 +0x122 created by vendor/github.com/vinllen/mgo.newSocket /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/socket.go:211 +0x1f5

goroutine 63040 [IO wait]: internal/poll.runtime_pollWait(0x7f418477c5a8, 0x72, 0xc422a11c88) /data/u4/boyi.gw/tools/go1.10.3/src/runtime/netpoll.go:173 +0x57 internal/poll.(pollDesc).wait(0xc4228ce998, 0x72, 0xffffffffffffff00, 0xa19f20, 0xc576a0) /data/u4/boyi.gw/tools/go1.10.3/src/internal/poll/fd_poll_runtime.go:85 +0x9b internal/poll.(pollDesc).waitRead(0xc4228ce998, 0xc4202da900, 0x10, 0x10) /data/u4/boyi.gw/tools/go1.10.3/src/internal/poll/fd_poll_runtime.go:90 +0x3d internal/poll.(FD).Read(0xc4228ce980, 0xc4202da910, 0x10, 0x10, 0x0, 0x0, 0x0) /data/u4/boyi.gw/tools/go1.10.3/src/internal/poll/fd_unix.go:157 +0x17d net.(netFD).Read(0xc4228ce980, 0xc4202da910, 0x10, 0x10, 0x0, 0x0, 0x0) /data/u4/boyi.gw/tools/go1.10.3/src/net/fd_unix.go:202 +0x4f net.(conn).Read(0xc420138128, 0xc4202da910, 0x10, 0x10, 0x0, 0x0, 0x0) /data/u4/boyi.gw/tools/go1.10.3/src/net/net.go:176 +0x6a vendor/github.com/vinllen/mgo.fill(0xa20200, 0xc420138128, 0xc4202da910, 0x10, 0x10, 0x0, 0x8e) /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/socket.go:601 +0x53 vendor/github.com/vinllen/mgo.(mongoSocket).readLoop(0xc42030e5a0) /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/socket.go:619 +0x122 created by vendor/github.com/vinllen/mgo.newSocket /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/socket.go:211 +0x1f5

goroutine 118891 [IO wait]: internal/poll.runtime_pollWait(0x7f418477c198, 0x72, 0xc4200e3658) /data/u4/boyi.gw/tools/go1.10.3/src/runtime/netpoll.go:173 +0x57 internal/poll.(pollDesc).wait(0xc422980018, 0x72, 0xffffffffffffff00, 0xa19f20, 0xc576a0) /data/u4/boyi.gw/tools/go1.10.3/src/internal/poll/fd_poll_runtime.go:85 +0x9b internal/poll.(pollDesc).waitRead(0xc422980018, 0xc420359800, 0x1, 0x1) /data/u4/boyi.gw/tools/go1.10.3/src/internal/poll/fd_poll_runtime.go:90 +0x3d internal/poll.(FD).Read(0xc422980000, 0xc420359871, 0x1, 0x1, 0x0, 0x0, 0x0) /data/u4/boyi.gw/tools/go1.10.3/src/internal/poll/fd_unix.go:157 +0x17d net.(netFD).Read(0xc422980000, 0xc420359871, 0x1, 0x1, 0x0, 0x0, 0x0) /data/u4/boyi.gw/tools/go1.10.3/src/net/fd_unix.go:202 +0x4f net.(conn).Read(0xc422830038, 0xc420359871, 0x1, 0x1, 0x0, 0x0, 0x0) /data/u4/boyi.gw/tools/go1.10.3/src/net/net.go:176 +0x6a net/http.(connReader).backgroundRead(0xc420359860) /data/u4/boyi.gw/tools/go1.10.3/src/net/http/server.go:668 +0x5a created by net/http.(*connReader).startBackgroundRead /data/u4/boyi.gw/tools/go1.10.3/src/net/http/server.go:664 +0xce

goroutine 63009 [sleep]: time.Sleep(0x37e11d600) /data/u4/boyi.gw/tools/go1.10.3/src/runtime/time.go:102 +0x166 vendor/github.com/vinllen/mgo.(*mongoServer).pinger(0xc4202bf770, 0xc4201cdb01) /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/server.go:426 +0x4b3 created by vendor/github.com/vinllen/mgo.newServer /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/server.go:107 +0x186

goroutine 62958 [IO wait]: internal/poll.runtime_pollWait(0x7f41847f9af0, 0x72, 0xc420378c88) /data/u4/boyi.gw/tools/go1.10.3/src/runtime/netpoll.go:173 +0x57 internal/poll.(pollDesc).wait(0xc42024c598, 0x72, 0xffffffffffffff00, 0xa19f20, 0xc576a0) /data/u4/boyi.gw/tools/go1.10.3/src/internal/poll/fd_poll_runtime.go:85 +0x9b internal/poll.(pollDesc).waitRead(0xc42024c598, 0xc420275b00, 0x10, 0x10) /data/u4/boyi.gw/tools/go1.10.3/src/internal/poll/fd_poll_runtime.go:90 +0x3d internal/poll.(FD).Read(0xc42024c580, 0xc420275bd0, 0x10, 0x10, 0x0, 0x0, 0x0) /data/u4/boyi.gw/tools/go1.10.3/src/internal/poll/fd_unix.go:157 +0x17d net.(netFD).Read(0xc42024c580, 0xc420275bd0, 0x10, 0x10, 0x0, 0x0, 0x0) /data/u4/boyi.gw/tools/go1.10.3/src/net/fd_unix.go:202 +0x4f net.(conn).Read(0xc4202d0140, 0xc420275bd0, 0x10, 0x10, 0x0, 0x0, 0x0) /data/u4/boyi.gw/tools/go1.10.3/src/net/net.go:176 +0x6a vendor/github.com/vinllen/mgo.fill(0xa20200, 0xc4202d0140, 0xc420275bd0, 0x10, 0x10, 0x0, 0x8e) /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/socket.go:601 +0x53 vendor/github.com/vinllen/mgo.(mongoSocket).readLoop(0xc42295b0e0) /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/socket.go:619 +0x122 created by vendor/github.com/vinllen/mgo.newSocket /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/socket.go:211 +0x1f5

goroutine 103355 [IO wait, 175 minutes]: internal/poll.runtime_pollWait(0x7f41847f9200, 0x72, 0xc4200edc88) /data/u4/boyi.gw/tools/go1.10.3/src/runtime/netpoll.go:173 +0x57 internal/poll.(pollDesc).wait(0xc422980118, 0x72, 0xffffffffffffff00, 0xa19f20, 0xc576a0) /data/u4/boyi.gw/tools/go1.10.3/src/internal/poll/fd_poll_runtime.go:85 +0x9b internal/poll.(pollDesc).waitRead(0xc422980118, 0xc4201ba600, 0x10, 0x10) /data/u4/boyi.gw/tools/go1.10.3/src/internal/poll/fd_poll_runtime.go:90 +0x3d internal/poll.(FD).Read(0xc422980100, 0xc4201ba680, 0x10, 0x10, 0x0, 0x0, 0x0) /data/u4/boyi.gw/tools/go1.10.3/src/internal/poll/fd_unix.go:157 +0x17d net.(netFD).Read(0xc422980100, 0xc4201ba680, 0x10, 0x10, 0x0, 0x0, 0x0) /data/u4/boyi.gw/tools/go1.10.3/src/net/fd_unix.go:202 +0x4f net.(conn).Read(0xc420138028, 0xc4201ba680, 0x10, 0x10, 0x0, 0x0, 0x0) /data/u4/boyi.gw/tools/go1.10.3/src/net/net.go:176 +0x6a vendor/github.com/vinllen/mgo.fill(0xa20200, 0xc420138028, 0xc4201ba680, 0x10, 0x10, 0x0, 0x8e) /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/socket.go:601 +0x53 vendor/github.com/vinllen/mgo.(mongoSocket).readLoop(0xc42024e5a0) /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/socket.go:619 +0x122 created by vendor/github.com/vinllen/mgo.newSocket /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/socket.go:211 +0x1f5

vinllen commented 5 years ago

这个应该是collector的堆栈，你打一下receiver的，用receiver.conf中对应的system_profile端口：

# profiling on net/http/profile
system_profile = 9500
curl http://127.0.0.1:$system_profile/debug/pprof/goroutine?debug=2
liunianjie commented 5 years ago

[root@localhost mongo-shake-2.0.7]# curl http://127.0.0.1:9500/debug/pprof/goroutine?debug=2 goroutine 7199 [running]: runtime/pprof.writeGoroutineStacks(0x956a40, 0xc4201b21c0, 0x411cd9, 0xc42028c630) /data/u4/boyi.gw/tools/go1.10.3/src/runtime/pprof/pprof.go:650 +0xa7 runtime/pprof.writeGoroutine(0x956a40, 0xc4201b21c0, 0x2, 0xc42008ec00, 0x956840) /data/u4/boyi.gw/tools/go1.10.3/src/runtime/pprof/pprof.go:639 +0x44 runtime/pprof.(Profile).WriteTo(0xb8b8c0, 0x956a40, 0xc4201b21c0, 0x2, 0xc4201b21c0, 0x8fd1b1) /data/u4/boyi.gw/tools/go1.10.3/src/runtime/pprof/pprof.go:310 +0x3e4 net/http/pprof.handler.ServeHTTP(0xc4202a83d1, 0x9, 0x95a660, 0xc4201b21c0, 0xc4201f2200) /data/u4/boyi.gw/tools/go1.10.3/src/net/http/pprof/pprof.go:243 +0x20d net/http/pprof.Index(0x95a660, 0xc4201b21c0, 0xc4201f2200) /data/u4/boyi.gw/tools/go1.10.3/src/net/http/pprof/pprof.go:254 +0x1d1 net/http.HandlerFunc.ServeHTTP(0x91b248, 0x95a660, 0xc4201b21c0, 0xc4201f2200) /data/u4/boyi.gw/tools/go1.10.3/src/net/http/server.go:1947 +0x44 net/http.(ServeMux).ServeHTTP(0xb99e00, 0x95a660, 0xc4201b21c0, 0xc4201f2200) /data/u4/boyi.gw/tools/go1.10.3/src/net/http/server.go:2337 +0x130 net/http.serverHandler.ServeHTTP(0xc4201b0dd0, 0x95a660, 0xc4201b21c0, 0xc4201f2200) /data/u4/boyi.gw/tools/go1.10.3/src/net/http/server.go:2694 +0xbc net/http.(conn).serve(0xc4201f60a0, 0x95ad20, 0xc4204ce000) /data/u4/boyi.gw/tools/go1.10.3/src/net/http/server.go:1830 +0x651 created by net/http.(Server).Serve /data/u4/boyi.gw/tools/go1.10.3/src/net/http/server.go:2795 +0x27b

goroutine 1 [select (no cases), 46 minutes]: main.main() /home/zhuzhao.cx/mongo-shake/src/mongoshake/receiver/main/receiver.go:65 +0x6de

goroutine 19 [chan receive, 46 minutes]: vendor/github.com/vinllen/log4go.(*ConsoleLogWriter).run(0xc4200c20a0, 0x956b40, 0xc4200b6008) /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/log4go/termlog.go:33 +0x10c created by vendor/github.com/vinllen/log4go.NewConsoleLogWriter /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/log4go/termlog.go:26 +0xbf

goroutine 20 [select]: vendor/github.com/vinllen/mgo.newcoarseTimeProvider.func1(0x17d7840, 0xc4200aa930) /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/coarse_time.go:51 +0x157 created by vendor/github.com/vinllen/mgo.newcoarseTimeProvider /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/mgo/coarse_time.go:48 +0x107

goroutine 21 [syscall, 46 minutes]: os/signal.signal_recv(0x0) /data/u4/boyi.gw/tools/go1.10.3/src/runtime/sigqueue.go:139 +0xa6 os/signal.loop() /data/u4/boyi.gw/tools/go1.10.3/src/os/signal/signal_unix.go:22 +0x22 created by os/signal.init.0 /data/u4/boyi.gw/tools/go1.10.3/src/os/signal/signal_unix.go:28 +0x41

goroutine 36 [select, 46 minutes]: vendor/github.com/vinllen/log4go.NewFileLogWriter.func1(0xc4201f6000) /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/log4go/filelog.go:87 +0x136 created by vendor/github.com/vinllen/log4go.NewFileLogWriter /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/log4go/filelog.go:78 +0x23d

goroutine 35 [chan receive, 46 minutes]: vendor/github.com/vinllen/log4go.(*ConsoleLogWriter).run(0xc42017e360, 0x956b40, 0xc4200b6008) /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/log4go/termlog.go:33 +0x10c created by vendor/github.com/vinllen/log4go.NewConsoleLogWriter /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/vinllen/log4go/termlog.go:26 +0xbf

goroutine 37 [IO wait]: internal/poll.runtime_pollWait(0x7f0b7727cf00, 0x72, 0x0) /data/u4/boyi.gw/tools/go1.10.3/src/runtime/netpoll.go:173 +0x57 internal/poll.(pollDesc).wait(0xc4200f4118, 0x72, 0xc4200bb600, 0x0, 0x0) /data/u4/boyi.gw/tools/go1.10.3/src/internal/poll/fd_poll_runtime.go:85 +0x9b internal/poll.(pollDesc).waitRead(0xc4200f4118, 0xffffffffffffff00, 0x0, 0x0) /data/u4/boyi.gw/tools/go1.10.3/src/internal/poll/fd_poll_runtime.go:90 +0x3d internal/poll.(FD).Accept(0xc4200f4100, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /data/u4/boyi.gw/tools/go1.10.3/src/internal/poll/fd_unix.go:372 +0x1a8 net.(netFD).accept(0xc4200f4100, 0xc4201f6120, 0xc420061db8, 0x402a68) /data/u4/boyi.gw/tools/go1.10.3/src/net/fd_unix.go:238 +0x42 net.(TCPListener).accept(0xc4200b6568, 0xc420061de8, 0x4019c7, 0xc4201f6120) /data/u4/boyi.gw/tools/go1.10.3/src/net/tcpsock_posix.go:136 +0x2e net.(TCPListener).AcceptTCP(0xc4200b6568, 0xc420061e30, 0xc420061e38, 0x18) /data/u4/boyi.gw/tools/go1.10.3/src/net/tcpsock.go:246 +0x49 net/http.tcpKeepAliveListener.Accept(0xc4200b6568, 0x91b048, 0xc4201f60a0, 0x95ade0, 0xc4201aea20) /data/u4/boyi.gw/tools/go1.10.3/src/net/http/server.go:3216 +0x2f net/http.(Server).Serve(0xc4201b0dd0, 0x95a9e0, 0xc4200b6568, 0x0, 0x0) /data/u4/boyi.gw/tools/go1.10.3/src/net/http/server.go:2770 +0x1a5 net/http.(Server).ListenAndServe(0xc4201b0dd0, 0xc4201b0dd0, 0xc42019c000) /data/u4/boyi.gw/tools/go1.10.3/src/net/http/server.go:2711 +0xa9 net/http.ListenAndServe(0xc4200b2848, 0x5, 0x0, 0x0, 0x1, 0xc4200b2848) /data/u4/boyi.gw/tools/go1.10.3/src/net/http/server.go:2969 +0x7a vendor/github.com/gugemichael/nimo4go.Profiling.func1() /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/gugemichael/nimo4go/profile.go:22 +0xb5 vendor/github.com/gugemichael/nimo4go.GoRoutine.func1(0xc42023bf80) /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/gugemichael/nimo4go/runtime.go:12 +0x25 created by vendor/github.com/gugemichael/nimo4go.GoRoutine /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/gugemichael/nimo4go/runtime.go:11 +0x3f

goroutine 38 [select, 46 minutes, locked to thread]: runtime.gopark(0x91b640, 0x0, 0x8f932b, 0x6, 0x18, 0x1) /data/u4/boyi.gw/tools/go1.10.3/src/runtime/proc.go:291 +0x11a runtime.selectgo(0xc42018b750, 0xc4201e4240) /data/u4/boyi.gw/tools/go1.10.3/src/runtime/select.go:392 +0xe50 runtime.ensureSigM.func1() /data/u4/boyi.gw/tools/go1.10.3/src/runtime/signal_unix.go:549 +0x1f4 runtime.goexit() /data/u4/boyi.gw/tools/go1.10.3/src/runtime/asm_amd64.s:2361 +0x1

goroutine 39 [chan receive, 46 minutes]: vendor/github.com/gugemichael/nimo4go.RegisterSignalForProfiling.func1(0xc4201e4180, 0xc42023bf76) /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/gugemichael/nimo4go/profile.go:46 +0x63 created by vendor/github.com/gugemichael/nimo4go.RegisterSignalForProfiling /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/gugemichael/nimo4go/profile.go:44 +0xb6

goroutine 25 [chan receive, 46 minutes]: vendor/github.com/gugemichael/nimo4go.RegisterSignalForPrintStack.func1(0xc4201e4300, 0x91ae40) /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/gugemichael/nimo4go/profile.go:31 +0xcd created by vendor/github.com/gugemichael/nimo4go.RegisterSignalForPrintStack /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/gugemichael/nimo4go/profile.go:30 +0xb1

goroutine 26 [chan receive, 46 minutes]: mongoshake/receiver.(*ExampleReplayer).handler(0xc4201aea80) /home/zhuzhao.cx/mongo-shake/src/mongoshake/receiver/replayer.go:113 +0x6f created by mongoshake/receiver.NewExampleReplayer /home/zhuzhao.cx/mongo-shake/src/mongoshake/receiver/replayer.go:42 +0xf3

goroutine 27 [chan receive, 46 minutes]: mongoshake/receiver.(*ExampleReplayer).handler(0xc4201aeab0) /home/zhuzhao.cx/mongo-shake/src/mongoshake/receiver/replayer.go:113 +0x6f created by mongoshake/receiver.NewExampleReplayer /home/zhuzhao.cx/mongo-shake/src/mongoshake/receiver/replayer.go:42 +0xf3

goroutine 28 [chan receive, 46 minutes]: mongoshake/receiver.(*ExampleReplayer).handler(0xc4201aeae0) /home/zhuzhao.cx/mongo-shake/src/mongoshake/receiver/replayer.go:113 +0x6f created by mongoshake/receiver.NewExampleReplayer /home/zhuzhao.cx/mongo-shake/src/mongoshake/receiver/replayer.go:42 +0xf3

goroutine 29 [chan receive, 46 minutes]: mongoshake/receiver.(*ExampleReplayer).handler(0xc4201aeb10) /home/zhuzhao.cx/mongo-shake/src/mongoshake/receiver/replayer.go:113 +0x6f created by mongoshake/receiver.NewExampleReplayer /home/zhuzhao.cx/mongo-shake/src/mongoshake/receiver/replayer.go:42 +0xf3

goroutine 30 [chan receive, 46 minutes]: mongoshake/receiver.(*ExampleReplayer).handler(0xc4201aeb40) /home/zhuzhao.cx/mongo-shake/src/mongoshake/receiver/replayer.go:113 +0x6f created by mongoshake/receiver.NewExampleReplayer /home/zhuzhao.cx/mongo-shake/src/mongoshake/receiver/replayer.go:42 +0xf3

goroutine 31 [chan receive, 46 minutes]: mongoshake/receiver.(*ExampleReplayer).handler(0xc4201aeb70) /home/zhuzhao.cx/mongo-shake/src/mongoshake/receiver/replayer.go:113 +0x6f created by mongoshake/receiver.NewExampleReplayer /home/zhuzhao.cx/mongo-shake/src/mongoshake/receiver/replayer.go:42 +0xf3

goroutine 32 [chan receive, 46 minutes]: mongoshake/receiver.(*ExampleReplayer).handler(0xc4201aeba0) /home/zhuzhao.cx/mongo-shake/src/mongoshake/receiver/replayer.go:113 +0x6f created by mongoshake/receiver.NewExampleReplayer /home/zhuzhao.cx/mongo-shake/src/mongoshake/receiver/replayer.go:42 +0xf3

goroutine 33 [chan receive, 46 minutes]: mongoshake/receiver.(*ExampleReplayer).handler(0xc4201aebd0) /home/zhuzhao.cx/mongo-shake/src/mongoshake/receiver/replayer.go:113 +0x6f created by mongoshake/receiver.NewExampleReplayer /home/zhuzhao.cx/mongo-shake/src/mongoshake/receiver/replayer.go:42 +0xf3

goroutine 66 [chan receive, 2 minutes]: vendor/github.com/rcrowley/go-metrics.(*meterArbiter).tick(0xb9a040) /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/rcrowley/go-metrics/meter.go:245 +0x31 created by vendor/github.com/rcrowley/go-metrics.NewMeter /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/rcrowley/go-metrics/meter.go:46 +0x10b

goroutine 67 [chan receive]: vendor/github.com/Shopify/sarama.(Broker).responseReceiver(0xc42011cb00) /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/Shopify/sarama/broker.go:682 +0xf7 vendor/github.com/Shopify/sarama.(Broker).(vendor/github.com/Shopify/sarama.responseReceiver)-fm() /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/Shopify/sarama/broker.go:149 +0x2a vendor/github.com/Shopify/sarama.withRecover(0xc420294200) /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/Shopify/sarama/utils.go:45 +0x43 created by vendor/github.com/Shopify/sarama.(*Broker).Open.func1 /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/Shopify/sarama/broker.go:149 +0x878

goroutine 69 [select, 8 minutes]: vendor/github.com/Shopify/sarama.(client).backgroundMetadataUpdater(0xc4201ac120) /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/Shopify/sarama/client.go:633 +0x156 vendor/github.com/Shopify/sarama.(client).(vendor/github.com/Shopify/sarama.backgroundMetadataUpdater)-fm() /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/Shopify/sarama/client.go:163 +0x2a vendor/github.com/Shopify/sarama.withRecover(0xc420294280) /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/Shopify/sarama/utils.go:45 +0x43 created by vendor/github.com/Shopify/sarama.NewClient /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/Shopify/sarama/client.go:163 +0x52e

goroutine 72 [select]: vendor/github.com/Shopify/sarama.(partitionConsumer).dispatcher(0xc420288510) /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/Shopify/sarama/consumer.go:337 +0x11e vendor/github.com/Shopify/sarama.(partitionConsumer).(vendor/github.com/Shopify/sarama.dispatcher)-fm() /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/Shopify/sarama/consumer.go:163 +0x2a vendor/github.com/Shopify/sarama.withRecover(0xc420294310) /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/Shopify/sarama/utils.go:45 +0x43 created by vendor/github.com/Shopify/sarama.(*consumer).ConsumePartition /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/Shopify/sarama/consumer.go:163 +0x2d7

goroutine 73 [chan receive, 46 minutes]: vendor/github.com/Shopify/sarama.(partitionConsumer).responseFeeder(0xc420288510) /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/Shopify/sarama/consumer.go:451 +0x2a0 vendor/github.com/Shopify/sarama.(partitionConsumer).(vendor/github.com/Shopify/sarama.responseFeeder)-fm() /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/Shopify/sarama/consumer.go:164 +0x2a vendor/github.com/Shopify/sarama.withRecover(0xc420294320) /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/Shopify/sarama/utils.go:45 +0x43 created by vendor/github.com/Shopify/sarama.(*consumer).ConsumePartition /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/Shopify/sarama/consumer.go:164 +0x32e

goroutine 6 [chan receive, 46 minutes]: mongoshake/tunnel/kafka.(*Reader).send(0xc4201ce2d0) /home/zhuzhao.cx/mongo-shake/src/mongoshake/tunnel/kafka/reader.go:53 +0x86 created by mongoshake/tunnel/kafka.NewReader /home/zhuzhao.cx/mongo-shake/src/mongoshake/tunnel/kafka/reader.go:44 +0x20b

goroutine 7 [chan receive, 46 minutes]: mongoshake/tunnel.(KafkaReader).replay(0xc4201aea50) /home/zhuzhao.cx/mongo-shake/src/mongoshake/tunnel/kafka_reader.go:40 +0x6d3 created by mongoshake/tunnel.(KafkaReader).Link /home/zhuzhao.cx/mongo-shake/src/mongoshake/tunnel/kafka_reader.go:27 +0x18e

goroutine 7219 [chan receive]: vendor/github.com/Shopify/sarama.(brokerConsumer).abort(0xc4201ce230, 0x957cc0, 0xc42027e1a0) /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/Shopify/sarama/consumer.go:775 +0x201 vendor/github.com/Shopify/sarama.(brokerConsumer).subscriptionConsumer(0xc4201ce230) /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/Shopify/sarama/consumer.go:696 +0x342 vendor/github.com/Shopify/sarama.(brokerConsumer).(vendor/github.com/Shopify/sarama.subscriptionConsumer)-fm() /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/Shopify/sarama/consumer.go:633 +0x2a vendor/github.com/Shopify/sarama.withRecover(0xc4201cc070) /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/Shopify/sarama/utils.go:45 +0x43 created by vendor/github.com/Shopify/sarama.(consumer).newBrokerConsumer /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/Shopify/sarama/consumer.go:633 +0x1bb

goroutine 7218 [select]: vendor/github.com/Shopify/sarama.(brokerConsumer).subscriptionManager(0xc4201ce230) /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/Shopify/sarama/consumer.go:659 +0x3de vendor/github.com/Shopify/sarama.(brokerConsumer).(vendor/github.com/Shopify/sarama.subscriptionManager)-fm() /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/Shopify/sarama/consumer.go:632 +0x2a vendor/github.com/Shopify/sarama.withRecover(0xc4201cc060) /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/Shopify/sarama/utils.go:45 +0x43 created by vendor/github.com/Shopify/sarama.(*consumer).newBrokerConsumer /home/zhuzhao.cx/mongo-shake/src/vendor/github.com/Shopify/sarama/consumer.go:632 +0x168

goroutine 7200 [runnable]: syscall.Syscall(0x0, 0x8, 0xc42028c431, 0x1, 0xffffffffffffffff, 0x0, 0xb) /data/u4/boyi.gw/tools/go1.10.3/src/syscall/asm_linux_amd64.s:18 +0x5 syscall.read(0x8, 0xc42028c431, 0x1, 0x1, 0xc4204cfe00, 0x0, 0x0) /data/u4/boyi.gw/tools/go1.10.3/src/syscall/zsyscall_linux_amd64.go:749 +0x5f syscall.Read(0x8, 0xc42028c431, 0x1, 0x1, 0x450101, 0xc420054790, 0x786992) /data/u4/boyi.gw/tools/go1.10.3/src/syscall/syscall_unix.go:162 +0x49 internal/poll.(FD).Read(0xc420082000, 0xc42028c431, 0x1, 0x1, 0x0, 0x0, 0x0) /data/u4/boyi.gw/tools/go1.10.3/src/internal/poll/fd_unix.go:153 +0x118 net.(netFD).Read(0xc420082000, 0xc42028c431, 0x1, 0x1, 0xc42036ede0, 0xb9a320, 0x590) /data/u4/boyi.gw/tools/go1.10.3/src/net/fd_unix.go:202 +0x4f net.(conn).Read(0xc42000e048, 0xc42028c431, 0x1, 0x1, 0x0, 0x0, 0x0) /data/u4/boyi.gw/tools/go1.10.3/src/net/net.go:176 +0x6a net/http.(connReader).backgroundRead(0xc42028c420) /data/u4/boyi.gw/tools/go1.10.3/src/net/http/server.go:668 +0x5a created by net/http.(*connReader).startBackgroundRead /data/u4/boyi.gw/tools/go1.10.3/src/net/http/server.go:664 +0xce

vinllen commented 5 years ago

image

goroutine 6 [chan receive, 46 minutes]:
mongoshake/tunnel/kafka.(*Reader).send(0xc4201ce2d0)
/home/zhuzhao.cx/mongo-shake/src/mongoshake/tunnel/kafka/reader.go:53 +0x86
created by mongoshake/tunnel/kafka.NewReader
/home/zhuzhao.cx/mongo-shake/src/mongoshake/tunnel/kafka/reader.go:44 +0x20b

这个堆栈看起来跟之前用户是一样的，driver没有读到数据。你的kafka版本是多少，另外数据是有写入的吗？

liunianjie commented 5 years ago

kafkatool工具跟自己java写的consumer都是可以正确消费的, kafka版本2.12-2.3.0

vinllen commented 5 years ago

看起来这块是有问题，有几个用户反馈同样的问题。这块我先看一下。方便的话你加下我微信（README最后）好了，方便沟通。

vinllen commented 5 years ago

@liunianjie 完整的运行日志也发一下吧

vinllen commented 5 years ago

完整启动日志：

[2019/09/06 15:39:09 CST] [INFO] [receiver.NewExampleReplayer:37] ExampleReplayer start. pending queue capacity 256
[2019/09/06 15:39:09 CST] [INFO] [receiver.NewExampleReplayer:37] ExampleReplayer start. pending queue capacity 256
[2019/09/06 15:39:09 CST] [INFO] [receiver.NewExampleReplayer:37] ExampleReplayer start. pending queue capacity 256
[2019/09/06 15:39:09 CST] [INFO] [receiver.NewExampleReplayer:37] ExampleReplayer start. pending queue capacity 256
[2019/09/06 15:39:09 CST] [INFO] [receiver.NewExampleReplayer:37] ExampleReplayer start. pending queue capacity 256
[2019/09/06 15:39:09 CST] [INFO] [receiver.NewExampleReplayer:37] ExampleReplayer start. pending queue capacity 256
[2019/09/06 15:39:09 CST] [INFO] [receiver.NewExampleReplayer:37] ExampleReplayer start. pending queue capacity 256
[2019/09/06 15:39:09 CST] [INFO] [receiver.NewExampleReplayer:37] ExampleReplayer start. pending queue capacity 256
[2019/09/06 15:39:09 CST] [INFO] [main.startup:96] receiver is starting...
vinllen commented 5 years ago

问题已定位，驱动代码存在问题，无法适配目前版本的kafka，已更新驱动代码解决，将会release在v2.0.8版本

zlklink commented 5 years ago

使用的如何？

发自我的iPhone

------------------ 原始邮件 ------------------ 发件人: 杨智华 <notifications@github.com> 发送时间: 2019年8月16日 17:15 收件人: alibaba/MongoShake <MongoShake@noreply.github.com> 抄送: zlklink <1067176522@qq.com>, Mention <mention@noreply.github.com> 主题: 回复：[alibaba/MongoShake] 关于receiver的使用 (#212)

最新的配置文件

global log level. lower level message

will be filter

log.level = info

log file name. use logs/ prefix folder path

log.file = reeciver.log

log buffer or unbuffer. If set true, logs may not be print when exit. If

set false, performance will be decreased

log.buffer = true

profiling on net/http/profile

system_profile = 9500

tunnel pipeline type. now we support rpc,tcp,file,mock,kafka

tunnel = kafka

tunnel target resource url

for rpc. this is receiver socket address

for tcp. this is receiver socket address

for file. this is the file path, for instance "data"

for mock. this is useless. mongoshake will generate random data including "i", "d", "u", "n"

for kafka. this is the topic and brokers address which split by comma, for

instance: topic@brokers1,brokers2, default topic is "mongoshake"

tunnel.address = mongoshake@10.0.0.35:9092

replayer worker concurrency. must equal to the collector worker number

replayer = 8

有部分下划线需要改为点

@vinllen ./receiver -conf=./receiver.conf -verbose 这样输出内容在控制台上面，我们如何调用呢？ 有通过api获取数据的方式？

[17:04:27 CST 2019/08/16] [INFO] (mongoshake/receiver.(ExampleReplayer).handler:130) &{6725671133426745374 d clue_dw.a [{_id ObjectIdHex("5d564f08a3980674c891e1ea")}] map[] map[] false map[] 138 0} [17:04:27 CST 2019/08/16] [INFO] (mongoshake/receiver.(ExampleReplayer).handler:130) &{6725671249390862337 i clue_dw.a [{_id ObjectIdHex("5d56625ef0094137c1f9697d")} {a 1}] map[] map[] false map[] 145 0} [17:04:27 CST 2019/08/16] [INFO] (mongoshake/receiver.(ExampleReplayer).handler:130) &{6725674208623329281 i clue_dw.a [{_id ObjectIdHex("5d56650ff0094137c1f96aa8")} {c 1}] map[] map[] false map[] 145 0} [17:05:00 CST 2019/08/16] [INFO] (mongoshake/receiver.(ExampleReplayer).handler:130) &{6725688141497237505 i clue_dw.a [{_id ObjectIdHex("5d5671bbf0094137c1f97041")} {yzh 1}] map[] map[] false map[] 147 0}

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

vinllen commented 5 years ago

v2.0.8版本已经发布

TIMPLIN commented 2 years ago

麻烦问下老哥，最后是怎么解析raw中的信息的，这个receiver源码你是怎么改的呀

yzhbest commented 2 years ago

邮件已收到，我会尽快给您答复！ 感谢你！