search

hyperledger-archives / fabric

THIS IS A READ-ONLY historic repository. Current development is at https://gerrit.hyperledger.org/r/#/admin/projects/fabric . pull requests not accepted
https://gerrit.hyperledger.org/
Apache License 2.0
1.17k stars 1.01k forks source link

Occasional peer crash #412

Closed frosenberg closed 8 years ago

frosenberg commented 8 years ago

I don't see this all the time but occasionally (3+ times at least), when deploying example-chaincode2. I'm running the peer in a docker container in the provided vagrant machine.

20:19:54.633 [main] serve -> DEBU 006 Running as validating peer - installing consensus noops
20:19:54.634 [db] createDBIfDBPathEmpty -> DEBU 007 Is db path [/var/openchain/production/db] empty [true]
20:19:54.635 [db] CreateDB -> DEBU 008 Creating DB at [/var/openchain/production/db]
20:19:54.648 [db] CreateDB -> DEBU 009 DB created at [/var/openchain/production/db]
20:19:54.659 [buckettree] Initialize -> INFO 00a Initializing bucket tree state implemetation with configurations &{maxGroupingAtEachLevel:10 lowestLevel:5 levelToNumBucketsMap:map[1:2 0:1 5:10009 4:1001 3:101 2:11] hashFunc:0xa01760}
20:19:54.661 [peer] chatWithPeer -> DEBU 00b Initiating Chat with peer address: 172.17.0.2:30303
20:19:54.668 [chaincode] NewChaincodeSupport -> INFO 00c Chaincode support using peerAddress: 0.0.0.0:30303
20:19:54.668 [server] NewOpenchainServer -> INFO 00d
Creating new OpenchainServer...
20:19:54.670 [main] serve -> INFO 00e Starting peer with id=name:"dev-fr2" , network id=dev, address=0.0.0.0:30303, discovery.rootnode=172.17.0.2:30303, validator=true
20:19:54.672 [genesis] func1 -> INFO 00f Creating genesis block.
20:19:54.672 [genesis] func1 -> INFO 010 No genesis block chaincodes defined.
20:19:54.673 [state] GetHash -> DEBU 011 Enter - GetHash()
20:19:54.673 [buckettree] ComputeCryptoHash -> DEBU 012 Enter - ComputeCryptoHash()
20:19:54.673 [buckettree] ComputeCryptoHash -> DEBU 013 Returing existing crypto-hash as recomputation not required
20:19:54.673 [state] GetHash -> DEBU 014 Exit - GetHash()
20:19:54.675 [indexes] addIndexDataForPersistence -> DEBU 015 Indexing block number [0] by hash = [dde954e7f6822485626092dce7c1c464747fe4f8795ff3045e1cb63e57395ee82993b9c441ec91069515975da8543271de4594b48488fe2540eca6c023f136fb]
20:19:54.675 [state] AddChangesForPersistence -> DEBU 016 state.addChangesForPersistence()...start
20:19:54.678 [peer] chatWithPeer -> DEBU 017 Established Chat with peer address: 172.17.0.2:30303
20:19:54.678 [peer] handleChat -> DEBU 018 Current context deadline = 0001-01-01 00:00:00 +0000 UTC, ok = false
20:19:54.682 [state] AddChangesForPersistence -> DEBU 019 Adding state-delta corresponding to block number[0]
20:19:54.682 [state] AddChangesForPersistence -> DEBU 01a Not deleting previous state-delta. Block number [0] is smaller than historyStateDeltaSize [500]
20:19:54.683 [state] AddChangesForPersistence -> DEBU 01b state.addChangesForPersistence()...finished
20:19:54.690 [ledger] resetForNextTxGroup -> DEBU 01c resetting ledger state for next transaction batch
20:19:54.693 [buckettree] ClearWorkingSet -> DEBU 01e Enter - ClearWorkingSet()
20:19:54.692 [consensus/controller] NewConsenter -> DEBU 01d Running with NOOPS consensus
20:19:54.693 [consensus/noops] NewNoops -> DEBU 01f Creating a NOOPS object
20:19:54.694 [consensus/handler] HandleMessage -> DEBU 020 Did not handle message of type DISC_HELLO, passing on to next MessageHandler
20:19:54.694 [peer] HandleMessage -> DEBU 021 Handling OpenchainMessage of type: DISC_HELLO
20:19:54.694 [peer] beforeHello -> DEBU 022 Received DISC_HELLO, parsing out Peer identification
20:19:54.694 [peer] beforeHello -> DEBU 023 Received DISC_HELLO from endpoint=peerEndpoint:<ID:<name:"dev-fr1" > address:"0.0.0.0:30303" type:VALIDATOR > blockNumber:1
20:19:54.694 [peer] RegisterHandler -> DEBU 024 registered handler with key: dev-fr1
20:19:54.694 [peer] enterState -> DEBU 025 The Peer's bi-directional stream to ID:<name:"dev-fr1" > address:"0.0.0.0:30303" type:VALIDATOR  is established, from event DISC_HELLO
20:19:54.696 [peer] start -> DEBU 026 Starting Peer discovery service
20:19:59.697 [consensus/handler] HandleMessage -> DEBU 027 Did not handle message of type DISC_GET_PEERS, passing on to next MessageHandler
20:19:59.698 [peer] HandleMessage -> DEBU 028 Handling OpenchainMessage of type: DISC_GET_PEERS
20:19:59.698 [peer] beforeGetPeers -> DEBU 029 Sending back DISC_PEERS
20:19:59.701 [consensus/handler] HandleMessage -> DEBU 02a Did not handle message of type DISC_PEERS, passing on to next MessageHandler
20:19:59.701 [peer] HandleMessage -> DEBU 02b Handling OpenchainMessage of type: DISC_PEERS
20:19:59.701 [peer] beforePeers -> DEBU 02c Received DISC_PEERS, grabbing peers message
20:19:59.703 [peer] beforePeers -> DEBU 02d Received PeersMessage with Peers: peers:<ID:<name:"dev-fr2" > address:"0.0.0.0:30303" type:VALIDATOR >
20:20:04.692 [consensus/handler] HandleMessage -> DEBU 02e Did not handle message of type DISC_GET_PEERS, passing on to next MessageHandler
20:20:04.692 [peer] HandleMessage -> DEBU 02f Handling OpenchainMessage of type: DISC_GET_PEERS
20:20:04.692 [peer] beforeGetPeers -> DEBU 030 Sending back DISC_PEERS
20:20:04.698 [consensus/handler] HandleMessage -> DEBU 031 Did not handle message of type DISC_PEERS, passing on to next MessageHandler
20:20:04.699 [peer] HandleMessage -> DEBU 032 Handling OpenchainMessage of type: DISC_PEERS
20:20:04.699 [peer] beforePeers -> DEBU 033 Received DISC_PEERS, grabbing peers message
20:20:04.699 [peer] beforePeers -> DEBU 034 Received PeersMessage with Peers: peers:<ID:<name:"dev-fr2" > address:"0.0.0.0:30303" type:VALIDATOR >
20:20:09.692 [consensus/handler] HandleMessage -> DEBU 035 Did not handle message of type DISC_GET_PEERS, passing on to next MessageHandler
20:20:09.695 [peer] HandleMessage -> DEBU 036 Handling OpenchainMessage of type: DISC_GET_PEERS
20:20:09.696 [peer] beforeGetPeers -> DEBU 037 Sending back DISC_PEERS
20:20:09.704 [consensus/handler] HandleMessage -> DEBU 038 Did not handle message of type DISC_PEERS, passing on to next MessageHandler
20:20:09.704 [peer] HandleMessage -> DEBU 039 Handling OpenchainMessage of type: DISC_PEERS
20:20:09.705 [peer] beforePeers -> DEBU 03a Received DISC_PEERS, grabbing peers message
20:20:09.705 [peer] beforePeers -> DEBU 03b Received PeersMessage with Peers: peers:<ID:<name:"dev-fr2" > address:"0.0.0.0:30303" type:VALIDATOR >
20:20:13.958 [devops] Build -> DEBU 03c Received build request for chaincode spec: type:GOLANG chaincodeID:<url:"github.com/openblockchain/obc-peer/openchain/example/chaincode/chaincode_example02" version:"0.0.1" > ctorMsg:<function:"init" args:"a" args:"100" args:"b" args:"200" >
20:20:13.958 [devops] CheckSpec -> DEBU 03d Validated spec:  type:GOLANG chaincodeID:<url:"github.com/openblockchain/obc-peer/openchain/example/chaincode/chaincode_example02" version:"0.0.1" > ctorMsg:<function:"init" args:"a" args:"100" args:"b" args:"200" >
20:20:13.959 [container] NewVM -> INFO 03e Creating VM with endpoint: http://172.17.0.1:4243
20:20:13.959 [container] writeGopathSrc -> INFO 03f rootDirectory = /go/src
20:20:14.692 [consensus/handler] HandleMessage -> DEBU 040 Did not handle message of type DISC_GET_PEERS, passing on to next MessageHandler
20:20:14.693 [peer] HandleMessage -> DEBU 041 Handling OpenchainMessage of type: DISC_GET_PEERS
20:20:14.694 [peer] beforeGetPeers -> DEBU 042 Sending back DISC_PEERS
20:20:14.704 [consensus/handler] HandleMessage -> DEBU 043 Did not handle message of type DISC_PEERS, passing on to next MessageHandler
20:20:14.705 [peer] HandleMessage -> DEBU 044 Handling OpenchainMessage of type: DISC_PEERS
20:20:14.705 [peer] beforePeers -> DEBU 045 Received DISC_PEERS, grabbing peers message
20:20:14.705 [peer] beforePeers -> DEBU 046 Received PeersMessage with Peers: peers:<ID:<name:"dev-fr2" > address:"0.0.0.0:30303" type:VALIDATOR >
20:20:19.692 [consensus/handler] HandleMessage -> DEBU 047 Did not handle message of type DISC_GET_PEERS, passing on to next MessageHandler
20:20:19.692 [peer] HandleMessage -> DEBU 048 Handling OpenchainMessage of type: DISC_GET_PEERS
20:20:19.692 [peer] beforeGetPeers -> DEBU 049 Sending back DISC_PEERS
20:20:19.706 [consensus/handler] HandleMessage -> DEBU 04a Did not handle message of type DISC_PEERS, passing on to next MessageHandler
20:20:19.706 [peer] HandleMessage -> DEBU 04b Handling OpenchainMessage of type: DISC_PEERS
20:20:19.714 [peer] beforePeers -> DEBU 04c Received DISC_PEERS, grabbing peers message
20:20:19.734 [peer] beforePeers -> DEBU 04d Received PeersMessage with Peers: peers:<ID:<name:"dev-fr2" > address:"0.0.0.0:30303" type:VALIDATOR >
20:20:24.692 [consensus/handler] HandleMessage -> DEBU 04e Did not handle message of type DISC_GET_PEERS, passing on to next MessageHandler
20:20:24.692 [peer] HandleMessage -> DEBU 04f Handling OpenchainMessage of type: DISC_GET_PEERS
20:20:24.692 [peer] beforeGetPeers -> DEBU 050 Sending back DISC_PEERS
20:20:24.697 [consensus/handler] HandleMessage -> DEBU 051 Did not handle message of type DISC_PEERS, passing on to next MessageHandler
20:20:24.697 [peer] HandleMessage -> DEBU 052 Handling OpenchainMessage of type: DISC_PEERS
20:20:24.697 [peer] beforePeers -> DEBU 053 Received DISC_PEERS, grabbing peers message
20:20:24.697 [peer] beforePeers -> DEBU 054 Received PeersMessage with Peers: peers:<ID:<name:"dev-fr2" > address:"0.0.0.0:30303" type:VALIDATOR >
20:20:29.691 [consensus/handler] HandleMessage -> DEBU 055 Did not handle message of type DISC_GET_PEERS, passing on to next MessageHandler
20:20:29.693 [peer] HandleMessage -> DEBU 056 Handling OpenchainMessage of type: DISC_GET_PEERS
20:20:29.693 [peer] beforeGetPeers -> DEBU 057 Sending back DISC_PEERS
20:20:29.797 [consensus/handler] HandleMessage -> DEBU 058 Did not handle message of type DISC_PEERS, passing on to next MessageHandler
20:20:29.797 [peer] HandleMessage -> DEBU 059 Handling OpenchainMessage of type: DISC_PEERS
20:20:29.802 [peer] beforePeers -> DEBU 05a Received DISC_PEERS, grabbing peers message
20:20:29.803 [peer] beforePeers -> DEBU 05b Received PeersMessage with Peers: peers:<ID:<name:"dev-fr2" > address:"0.0.0.0:30303" type:VALIDATOR >
20:20:34.692 [consensus/handler] HandleMessage -> DEBU 05c Did not handle message of type DISC_GET_PEERS, passing on to next MessageHandler
20:20:34.693 [peer] HandleMessage -> DEBU 05d Handling OpenchainMessage of type: DISC_GET_PEERS
20:20:34.695 [peer] beforeGetPeers -> DEBU 05e Sending back DISC_PEERS
20:20:34.709 [consensus/handler] HandleMessage -> DEBU 05f Did not handle message of type DISC_PEERS, passing on to next MessageHandler
20:20:34.711 [peer] HandleMessage -> DEBU 060 Handling OpenchainMessage of type: DISC_PEERS
20:20:34.713 [peer] beforePeers -> DEBU 061 Received DISC_PEERS, grabbing peers message
20:20:34.715 [peer] beforePeers -> DEBU 062 Received PeersMessage with Peers: peers:<ID:<name:"dev-fr2" > address:"0.0.0.0:30303" type:VALIDATOR >
20:20:38.355 [devops] Deploy -> DEBU 063 Creating deployment transaction (6e4d3a26-aa09-4d34-8000-790804bbe038)
20:20:38.362 [devops] Deploy -> DEBU 064 Sending deploy transaction (6e4d3a26-aa09-4d34-8000-790804bbe038) to validator
20:20:38.365 [peer] sendTransactionsToThisPeer -> DEBU 065 Sending transaction CHAINCODE_NEW to self
20:20:38.369 [peer] handleChat -> DEBU 066 Current context deadline = 0001-01-01 00:00:00 +0000 UTC, ok = false
20:20:38.372 [consensus/controller] NewConsenter -> DEBU 067 Running with NOOPS consensus
20:20:38.378 [peer] sendTransactionsToThisPeer -> DEBU 068 Sending message CHAIN_TRANSACTION with timestamp seconds:1450815638 nanos:378315806  to self
20:20:38.472 [consensus/handler] SendMessage -> DEBU 069 Sending to stream a message of type: RESPONSE
20:20:38.474 [peer] func1 -> DEBU 06a Received RESPONSE message as expected, exiting out of receive loop
20:20:38.481 [consensus/noops] RecvMsg -> DEBU 06b Handling OpenchainMessage of type: CHAIN_TRANSACTION
20:20:38.493 [consensus/noops] broadcastConsensusMsg -> DEBU 06c Broadcasting CONSENSUS
20:20:38.502 [peer] SendMessage -> DEBU 06d Sending message to stream of type: CONSENSUS
20:20:39.002 [consensus/noops] doTransactions -> DEBU 06e Executing transactions
20:20:39.002 [consensus/noops] doTransactions -> DEBU 06f Starting TX batch with timestamp: seconds:1450815638 nanos:378315806
20:20:39.002 [consensus/noops] doTransactions -> DEBU 070 Executing batch of 1 transactions
20:20:39.012 [chaincode] DeployChaincode -> DEBU 071 Executable is /go/bin/chaincode_example02
20:20:39.012 [chaincode] DeployChaincode -> DEBU 072 deploying vmname dev-dev-fr2-github.com.openblockchain.obc-peer.openchain.example.chaincode.chaincode_example02:0.0.1
20:20:39.012 [container] lockContainer -> DEBU 073 waiting for container(dev-dev-fr2-github.com.openblockchain.obc-peer.openchain.example.chaincode.chaincode_example02:0.0.1) lock
20:20:39.013 [container] lockContainer -> DEBU 074 got container (dev-dev-fr2-github.com.openblockchain.obc-peer.openchain.example.chaincode.chaincode_example02:0.0.1) lock
20:20:39.723 [consensus/handler] HandleMessage -> DEBU 075 Did not handle message of type SYNC_BLOCK_ADDED, passing on to next MessageHandler
20:20:39.724 [peer] HandleMessage -> DEBU 076 Handling OpenchainMessage of type: SYNC_BLOCK_ADDED
20:20:39.724 [peer] beforeBlockAdded -> DEBU 077 Received message: SYNC_BLOCK_ADDED
fatal error: runtime: out of memory

runtime stack:
runtime.throw(0xd3b600, 0x16)
    /usr/local/go/src/runtime/panic.go:527 +0x90
runtime.sysMap(0xc825c60000, 0xf3b80000, 0x129e300, 0x12d8738)
    /usr/local/go/src/runtime/mem_linux.go:143 +0x9b
runtime.mHeap_SysAlloc(0x12b8820, 0xf3b80000, 0x2d2e)
    /usr/local/go/src/runtime/malloc.go:423 +0x160
runtime.mHeap_Grow(0x12b8820, 0x79dc0, 0x0)
    /usr/local/go/src/runtime/mheap.go:628 +0x63
runtime.mHeap_AllocSpanLocked(0x12b8820, 0x79dba, 0xc82010b950)
    /usr/local/go/src/runtime/mheap.go:532 +0x5f1
runtime.mHeap_Alloc_m(0x12b8820, 0x79dba, 0xffffff0100000000, 0x7f596637ac10)
    /usr/local/go/src/runtime/mheap.go:425 +0x1ac
runtime.mHeap_Alloc.func1()
    /usr/local/go/src/runtime/mheap.go:484 +0x41
runtime.systemstack(0x7f596637ac28)
    /usr/local/go/src/runtime/asm_amd64.s:278 +0xab
runtime.mHeap_Alloc(0x12b8820, 0x79dba, 0x10100000000, 0xc82001b500)
    /usr/local/go/src/runtime/mheap.go:485 +0x63
runtime.largeAlloc(0xf3b72012, 0xc800000001, 0x524590)
    /usr/local/go/src/runtime/malloc.go:745 +0xb3
runtime.mallocgc.func3()
    /usr/local/go/src/runtime/malloc.go:634 +0x33
runtime.systemstack(0xc82001b500)
    /usr/local/go/src/runtime/asm_amd64.s:262 +0x79
runtime.mstart()
    /usr/local/go/src/runtime/proc1.go:674

goroutine 8 [running]:
runtime.systemstack_switch()
    /usr/local/go/src/runtime/asm_amd64.s:216 fp=0xc82050f8a0 sp=0xc82050f898
runtime.mallocgc(0xf3b72012, 0xa942c0, 0x1, 0x0)
    /usr/local/go/src/runtime/malloc.go:635 +0x9c4 fp=0xc82050f970 sp=0xc82050f8a0
runtime.newarray(0xa942c0, 0xf3b72012, 0xc8201aebd0)
    /usr/local/go/src/runtime/malloc.go:777 +0xc9 fp=0xc82050f9b0 sp=0xc82050f970
runtime.makeslice(0xa7b360, 0xf3b72012, 0xf3b72012, 0x0, 0x0, 0x0)
    /usr/local/go/src/runtime/slice.go:32 +0x165 fp=0xc82050fa00 sp=0xc82050f9b0
github.com/openblockchain/obc-peer/vendor/google.golang.org/grpc.(*parser).recvMsg(0xc820219350, 0xa8019d, 0x0, 0x0, 0x0, 0x0, 0x0)
    /go/src/github.com/openblockchain/obc-peer/vendor/google.golang.org/grpc/rpc_util.go:149 +0x205 fp=0xc82050faa0 sp=0xc82050fa00
github.com/openblockchain/obc-peer/vendor/google.golang.org/grpc.recv(0xc820219350, 0x7f5969938850, 0x12d6270, 0xc11540, 0xc820255320, 0x0, 0x0)
    /go/src/github.com/openblockchain/obc-peer/vendor/google.golang.org/grpc/rpc_util.go:195 +0x42 fp=0xc82050fb88 sp=0xc82050faa0
github.com/openblockchain/obc-peer/vendor/google.golang.org/grpc.(*clientStream).RecvMsg(0xc8201d2780, 0xc11540, 0xc820255320, 0x0, 0x0)
    /go/src/github.com/openblockchain/obc-peer/vendor/google.golang.org/grpc/stream.go:202 +0x89 fp=0xc82050fc50 sp=0xc82050fb88
github.com/openblockchain/obc-peer/protos.(*peerChatClient).Recv(0xc820219360, 0xc820170ed0, 0x0, 0x0)
    /go/src/github.com/openblockchain/obc-peer/protos/openchain.pb.go:585 +0x7e fp=0xc82050fca8 sp=0xc82050fc50
github.com/openblockchain/obc-peer/openchain/peer.(*PeerImpl).handleChat(0xc8201b15f0, 0x7f596993cde8, 0xc820010660, 0x7f596993d3a8, 0xc820219360, 0x1, 0x0, 0x0)
    /go/src/github.com/openblockchain/obc-peer/openchain/peer/peer.go:515 +0x402 fp=0xc82050fe08 sp=0xc82050fca8
github.com/openblockchain/obc-peer/openchain/peer.(*PeerImpl).chatWithPeer(0xc8201b15f0, 0xc820014022, 0x10, 0x0, 0x0)
    /go/src/github.com/openblockchain/obc-peer/openchain/peer/peer.go:501 +0x8db fp=0xc82050ff78 sp=0xc82050fe08
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc82050ff80 sp=0xc82050ff78
created by github.com/openblockchain/obc-peer/openchain/peer.NewPeerWithHandler
    /go/src/github.com/openblockchain/obc-peer/openchain/peer/peer.go:242 +0x830

goroutine 1 [chan receive]:
main.serve(0x12d6270, 0x0, 0x0, 0x0, 0x0)
    /go/src/github.com/openblockchain/obc-peer/main.go:419 +0x143d
main.glob.func1(0x1292560, 0x12d6270, 0x0, 0x0)
    /go/src/github.com/openblockchain/obc-peer/main.go:74 +0x35
github.com/openblockchain/obc-peer/vendor/github.com/spf13/cobra.(*Command).execute(0x1292560, 0x12d6270, 0x0, 0x0, 0x0, 0x0)
    /go/src/github.com/openblockchain/obc-peer/vendor/github.com/spf13/cobra/command.go:501 +0x869
github.com/openblockchain/obc-peer/vendor/github.com/spf13/cobra.(*Command).Execute(0x12923a0, 0x0, 0x0)
    /go/src/github.com/openblockchain/obc-peer/vendor/github.com/spf13/cobra/command.go:584 +0x46a
main.main()
    /go/src/github.com/openblockchain/obc-peer/main.go:272 +0x1691

goroutine 17 [syscall, locked to thread]:
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1696 +0x1

goroutine 7 [chan receive]:
github.com/openblockchain/obc-peer/events/producer.(*eventProcessor).start(0xc8201b0300)
    /go/src/github.com/openblockchain/obc-peer/events/producer/events.go:72 +0xbe
created by github.com/openblockchain/obc-peer/events/producer.initializeEvents
    /go/src/github.com/openblockchain/obc-peer/events/producer/events.go:120 +0x178

goroutine 9 [IO wait]:
net.runtime_pollWait(0x7f5969938550, 0x72, 0xc820010150)
    /usr/local/go/src/runtime/netpoll.go:157 +0x60
net.(*pollDesc).Wait(0xc82021c760, 0x72, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc82021c760, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).accept(0xc82021c700, 0x0, 0x7f596993d008, 0xc8201f8fe0)
    /usr/local/go/src/net/fd_unix.go:408 +0x27c
net.(*TCPListener).AcceptTCP(0xc820026320, 0xc8200a7d00, 0x0, 0x0)
    /usr/local/go/src/net/tcpsock_posix.go:254 +0x4d
net/http.tcpKeepAliveListener.Accept(0xc820026320, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:2135 +0x41
net/http.(*Server).Serve(0xc82020d380, 0x7f596993d058, 0xc820026320, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:1887 +0xb3
net/http.(*Server).ListenAndServe(0xc82020d380, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:1877 +0x136
net/http.ListenAndServe(0xc820164980, 0xc, 0x7f596993d030, 0xc8201c56c0, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:1967 +0x8f
github.com/openblockchain/obc-peer/openchain/rest.StartOpenchainRESTServer(0xc8200262b0, 0xc82020a160)
    /go/src/github.com/openblockchain/obc-peer/openchain/rest/rest_api.go:384 +0x3ac
created by main.serve
    /go/src/github.com/openblockchain/obc-peer/main.go:381 +0xef6

goroutine 10 [IO wait]:
net.runtime_pollWait(0x7f5969938790, 0x72, 0xc820010150)
    /usr/local/go/src/runtime/netpoll.go:157 +0x60
net.(*pollDesc).Wait(0xc820158f40, 0x72, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc820158f40, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).accept(0xc820158ee0, 0x0, 0x7f596993d008, 0xc821dd6560)
    /usr/local/go/src/net/fd_unix.go:408 +0x27c
net.(*TCPListener).AcceptTCP(0xc820026200, 0xc820037e28, 0x0, 0x0)
    /usr/local/go/src/net/tcpsock_posix.go:254 +0x4d
net.(*TCPListener).Accept(0xc820026200, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/tcpsock_posix.go:264 +0x3d
github.com/openblockchain/obc-peer/vendor/google.golang.org/grpc.(*Server).Serve(0xc82019fe00, 0x7f5969937810, 0xc820026200, 0x0, 0x0)
    /go/src/github.com/openblockchain/obc-peer/vendor/google.golang.org/grpc/server.go:224 +0x1c1
main.serve.func1(0xc82019fe00, 0x7f5969937810, 0xc820026200, 0xc82020c900)
    /go/src/github.com/openblockchain/obc-peer/main.go:401 +0x35
created by main.serve
    /go/src/github.com/openblockchain/obc-peer/main.go:403 +0x132e

goroutine 11 [select]:
github.com/openblockchain/obc-peer/vendor/google.golang.org/grpc/transport.(*http2Client).controller(0xc8200824b0)
    /go/src/github.com/openblockchain/obc-peer/vendor/google.golang.org/grpc/transport/http2_client.go:810 +0x629
created by github.com/openblockchain/obc-peer/vendor/google.golang.org/grpc/transport.newHTTP2Client
    /go/src/github.com/openblockchain/obc-peer/vendor/google.golang.org/grpc/transport/http2_client.go:193 +0x1536

goroutine 12 [IO wait]:
net.runtime_pollWait(0x7f5969938610, 0x72, 0xc820010150)
    /usr/local/go/src/runtime/netpoll.go:157 +0x60
net.(*pollDesc).Wait(0xc82011e220, 0x72, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc82011e220, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).Read(0xc82011e1c0, 0xc82021a4a4, 0x9, 0x9, 0x0, 0x7f5969932050, 0xc820010150)
    /usr/local/go/src/net/fd_unix.go:232 +0x23a
net.(*conn).Read(0xc820026328, 0xc82021a4a4, 0x9, 0x9, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:172 +0xe4
io.ReadAtLeast(0x7f596993d0b8, 0xc820026328, 0xc82021a4a4, 0x9, 0x9, 0x9, 0x0, 0x0, 0x0)
    /usr/local/go/src/io/io.go:298 +0xe6
io.ReadFull(0x7f596993d0b8, 0xc820026328, 0xc82021a4a4, 0x9, 0x9, 0x7f596993d828, 0x0, 0x0)
    /usr/local/go/src/io/io.go:316 +0x62
github.com/openblockchain/obc-peer/vendor/golang.org/x/net/http2.readFrameHeader(0xc82021a4a4, 0x9, 0x9, 0x7f596993d0b8, 0xc820026328, 0x20000000, 0x0, 0x0, 0x0)
    /go/src/github.com/openblockchain/obc-peer/vendor/golang.org/x/net/http2/frame.go:227 +0xa5
github.com/openblockchain/obc-peer/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc82021a480, 0x0, 0x0, 0x0, 0x0)
    /go/src/github.com/openblockchain/obc-peer/vendor/golang.org/x/net/http2/frame.go:372 +0xe6
github.com/openblockchain/obc-peer/vendor/google.golang.org/grpc/transport.(*framer).readFrame(0xc820211e90, 0x0, 0x0, 0x0, 0x0)
    /go/src/github.com/openblockchain/obc-peer/vendor/google.golang.org/grpc/transport/http_util.go:450 +0x3d
github.com/openblockchain/obc-peer/vendor/google.golang.org/grpc/transport.(*http2Client).reader(0xc8200824b0)
    /go/src/github.com/openblockchain/obc-peer/vendor/google.golang.org/grpc/transport/http2_client.go:740 +0x115
created by github.com/openblockchain/obc-peer/vendor/google.golang.org/grpc/transport.newHTTP2Client
    /go/src/github.com/openblockchain/obc-peer/vendor/google.golang.org/grpc/transport/http2_client.go:199 +0x1595

goroutine 13 [select]:
github.com/openblockchain/obc-peer/vendor/google.golang.org/grpc.(*Conn).transportMonitor(0xc8201d40c0)
    /go/src/github.com/openblockchain/obc-peer/vendor/google.golang.org/grpc/clientconn.go:446 +0x3e1
created by github.com/openblockchain/obc-peer/vendor/google.golang.org/grpc.NewConn
    /go/src/github.com/openblockchain/obc-peer/vendor/google.golang.org/grpc/clientconn.go:282 +0x467

goroutine 15 [chan receive]:
github.com/openblockchain/obc-peer/vendor/google.golang.org/grpc.NewClientStream.func1(0xc8200969c0, 0xc8201d2780)
    /go/src/github.com/openblockchain/obc-peer/vendor/google.golang.org/grpc/stream.go:136 +0x7c
created by github.com/openblockchain/obc-peer/vendor/google.golang.org/grpc.NewClientStream
    /go/src/github.com/openblockchain/obc-peer/vendor/google.golang.org/grpc/stream.go:138 +0x869

goroutine 20 [IO wait]:
net.runtime_pollWait(0x7f59699386d0, 0x72, 0xc820010150)
    /usr/local/go/src/runtime/netpoll.go:157 +0x60
net.(*pollDesc).Wait(0xc820158fb0, 0x72, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc820158fb0, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).accept(0xc820158f50, 0x0, 0x7f596993d008, 0xc820103660)
    /usr/local/go/src/net/fd_unix.go:408 +0x27c
net.(*TCPListener).AcceptTCP(0xc820026208, 0x524460, 0x0, 0x0)
    /usr/local/go/src/net/tcpsock_posix.go:254 +0x4d
net.(*TCPListener).Accept(0xc820026208, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/tcpsock_posix.go:264 +0x3d
github.com/openblockchain/obc-peer/vendor/google.golang.org/grpc.(*Server).Serve(0xc82019e780, 0x7f5969937810, 0xc820026208, 0x0, 0x0)
    /go/src/github.com/openblockchain/obc-peer/vendor/google.golang.org/grpc/server.go:224 +0x1c1
created by main.serve
    /go/src/github.com/openblockchain/obc-peer/main.go:415 +0x141a

goroutine 21 [select]:
github.com/openblockchain/obc-peer/openchain/peer.(*Handler).start(0xc82020d6e0, 0x0, 0x0)
    /go/src/github.com/openblockchain/obc-peer/openchain/peer/handler.go:293 +0x408
created by github.com/openblockchain/obc-peer/openchain/peer.(*Handler).beforeHello
    /go/src/github.com/openblockchain/obc-peer/openchain/peer/handler.go:198 +0xf52

goroutine 39 [select]:
net/http.(*persistConn).roundTrip(0xc820154580, 0xc8209374e0, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/http/transport.go:1164 +0xb0d
net/http.(*Transport).RoundTrip(0xc820076090, 0xc820118380, 0xef, 0x0, 0x0)
    /usr/local/go/src/net/http/transport.go:235 +0x530
net/http.send(0xc820118380, 0x7f59699367e0, 0xc820076090, 0xc820296c60, 0x0, 0x0)
    /usr/local/go/src/net/http/client.go:220 +0x52c
net/http.(*Client).send(0x12afdc0, 0xc820118380, 0x8c, 0x0, 0x0)
    /usr/local/go/src/net/http/client.go:143 +0x15a
net/http.(*Client).doFollowingRedirects(0x12afdc0, 0xc820118380, 0xe0c140, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/http/client.go:380 +0xbc0
net/http.(*Client).Do(0x12afdc0, 0xc820118380, 0xcb0df8, 0x0, 0x0)
    /usr/local/go/src/net/http/client.go:178 +0x188
github.com/openblockchain/obc-peer/vendor/github.com/fsouza/go-dockerclient.(*Client).stream(0xc820296ab0, 0xca9c10, 0x4, 0xc82010c980, 0x76, 0x1, 0xc8202a0a20, 0x7f596993f0c8, 0xc82021c150, 0x7f59699363d8, ...)
    /go/src/github.com/openblockchain/obc-peer/vendor/github.com/fsouza/go-dockerclient/client.go:505 +0x15b5
github.com/openblockchain/obc-peer/vendor/github.com/fsouza/go-dockerclient.(*Client).BuildImage(0xc820296ab0, 0xc82021c000, 0x64, 0x0, 0x0, 0x10000, 0x0, 0x0, 0x0, 0x0, ...)
    /go/src/github.com/openblockchain/obc-peer/vendor/github.com/fsouza/go-dockerclient/image.go:458 +0x601
github.com/openblockchain/obc-peer/openchain/container.(*dockerVM).build(0xc820936bd0, 0x7f596993cde8, 0xc820010660, 0xc82021c000, 0x64, 0xc820936960, 0x1, 0x1, 0xc8202a0720, 0x3, ...)
    /go/src/github.com/openblockchain/obc-peer/openchain/container/controller.go:74 +0x1bf
github.com/openblockchain/obc-peer/openchain/container.CreateImageReq.do(0xc82021c000, 0x64, 0x7f596993f0c8, 0xc82021c150, 0x0, 0xc820936960, 0x1, 0x1, 0xc8202a0720, 0x3, ...)
    /go/src/github.com/openblockchain/obc-peer/openchain/container/controller.go:254 +0x12b
github.com/openblockchain/obc-peer/openchain/container.(*CreateImageReq).do(0xc82020c840, 0x7f596993cde8, 0xc820010660, 0x7f596993f120, 0xc820936bd0, 0x0, 0x0, 0x0, 0x0)
    <autogenerated>:9 +0x101
github.com/openblockchain/obc-peer/openchain/container.VMCProcess.func1(0xc82020c8a0, 0x7f596993f0f0, 0xc82020c840, 0xc820936be0, 0x7f596993cde8, 0xc820010660, 0x7f596993f120, 0xc820936bd0)
    /go/src/github.com/openblockchain/obc-peer/openchain/container/controller.go:337 +0xe3
created by github.com/openblockchain/obc-peer/openchain/container.VMCProcess
    /go/src/github.com/openblockchain/obc-peer/openchain/container/controller.go:339 +0x285

goroutine 26 [select]:
github.com/openblockchain/obc-peer/vendor/google.golang.org/grpc/transport.(*http2Server).controller(0xc820917710)
    /go/src/github.com/openblockchain/obc-peer/vendor/google.golang.org/grpc/transport/http2_server.go:611 +0x629
created by github.com/openblockchain/obc-peer/vendor/google.golang.org/grpc/transport.newHTTP2Server
    /go/src/github.com/openblockchain/obc-peer/vendor/google.golang.org/grpc/transport/http2_server.go:133 +0x852

goroutine 24 [IO wait]:
net.runtime_pollWait(0x7f59699383d0, 0x72, 0xc820010150)
    /usr/local/go/src/runtime/netpoll.go:157 +0x60
net.(*pollDesc).Wait(0xc82021c220, 0x72, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc82021c220, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).Read(0xc82021c1c0, 0xc82027a000, 0x1000, 0x1000, 0x0, 0x7f5969932050, 0xc820010150)
    /usr/local/go/src/net/fd_unix.go:232 +0x23a
net.(*conn).Read(0xc820026948, 0xc82027a000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:172 +0xe4
net/http.noteEOFReader.Read(0x7f596993d0b8, 0xc820026948, 0xc8201545d8, 0xc82027a000, 0x1000, 0x1000, 0x12b0900, 0x0, 0x0)
    /usr/local/go/src/net/http/transport.go:1370 +0x67
net/http.(*noteEOFReader).Read(0xc8201e3f20, 0xc82027a000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    <autogenerated>:126 +0xd0
bufio.(*Reader).fill(0xc82028cf00)
    /usr/local/go/src/bufio/bufio.go:97 +0x1e9
bufio.(*Reader).Peek(0xc82028cf00, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/bufio/bufio.go:132 +0xcc
net/http.(*persistConn).readLoop(0xc820154580)
    /usr/local/go/src/net/http/transport.go:876 +0xf7
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:685 +0xc78

goroutine 25 [running]:
    goroutine running on other thread; stack unavailable
created by net/http.(*Transport).dialConn
    /usr/local/go/src/net/http/transport.go:686 +0xc9d

goroutine 27 [IO wait]:
net.runtime_pollWait(0x7f5969938250, 0x72, 0xc820010150)
    /usr/local/go/src/runtime/netpoll.go:157 +0x60
net.(*pollDesc).Wait(0xc82021da30, 0x72, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc82021da30, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).Read(0xc82021d9d0, 0xc8209176a4, 0x9, 0x9, 0x0, 0x7f5969932050, 0xc820010150)
    /usr/local/go/src/net/fd_unix.go:232 +0x23a
net.(*conn).Read(0xc820026998, 0xc8209176a4, 0x9, 0x9, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:172 +0xe4
io.ReadAtLeast(0x7f596993d0b8, 0xc820026998, 0xc8209176a4, 0x9, 0x9, 0x9, 0x0, 0x0, 0x0)
    /usr/local/go/src/io/io.go:298 +0xe6
io.ReadFull(0x7f596993d0b8, 0xc820026998, 0xc8209176a4, 0x9, 0x9, 0x0, 0x0, 0x0)
    /usr/local/go/src/io/io.go:316 +0x62
github.com/openblockchain/obc-peer/vendor/golang.org/x/net/http2.readFrameHeader(0xc8209176a4, 0x9, 0x9, 0x7f596993d0b8, 0xc820026998, 0x69000000, 0xc800000000, 0x0, 0x0)
    /go/src/github.com/openblockchain/obc-peer/vendor/golang.org/x/net/http2/frame.go:227 +0xa5
github.com/openblockchain/obc-peer/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc820917680, 0x0, 0x0, 0x0, 0x0)
    /go/src/github.com/openblockchain/obc-peer/vendor/golang.org/x/net/http2/frame.go:372 +0xe6
github.com/openblockchain/obc-peer/vendor/google.golang.org/grpc/transport.(*framer).readFrame(0xc82012b950, 0x0, 0x0, 0x0, 0x0)
    /go/src/github.com/openblockchain/obc-peer/vendor/google.golang.org/grpc/transport/http_util.go:450 +0x3d
github.com/openblockchain/obc-peer/vendor/google.golang.org/grpc/transport.(*http2Server).HandleStreams(0xc820917710, 0xc82012b9e0)
    /go/src/github.com/openblockchain/obc-peer/vendor/google.golang.org/grpc/transport/http2_server.go:242 +0x663
github.com/openblockchain/obc-peer/vendor/google.golang.org/grpc.(*Server).Serve.func2(0x7f596993d918, 0xc820917710, 0xc82019fe00)
    /go/src/github.com/openblockchain/obc-peer/vendor/google.golang.org/grpc/server.go:281 +0xcd
created by github.com/openblockchain/obc-peer/vendor/google.golang.org/grpc.(*Server).Serve
    /go/src/github.com/openblockchain/obc-peer/vendor/google.golang.org/grpc/server.go:286 +0xc78

goroutine 28 [select]:
github.com/openblockchain/obc-peer/vendor/google.golang.org/grpc/transport.(*http2Client).controller(0xc8209182d0)
    /go/src/github.com/openblockchain/obc-peer/vendor/google.golang.org/grpc/transport/http2_client.go:810 +0x629
created by github.com/openblockchain/obc-peer/vendor/google.golang.org/grpc/transport.newHTTP2Client
    /go/src/github.com/openblockchain/obc-peer/vendor/google.golang.org/grpc/transport/http2_client.go:193 +0x1536

goroutine 29 [IO wait]:
net.runtime_pollWait(0x7f5969938310, 0x72, 0xc820010150)
    /usr/local/go/src/runtime/netpoll.go:157 +0x60
net.(*pollDesc).Wait(0xc82021d9c0, 0x72, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc82021d9c0, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).Read(0xc82021d960, 0xc8209177c4, 0x9, 0x9, 0x0, 0x7f5969932050, 0xc820010150)
    /usr/local/go/src/net/fd_unix.go:232 +0x23a
net.(*conn).Read(0xc8200269a0, 0xc8209177c4, 0x9, 0x9, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/net.go:172 +0xe4
io.ReadAtLeast(0x7f596993d0b8, 0xc8200269a0, 0xc8209177c4, 0x9, 0x9, 0x9, 0x0, 0x0, 0x0)
    /usr/local/go/src/io/io.go:298 +0xe6
io.ReadFull(0x7f596993d0b8, 0xc8200269a0, 0xc8209177c4, 0x9, 0x9, 0x7f596993d828, 0x0, 0x0)
    /usr/local/go/src/io/io.go:316 +0x62
github.com/openblockchain/obc-peer/vendor/golang.org/x/net/http2.readFrameHeader(0xc8209177c4, 0x9, 0x9, 0x7f596993d0b8, 0xc8200269a0, 0x20000000, 0x0, 0x0, 0x0)
    /go/src/github.com/openblockchain/obc-peer/vendor/golang.org/x/net/http2/frame.go:227 +0xa5
github.com/openblockchain/obc-peer/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc8209177a0, 0x0, 0x0, 0x0, 0x0)
    /go/src/github.com/openblockchain/obc-peer/vendor/golang.org/x/net/http2/frame.go:372 +0xe6
github.com/openblockchain/obc-peer/vendor/google.golang.org/grpc/transport.(*framer).readFrame(0xc82012ba70, 0x0, 0x0, 0x0, 0x0)
    /go/src/github.com/openblockchain/obc-peer/vendor/google.golang.org/grpc/transport/http_util.go:450 +0x3d
github.com/openblockchain/obc-peer/vendor/google.golang.org/grpc/transport.(*http2Client).reader(0xc8209182d0)
    /go/src/github.com/openblockchain/obc-peer/vendor/google.golang.org/grpc/transport/http2_client.go:740 +0x115
created by github.com/openblockchain/obc-peer/vendor/google.golang.org/grpc/transport.newHTTP2Client
    /go/src/github.com/openblockchain/obc-peer/vendor/google.golang.org/grpc/transport/http2_client.go:199 +0x1595

goroutine 30 [select]:
github.com/openblockchain/obc-peer/vendor/google.golang.org/grpc.(*Conn).transportMonitor(0xc8201d4480)
    /go/src/github.com/openblockchain/obc-peer/vendor/google.golang.org/grpc/clientconn.go:446 +0x3e1
created by github.com/openblockchain/obc-peer/vendor/google.golang.org/grpc.NewConn
    /go/src/github.com/openblockchain/obc-peer/vendor/google.golang.org/grpc/clientconn.go:282 +0x467

goroutine 31 [chan receive]:
github.com/openblockchain/obc-peer/vendor/google.golang.org/grpc.NewClientStream.func1(0xc8201d4600, 0xc820094a80)
    /go/src/github.com/openblockchain/obc-peer/vendor/google.golang.org/grpc/stream.go:136 +0x7c
created by github.com/openblockchain/obc-peer/vendor/google.golang.org/grpc.NewClientStream
    /go/src/github.com/openblockchain/obc-peer/vendor/google.golang.org/grpc/stream.go:138 +0x869

goroutine 33 [select]:
github.com/openblockchain/obc-peer/openchain/container.VMCProcess(0x7f596993cde8, 0xc820010660, 0xca86f8, 0x6, 0x7f596993f0f0, 0xc82020c840, 0x0, 0x0, 0x0, 0x0)
    /go/src/github.com/openblockchain/obc-peer/openchain/container/controller.go:341 +0x3f2
github.com/openblockchain/obc-peer/openchain/chaincode.(*ChaincodeSupport).DeployChaincode(0xc820166eb0, 0x7f596993cde8, 0xc820010660, 0xc820096180, 0x0, 0x0, 0x0)
    /go/src/github.com/openblockchain/obc-peer/openchain/chaincode/chaincode_support.go:494 +0xf9f
github.com/openblockchain/obc-peer/openchain/chaincode.Execute(0x7f596993cde8, 0xc820010660, 0xc820166eb0, 0xc820096180, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    /go/src/github.com/openblockchain/obc-peer/openchain/chaincode/exectransaction.go:56 +0x278
github.com/openblockchain/obc-peer/openchain/chaincode.ExecuteTransactions(0x7f596993cde8, 0xc820010660, 0xcac490, 0x7, 0xc820026020, 0x1, 0x1, 0x0, 0x0, 0x0, ...)
    /go/src/github.com/openblockchain/obc-peer/openchain/chaincode/exectransaction.go:146 +0x32b
github.com/openblockchain/obc-peer/openchain/consensus/helper.(*Helper).ExecTXs(0xc820235fe0, 0xc820026020, 0x1, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
    /go/src/github.com/openblockchain/obc-peer/openchain/consensus/helper/helper.go:110 +0x146
github.com/openblockchain/obc-peer/openchain/consensus/noops.(*Noops).doTransactions(0xc8201036a0, 0xc8202a1020, 0x0, 0x0)
    /go/src/github.com/openblockchain/obc-peer/openchain/consensus/noops/noops.go:167 +0x52c
github.com/openblockchain/obc-peer/openchain/consensus/noops.(*Noops).RecvMsg(0xc8201036a0, 0xc8202a1020, 0x0, 0x0)
    /go/src/github.com/openblockchain/obc-peer/openchain/consensus/noops/noops.go:93 +0x33d
github.com/openblockchain/obc-peer/openchain/consensus/helper.(*ConsensusHandler).doChainTransaction(0xc8201661e0, 0xc8202a1020, 0x0, 0x0)
    /go/src/github.com/openblockchain/obc-peer/openchain/consensus/helper/handler.go:125 +0x4c3
github.com/openblockchain/obc-peer/openchain/consensus/helper.(*ConsensusHandler).HandleMessage(0xc8201661e0, 0xc8202a1020, 0x0, 0x0)
    /go/src/github.com/openblockchain/obc-peer/openchain/consensus/helper/handler.go:79 +0x9f
github.com/openblockchain/obc-peer/openchain/peer.(*PeerImpl).handleChat(0xc8201b15f0, 0x7f596993d1f8, 0xc8202a0240, 0x7f596638a990, 0xc8201fc0c0, 0x12d5500, 0x0, 0x0)
    /go/src/github.com/openblockchain/obc-peer/openchain/peer/peer.go:525 +0x679
github.com/openblockchain/obc-peer/openchain/peer.(*PeerImpl).Chat(0xc8201b15f0, 0x7f596638a938, 0xc8201fc0c0, 0x0, 0x0)
    /go/src/github.com/openblockchain/obc-peer/openchain/peer/peer.go:248 +0xae
github.com/openblockchain/obc-peer/protos._Peer_Chat_Handler(0xc76c80, 0xc8201b15f0, 0x7f596638a8f0, 0xc82028c120, 0x0, 0x0)
    /go/src/github.com/openblockchain/obc-peer/protos/openchain.pb.go:604 +0xd8
github.com/openblockchain/obc-peer/vendor/google.golang.org/grpc.(*Server).processStreamingRPC(0xc82019fe00, 0x7f596993d918, 0xc820917710, 0xc8201d4000, 0xc82019d9c0, 0x128c460, 0xc8202a0210, 0x0, 0x0)
    /go/src/github.com/openblockchain/obc-peer/vendor/google.golang.org/grpc/server.go:421 +0x2e4
github.com/openblockchain/obc-peer/vendor/google.golang.org/grpc.(*Server).handleStream(0xc82019fe00, 0x7f596993d918, 0xc820917710, 0xc8201d4000, 0xc8202a0210)
    /go/src/github.com/openblockchain/obc-peer/vendor/google.golang.org/grpc/server.go:471 +0x7f4
github.com/openblockchain/obc-peer/vendor/google.golang.org/grpc.(*Server).Serve.func2.1.1(0xc82019fe00, 0x7f596993d918, 0xc820917710, 0xc8201d4000, 0xc8202a0210, 0xc821dde290)
    /go/src/github.com/openblockchain/obc-peer/vendor/google.golang.org/grpc/server.go:278 +0x49
created by github.com/openblockchain/obc-peer/vendor/google.golang.org/grpc.(*Server).Serve.func2.1
    /go/src/github.com/openblockchain/obc-peer/vendor/google.golang.org/grpc/server.go:280 +0x5ac
muralisrini commented 8 years ago

Any pattern (however slight) to the crashes ? For instance do you always see the crash after the following message ?

20:20:39.724 [peer] HandleMessage -> DEBU 076 Handling OpenchainMessage of type: SYNC_BLOCK_ADDED 20:20:39.724 [peer] beforeBlockAdded -> DEBU 077 Received message: SYNC_BLOCK_ADDED

binhn commented 8 years ago

@muralisrini it sounds like related to leaking handles you mentioned last week. Do we still see this? Probably a good case for Behave feature

binhn commented 8 years ago

Assuming the same behavior as #520