xueqiu / rdr

Apache License 2.0
1.18k stars 303 forks source link

out of memory #10

Closed SuperDubbo closed 6 years ago

SuperDubbo commented 6 years ago

start parsing... runtime: out of memory: cannot allocate 227278848-byte block (1839726592 in use)

fatal error: out of memory

runtime stack: runtime.throw(0x79a639, 0xd) c:/go/src/runtime/panic.go:616 +0x6b runtime.largeAlloc(0xd8c0000, 0x12b10001, 0x4) c:/go/src/runtime/malloc.go:828 +0xdc runtime.mallocgc.func1() c:/go/src/runtime/malloc.go:721 +0x39 runtime.systemstack(0x0) c:/go/src/runtime/asm_386.s:464 +0x5e runtime.mstart() c:/go/src/runtime/proc.go:1175

goroutine 1 [running]: runtime.systemstack_switch() c:/go/src/runtime/asm_386.s:418 fp=0x12d41ae4 sp=0x12d41ae0 pc=0x44ac30 runtime.mallocgc(0xd8c0000, 0x75aec0, 0x3000101, 0x0) c:/go/src/runtime/malloc.go:720 +0x67b fp=0x12d41b38 sp=0x12d41ae4 pc=0x 40dd3b runtime.newarray(0x75aec0, 0x110000, 0x8c5b0332) c:/go/src/runtime/malloc.go:855 +0x52 fp=0x12d41b4c sp=0x12d41b38 pc=0x4 0dff2 runtime.makeBucketArray(0x741680, 0x401414, 0x35476b40, 0x8c5b0332) c:/go/src/runtime/hashmap.go:881 +0xbf fp=0x12d41b6c sp=0x12d41b4c pc=0x 40739f runtime.hashGrow(0x741680, 0x12bc5cc0) c:/go/src/runtime/hashmap.go:905 +0x6a fp=0x12d41b98 sp=0x12d41b6c pc=0x 4074fa runtime.mapassign(0x741680, 0x12bc5cc0, 0x12d41c48, 0xbf8e00) c:/go/src/runtime/hashmap.go:579 +0x262 fp=0x12d41bf8 sp=0x12d41b98 pc=0 x4065b2 main.(Counter).countByKeyPrefix(0x12b48190, 0x2950a4b0) E:/redis-rdb-tools/rdr/counter.go:187 +0x11f fp=0x12d41c6c sp=0x12d41bf8 pc=0x6edd2f main.(Counter).count(0x12b48190, 0x2950a4b0) E:/redis-rdb-tools/rdr/counter.go:124 +0x17f fp=0x12d41ca4 sp=0x12d41c6c pc=0x6ed31f main.(Counter).Count(0x12b48190, 0x12b3a440) E:/redis-rdb-tools/rdr/counter.go:69 +0x3b fp=0x12d41cb8 sp=0x12d41ca4 p c=0x6ecb3b main.show(0x12b4e370) E:/redis-rdb-tools/rdr/main.go:68 +0x281 fp=0x12d41d88 sp=0x12d41cb8 pc= 0x6f0c31 github.com/urfave/cli.HandleAction(0x72b3c0, 0x7b6bf0, 0x12b4e370, 0x0, 0x12b661 80) C:/GOPATH/src/github.com/urfave/cli/app.go:503 +0x62 fp=0x12d41d98 sp=0x 12d41d88 pc=0x63ca72 github.com/urfave/cli.Command.Run(0x794903, 0x4, 0x0, 0x0, 0x0, 0x0, 0x0, 0x7add 81, 0x32, 0x0, ...) C:/GOPATH/src/github.com/urfave/cli/command.go:165 +0x3d7 fp=0x12d41e74 sp=0x12d41d98 pc=0x63d517 github.com/urfave/cli.(App).Run(0x12b1ca80, 0x12b3a040, 0x5, 0x8, 0x0, 0x0) C:/GOPATH/src/github.com/urfave/cli/app.go:259 +0x5cc fp=0x12d41f70 sp=0 x12d41e74 pc=0x63b19c main.main() E:/redis-rdb-tools/rdr/main.go:146 +0x24d fp=0x12d41fc8 sp=0x12d41f70 pc =0x6f164d runtime.main() c:/go/src/runtime/proc.go:198 +0x1ea fp=0x12d41ff0 sp=0x12d41fc8 pc=0x42 8a1a runtime.goexit() c:/go/src/runtime/asm_386.s:1665 +0x1 fp=0x12d41ff4 sp=0x12d41ff0 pc=0x4 4c6f1

goroutine 5 [chan send]: main.(Decoder).Set(0x12b9d640, 0x35415270, 0x4b, 0x4b, 0x2d0c9200, 0x208, 0x208 , 0x0, 0x0) E:/redis-rdb-tools/rdr/decoder.go:61 +0x22f github.com/cupcake/rdb.(decode).readObject(0x12b25f58, 0x35415270, 0x4b, 0x4b, 0x0, 0x0, 0x0, 0x0, 0x0) C:/GOPATH/src/github.com/cupcake/rdb/decoder.go:233 +0xc0 github.com/cupcake/rdb.(*decode).decode(0x12b25f58, 0x1000, 0x1000) C:/GOPATH/src/github.com/cupcake/rdb/decoder.go:215 +0xe0 github.com/cupcake/rdb.Decode(0x7ea4b0, 0x12b053f8, 0x7ee2c0, 0x12b9d640, 0x0, 0 x0) C:/GOPATH/src/github.com/cupcake/rdb/decoder.go:67 +0xbf main.decode(0x12b4e370, 0x12b9d640, 0x12b08100, 0xb) E:/redis-rdb-tools/rdr/main.go:43 +0xdc created by main.show E:/redis-rdb-tools/rdr/main.go:66 +0x266

E:\redis\Redis-x64-3.0.504\WEB_TMP>

环境:win10 rdb大小:4G 机器内存:8G 跑几次,直接爆掉 你说“5GB rdbfile takes about 2mins on my PC”是在个人PC Windows下面吗? @mengxy

SuperDubbo commented 6 years ago

已解决,开始构建错了,构建成32位版本,后面重新构建一个64位版本,4Grdb文件很轻松就解析完了,不错! @mengxy