search

wangkuiyi / recordio

Apache License 2.0
11 stars 2 forks source link

Use Go's I/O pipeline #57

Closed wangkuiyi closed 5 years ago

wangkuiyi commented 5 years ago

The benchmark shows that this PR improves the reading speed for about 1000 times!

The benchmark on develop branch at this moment:

yi@WangYis-iMac:/go/src/github.com/wangkuiyi/recordio (develop)$ go test -bench=.
goos: darwin
goarch: amd64
pkg: github.com/wangkuiyi/recordio
BenchmarkRead/reading_records_00000_to_00050-4                 5     297171427 ns/op
BenchmarkRead/reading_records_00050_to_00100-4                 3     486892504 ns/op
BenchmarkRead/reading_records_00100_to_00150-4                 5     293632646 ns/op
PASS
ok      github.com/wangkuiyi/recordio   8.859s

The benchmark of this PR:

yi@WangYis-iMac:/go/src/github.com/wangkuiyi/recordio (use_pipeline_in_io)$ go test -bench=.
goos: darwin
goarch: amd64
pkg: github.com/wangkuiyi/recordio
BenchmarkRead/reading_records_00000_to_00050-4              5000        414011 ns/op
BenchmarkRead/reading_records_00050_to_00100-4              3000        535504 ns/op
BenchmarkRead/reading_records_00100_to_00150-4              5000        473778 ns/op
PASS
ok      github.com/wangkuiyi/recordio   6.589s

The performance improvement hopefully fixes https://github.com/wangkuiyi/recordio/issues/50.