wangkuiyi / recordio

Apache License 2.0
11 stars 2 forks source link

Use bufio.NewReader and add benchmark #51

Closed wangkuiyi closed 5 years ago

wangkuiyi commented 5 years ago

This PR adds a benchmark of RecordIO reading. The following command runs unit tests and benchmarks.

go test -bench=.

Without -bench=., it runs no benchmark.

This PR also tries to improve reading performance by creating a bufio.Reader for each chunk to be read. However, it doesn't seem a significant change. Maybe we should try some other way.

Before using bufio.Reader:

goos: darwin
goarch: amd64
pkg: github.com/wangkuiyi/recordio
BenchmarkRead/reading_records_00000_to_00050-4                 5     293563564 ns/op
BenchmarkRead/reading_records_00050_to_00100-4                 3     445585152 ns/op
BenchmarkRead/reading_records_00100_to_00150-4                 5     298169020 ns/op
PASS
ok      github.com/wangkuiyi/recordio   8.330s

After using it:

goos: darwin
goarch: amd64
pkg: github.com/wangkuiyi/recordio
BenchmarkRead/reading_records_00000_to_00050-4                 5     296029121 ns/op
BenchmarkRead/reading_records_00050_to_00100-4                 3     455643627 ns/op
BenchmarkRead/reading_records_00100_to_00150-4                 3     365613422 ns/op
PASS
ok      github.com/wangkuiyi/recordio   7.143s
ywskycn commented 5 years ago

Tried locally. Using bufio or not, deliver similar performance.

wangkuiyi commented 5 years ago

I wrote another benchmark that focuses on benchmarking the use/not-use bufio.Reader to wrap an os.File returned by os.Open:


func BenchmarkGoFileReader(b *testing.B) {
    fn, e := synthesizeTempFile(1000)
    if e != nil {
        b.Fatalf("Cannot synthesize file %s: %v", fn, e)
    }
    defer os.Remove(fn)

    buf := make([]byte, 4*1000)
    b.Run("", func(b *testing.B) {
        for i := 0; i < b.N; i++ {
            f, e := os.Open(fn)
            if e != nil {
                b.Fatalf("Cannot open synthesized file %s: %v", fn, e)
            }
            c := 0
            ff := bufio.NewReader(f)  # <==== We can remove this line to disable bufio.Reader.
            for _, e := ff.Read(buf); e == nil; _, e = ff.Read(buf) {
                c++
            }
            f.Close()
        }
    })
}

Again, the result shows that no performance difference when reading files from the local filesystem on macOS.