Closed wangkuiyi closed 5 years ago
Tried locally. Using bufio or not, deliver similar performance.
I wrote another benchmark that focuses on benchmarking the use/not-use bufio.Reader
to wrap
an os.File
returned by os.Open
:
func BenchmarkGoFileReader(b *testing.B) {
fn, e := synthesizeTempFile(1000)
if e != nil {
b.Fatalf("Cannot synthesize file %s: %v", fn, e)
}
defer os.Remove(fn)
buf := make([]byte, 4*1000)
b.Run("", func(b *testing.B) {
for i := 0; i < b.N; i++ {
f, e := os.Open(fn)
if e != nil {
b.Fatalf("Cannot open synthesized file %s: %v", fn, e)
}
c := 0
ff := bufio.NewReader(f) # <==== We can remove this line to disable bufio.Reader.
for _, e := ff.Read(buf); e == nil; _, e = ff.Read(buf) {
c++
}
f.Close()
}
})
}
Again, the result shows that no performance difference when reading files from the local filesystem on macOS.
This PR adds a benchmark of RecordIO reading. The following command runs unit tests and benchmarks.
Without
-bench=.
, it runs no benchmark.This PR also tries to improve reading performance by creating a
bufio.Reader
for each chunk to be read. However, it doesn't seem a significant change. Maybe we should try some other way.Before using
bufio.Reader
:After using it: