kawasin73
commented
6 years ago syscall.Mmap() でファイルをマッピングしたメモリにレコードを順に追記する処理のパフォーマンスがバッファメモリのサイズ（cachesize）によってどのように変わるかを比較した。ファイルの最後に来た場合はリングバッファのように周回する。
10MB、100MB、1GB のファイルサイズそれぞれについて、レコードサイズは 10B、100B、1KB、2KB の4種類を試行し、キャッシュサイズは キャッシュなし、100B、1KB、2KB、4KB、8KB、12KB、16KB、1MB の9種類を試行した。
キャッシュサイズがレコードサイズ以下となる場合はキャッシュなしとして（無駄ではあるが）試行した。
この一連のベンチマークを macOS + SSD、macOS + HDD、Ubuntu on EC2 + SSD、Ubuntu on EC2 + HDD の4つのデバイスで実行した。
結果は長いため、考察の後に記載する。
Benchmark Source Code
```go package mmap import ( "fmt" "math/rand" "os" "syscall" "testing" ) const ( rn = 97 KB = 1024 MB = KB * KB GB = MB * KB _10MB = 10 * MB _100MB = 100 * MB ) var records [rn][1024 * 1024]byte var files map[int]string var pageSize int func init() { for i := 0; i < rn; i++ { for j := 0; j < 1024*1024; j++ { records[i][j] = byte(rand.Intn(255)) } } files = make(map[int]string) for _, size := range []int{_10MB, _100MB, GB} { file := fmt.Sprintf("%v.mmap", size) f, err := os.Create(file) if err != nil { panic(err) } f.Truncate(int64(size)) f.Close() files[size] = file } pageSize = os.Getpagesize() } func BenchmarkMmap(b *testing.B) { b.Log("host page size :", pageSize) for _, filesize := range []int{_10MB, _100MB, GB} { for _, recordsize := range []int{10, 100, KB, 2 * KB} { for _, cachesize := range []int{0, 100, KB, 2 * KB, pageSize, 2 * pageSize, 3 * pageSize, 4 * pageSize, MB} { b.Run(fmt.Sprintf("file:%v record:%v cache:%v", ByteCountBinary(filesize), ByteCountBinary(cachesize), ByteCountBinary(recordsize)), func(b *testing.B) { if cachesize <= recordsize { benchmarkMmap(b, filesize, recordsize) } else { benchmarkMmapBuffer(b, filesize, cachesize, recordsize) } }) } } } } func benchmarkMmap(b *testing.B, length, size int) { f, err := os.OpenFile(files[length], os.O_RDWR, 0666) if err != nil { b.Fatal(err) } defer f.Close() buf, err := syscall.Mmap(int(f.Fd()), 0, length, syscall.PROT_READ|syscall.PROT_WRITE, syscall.MAP_SHARED) if err != nil { b.Fatal(err) } defer syscall.Munmap(buf) var idx int b.ResetTimer() for i := 0; i < b.N; i++ { copy(buf[idx:], records[i%97][:size]) idx += size if idx >= length { idx = 0 } } } func benchmarkMmapBuffer(b *testing.B, length, cachesize, size int) { cache := make([]byte, cachesize) f, err := os.OpenFile(files[length], os.O_RDWR, 0666) if err != nil { b.Fatal(err) } defer f.Close() buf, err := syscall.Mmap(int(f.Fd()), 0, length, syscall.PROT_READ|syscall.PROT_WRITE, syscall.MAP_SHARED) if err != nil { b.Fatal(err) } defer syscall.Munmap(buf) var idx, cidx int b.ResetTimer() for i := 0; i < b.N; i++ { copy(cache[cidx:], records[i%97][:size]) if cidx+size >= cachesize { copy(buf[idx:], cache[:cidx]) cidx = 0 idx += size if idx >= length { idx = 0 } } else { cidx += size } } copy(buf[idx:], cache[:cidx]) } // URL: https://programming.guide/go/formatting-byte-size-to-human-readable-format.html func ByteCountBinary(b int) string { const unit = 1024 if b < unit { return fmt.Sprintf("%d B", b) } div, exp := int64(unit), 0 for n := b / unit; n >= unit; n /= unit { div *= unit exp++ } return fmt.Sprintf("%.1f%ciB", float64(b)/float64(div), "KMGTPE"[exp]) } ```考察
キャッシュサイズが小さい場合はディスク操作の回数が増えるため、キャッシュサイズが増えるほどパフォーマンスが向上すると考えられる。
一方、キャッシュサイズが大きすぎるとメモリコピーの回数が レコード -> キャッシュ と キャッシュ -> ファイル の 2 回に増えるオーバーヘッドの影響が大きくなり、パフォーマンスが低下する。
手元の MacBook Pro では、どのレコードサイズでもバッファのサイズが ワードサイズの 2 ~ 3 倍 で書き込みパフォーマンスが極大化した。
EC2 上で動く Ubuntu ではバッファのサイズが ワードサイズの 3 ~ 4 倍 で急激にパフォーマンスが改善するが、ファイルサイズが 1GB でレコードサイズが 1KB を超えるとき、バッファのサイズが 1MB の方がより大きな最大パフォーマンスを計測することができた。
よって、バッファリングを行うことで mmap 経由での書き込みパフォーマンスは向上する が、最適なバッファサイズは 1 レコードのサイズと実行するマシンによって異なる ことがわかるが、大体のケースでは ワードサイズの 2 ~ 4 倍 程度のバッファサイズが良いパフォーマンスを出すと推測できる。
またバッファサイズとは関係ないが、ファイルサイズが大きくなるにつれて書き込みパフォーマンスが低下する傾向にあり、特にファイルサイズが 1GB で、EC2 インスタンスで試行したときにパフォーマンスの劣化が顕著である。 これは、 ページアウトが頻繁に発生 したためであると考えられる。ファイルサイズが大きいほど Mmap のページアウトが発生する頻度が増し、またEC2 はマシンメモリが 1GB と小さいためページアウトが発生する頻度が増えたものであると考えられる。
結果
それぞれのデバイスでの結果は以下の通りである。 また、データグラフの原本は こちらのスプレッドシート で公開している。
macOS + SSD
| 項目 | 内容 |
|---|---|
| OS | macOS 10.12.6 |
| Disk | APPLE SSD SM0512G |
| PC | MacBook Pro (Retina, 13-inch, Early 2015) |
| Page Size | 4096 B |
| Go Version | 1.10.1 |
Source
``` $ go test -bench=. -v goos: darwin goarch: amd64 pkg: github.com/kawasin73/sample-go/mmap BenchmarkMmap/file:10.0MiB_record:0_B_cache:10_B-4 200000000 8.25 ns/op BenchmarkMmap/file:10.0MiB_record:100_B_cache:10_B-4 200000000 8.62 ns/op BenchmarkMmap/file:10.0MiB_record:1.0KiB_cache:10_B-4 200000000 8.35 ns/op BenchmarkMmap/file:10.0MiB_record:2.0KiB_cache:10_B-4 200000000 8.24 ns/op BenchmarkMmap/file:10.0MiB_record:4.0KiB_cache:10_B-4 200000000 8.35 ns/op BenchmarkMmap/file:10.0MiB_record:8.0KiB_cache:10_B-4 200000000 8.43 ns/op BenchmarkMmap/file:10.0MiB_record:12.0KiB_cache:10_B-4 200000000 8.20 ns/op BenchmarkMmap/file:10.0MiB_record:16.0KiB_cache:10_B-4 200000000 8.13 ns/op BenchmarkMmap/file:10.0MiB_record:1.0MiB_cache:10_B-4 200000000 8.97 ns/op BenchmarkMmap/file:10.0MiB_record:0_B_cache:100_B-4 100000000 18.9 ns/op BenchmarkMmap/file:10.0MiB_record:100_B_cache:100_B-4 100000000 19.2 ns/op BenchmarkMmap/file:10.0MiB_record:1.0KiB_cache:100_B-4 100000000 14.8 ns/op BenchmarkMmap/file:10.0MiB_record:2.0KiB_cache:100_B-4 100000000 15.2 ns/op BenchmarkMmap/file:10.0MiB_record:4.0KiB_cache:100_B-4 100000000 15.0 ns/op BenchmarkMmap/file:10.0MiB_record:8.0KiB_cache:100_B-4 100000000 15.2 ns/op BenchmarkMmap/file:10.0MiB_record:12.0KiB_cache:100_B-4 100000000 15.2 ns/op BenchmarkMmap/file:10.0MiB_record:16.0KiB_cache:100_B-4 100000000 15.7 ns/op BenchmarkMmap/file:10.0MiB_record:1.0MiB_cache:100_B-4 50000000 22.4 ns/op BenchmarkMmap/file:10.0MiB_record:0_B_cache:1.0KiB-4 10000000 138 ns/op BenchmarkMmap/file:10.0MiB_record:100_B_cache:1.0KiB-4 10000000 141 ns/op BenchmarkMmap/file:10.0MiB_record:1.0KiB_cache:1.0KiB-4 10000000 139 ns/op BenchmarkMmap/file:10.0MiB_record:2.0KiB_cache:1.0KiB-4 20000000 92.9 ns/op BenchmarkMmap/file:10.0MiB_record:4.0KiB_cache:1.0KiB-4 20000000 71.8 ns/op BenchmarkMmap/file:10.0MiB_record:8.0KiB_cache:1.0KiB-4 20000000 63.6 ns/op BenchmarkMmap/file:10.0MiB_record:12.0KiB_cache:1.0KiB-4 20000000 63.7 ns/op BenchmarkMmap/file:10.0MiB_record:16.0KiB_cache:1.0KiB-4 20000000 66.0 ns/op BenchmarkMmap/file:10.0MiB_record:1.0MiB_cache:1.0KiB-4 10000000 153 ns/op BenchmarkMmap/file:10.0MiB_record:0_B_cache:2.0KiB-4 5000000 263 ns/op BenchmarkMmap/file:10.0MiB_record:100_B_cache:2.0KiB-4 5000000 262 ns/op BenchmarkMmap/file:10.0MiB_record:1.0KiB_cache:2.0KiB-4 5000000 260 ns/op BenchmarkMmap/file:10.0MiB_record:2.0KiB_cache:2.0KiB-4 5000000 259 ns/op BenchmarkMmap/file:10.0MiB_record:4.0KiB_cache:2.0KiB-4 10000000 176 ns/op BenchmarkMmap/file:10.0MiB_record:8.0KiB_cache:2.0KiB-4 10000000 141 ns/op BenchmarkMmap/file:10.0MiB_record:12.0KiB_cache:2.0KiB-4 10000000 132 ns/op BenchmarkMmap/file:10.0MiB_record:16.0KiB_cache:2.0KiB-4 10000000 150 ns/op BenchmarkMmap/file:10.0MiB_record:1.0MiB_cache:2.0KiB-4 5000000 288 ns/op BenchmarkMmap/file:100.0MiB_record:0_B_cache:10_B-4 200000000 8.22 ns/op BenchmarkMmap/file:100.0MiB_record:100_B_cache:10_B-4 200000000 9.00 ns/op BenchmarkMmap/file:100.0MiB_record:1.0KiB_cache:10_B-4 200000000 8.23 ns/op BenchmarkMmap/file:100.0MiB_record:2.0KiB_cache:10_B-4 200000000 8.24 ns/op BenchmarkMmap/file:100.0MiB_record:4.0KiB_cache:10_B-4 200000000 8.19 ns/op BenchmarkMmap/file:100.0MiB_record:8.0KiB_cache:10_B-4 200000000 8.27 ns/op BenchmarkMmap/file:100.0MiB_record:12.0KiB_cache:10_B-4 200000000 8.12 ns/op BenchmarkMmap/file:100.0MiB_record:16.0KiB_cache:10_B-4 200000000 8.28 ns/op BenchmarkMmap/file:100.0MiB_record:1.0MiB_cache:10_B-4 200000000 8.84 ns/op BenchmarkMmap/file:100.0MiB_record:0_B_cache:100_B-4 100000000 19.5 ns/op BenchmarkMmap/file:100.0MiB_record:100_B_cache:100_B-4 100000000 19.4 ns/op BenchmarkMmap/file:100.0MiB_record:1.0KiB_cache:100_B-4 100000000 15.7 ns/op BenchmarkMmap/file:100.0MiB_record:2.0KiB_cache:100_B-4 100000000 15.7 ns/op BenchmarkMmap/file:100.0MiB_record:4.0KiB_cache:100_B-4 100000000 15.6 ns/op BenchmarkMmap/file:100.0MiB_record:8.0KiB_cache:100_B-4 100000000 16.0 ns/op BenchmarkMmap/file:100.0MiB_record:12.0KiB_cache:100_B-4 100000000 15.7 ns/op BenchmarkMmap/file:100.0MiB_record:16.0KiB_cache:100_B-4 100000000 16.1 ns/op BenchmarkMmap/file:100.0MiB_record:1.0MiB_cache:100_B-4 50000000 22.9 ns/op BenchmarkMmap/file:100.0MiB_record:0_B_cache:1.0KiB-4 10000000 150 ns/op BenchmarkMmap/file:100.0MiB_record:100_B_cache:1.0KiB-4 10000000 149 ns/op BenchmarkMmap/file:100.0MiB_record:1.0KiB_cache:1.0KiB-4 10000000 149 ns/op BenchmarkMmap/file:100.0MiB_record:2.0KiB_cache:1.0KiB-4 10000000 101 ns/op BenchmarkMmap/file:100.0MiB_record:4.0KiB_cache:1.0KiB-4 20000000 73.3 ns/op BenchmarkMmap/file:100.0MiB_record:8.0KiB_cache:1.0KiB-4 20000000 66.9 ns/op BenchmarkMmap/file:100.0MiB_record:12.0KiB_cache:1.0KiB-4 20000000 66.9 ns/op BenchmarkMmap/file:100.0MiB_record:16.0KiB_cache:1.0KiB-4 20000000 70.2 ns/op BenchmarkMmap/file:100.0MiB_record:1.0MiB_cache:1.0KiB-4 10000000 156 ns/op BenchmarkMmap/file:100.0MiB_record:0_B_cache:2.0KiB-4 5000000 285 ns/op BenchmarkMmap/file:100.0MiB_record:100_B_cache:2.0KiB-4 5000000 284 ns/op BenchmarkMmap/file:100.0MiB_record:1.0KiB_cache:2.0KiB-4 5000000 284 ns/op BenchmarkMmap/file:100.0MiB_record:2.0KiB_cache:2.0KiB-4 5000000 287 ns/op BenchmarkMmap/file:100.0MiB_record:4.0KiB_cache:2.0KiB-4 10000000 193 ns/op BenchmarkMmap/file:100.0MiB_record:8.0KiB_cache:2.0KiB-4 10000000 146 ns/op BenchmarkMmap/file:100.0MiB_record:12.0KiB_cache:2.0KiB-4 10000000 138 ns/op BenchmarkMmap/file:100.0MiB_record:16.0KiB_cache:2.0KiB-4 10000000 156 ns/op BenchmarkMmap/file:100.0MiB_record:1.0MiB_cache:2.0KiB-4 5000000 296 ns/op BenchmarkMmap/file:1.0GiB_record:0_B_cache:10_B-4 50000000 21.1 ns/op BenchmarkMmap/file:1.0GiB_record:100_B_cache:10_B-4 200000000 9.37 ns/op BenchmarkMmap/file:1.0GiB_record:1.0KiB_cache:10_B-4 200000000 8.33 ns/op BenchmarkMmap/file:1.0GiB_record:2.0KiB_cache:10_B-4 200000000 8.11 ns/op BenchmarkMmap/file:1.0GiB_record:4.0KiB_cache:10_B-4 200000000 8.26 ns/op BenchmarkMmap/file:1.0GiB_record:8.0KiB_cache:10_B-4 200000000 8.12 ns/op BenchmarkMmap/file:1.0GiB_record:12.0KiB_cache:10_B-4 200000000 8.28 ns/op BenchmarkMmap/file:1.0GiB_record:16.0KiB_cache:10_B-4 200000000 8.25 ns/op BenchmarkMmap/file:1.0GiB_record:1.0MiB_cache:10_B-4 200000000 8.93 ns/op BenchmarkMmap/file:1.0GiB_record:0_B_cache:100_B-4 20000000 94.0 ns/op BenchmarkMmap/file:1.0GiB_record:100_B_cache:100_B-4 30000000 36.6 ns/op BenchmarkMmap/file:1.0GiB_record:1.0KiB_cache:100_B-4 100000000 19.9 ns/op BenchmarkMmap/file:1.0GiB_record:2.0KiB_cache:100_B-4 100000000 17.8 ns/op BenchmarkMmap/file:1.0GiB_record:4.0KiB_cache:100_B-4 100000000 16.5 ns/op BenchmarkMmap/file:1.0GiB_record:8.0KiB_cache:100_B-4 100000000 16.1 ns/op BenchmarkMmap/file:1.0GiB_record:12.0KiB_cache:100_B-4 100000000 16.1 ns/op BenchmarkMmap/file:1.0GiB_record:16.0KiB_cache:100_B-4 100000000 16.4 ns/op BenchmarkMmap/file:1.0GiB_record:1.0MiB_cache:100_B-4 50000000 22.8 ns/op BenchmarkMmap/file:1.0GiB_record:0_B_cache:1.0KiB-4 5000000 241 ns/op BenchmarkMmap/file:1.0GiB_record:100_B_cache:1.0KiB-4 5000000 246 ns/op BenchmarkMmap/file:1.0GiB_record:1.0KiB_cache:1.0KiB-4 5000000 242 ns/op BenchmarkMmap/file:1.0GiB_record:2.0KiB_cache:1.0KiB-4 5000000 200 ns/op BenchmarkMmap/file:1.0GiB_record:4.0KiB_cache:1.0KiB-4 10000000 124 ns/op BenchmarkMmap/file:1.0GiB_record:8.0KiB_cache:1.0KiB-4 10000000 127 ns/op BenchmarkMmap/file:1.0GiB_record:12.0KiB_cache:1.0KiB-4 10000000 118 ns/op BenchmarkMmap/file:1.0GiB_record:16.0KiB_cache:1.0KiB-4 10000000 114 ns/op BenchmarkMmap/file:1.0GiB_record:1.0MiB_cache:1.0KiB-4 10000000 158 ns/op BenchmarkMmap/file:1.0GiB_record:0_B_cache:2.0KiB-4 2000000 531 ns/op BenchmarkMmap/file:1.0GiB_record:100_B_cache:2.0KiB-4 2000000 532 ns/op BenchmarkMmap/file:1.0GiB_record:1.0KiB_cache:2.0KiB-4 2000000 518 ns/op BenchmarkMmap/file:1.0GiB_record:2.0KiB_cache:2.0KiB-4 2000000 517 ns/op BenchmarkMmap/file:1.0GiB_record:4.0KiB_cache:2.0KiB-4 3000000 360 ns/op BenchmarkMmap/file:1.0GiB_record:8.0KiB_cache:2.0KiB-4 5000000 252 ns/op BenchmarkMmap/file:1.0GiB_record:12.0KiB_cache:2.0KiB-4 5000000 249 ns/op BenchmarkMmap/file:1.0GiB_record:16.0KiB_cache:2.0KiB-4 5000000 269 ns/op BenchmarkMmap/file:1.0GiB_record:1.0MiB_cache:2.0KiB-4 5000000 293 ns/op --- BENCH: BenchmarkMmap bench_test.go:45: host page size : 4096 PASS ok github.com/kawasin73/sample-go/mmap 216.987s ```macOS + HDD
| 項目 | 内容 |
|---|---|
| OS | macOS 10.12.6 |
| Disk | StoreJet Transcend (HDD) |
| PC | MacBook Pro (Retina, 13-inch, Early 2015) |
| Page Size | 4096 B |
| Go Version | 1.10.1 |
Source
``` $ sudo go test -bench=. -v goos: darwin goarch: amd64 BenchmarkMmap/file:10.0MiB_record:0_B_cache:10_B-4 200000000 8.32 ns/op BenchmarkMmap/file:10.0MiB_record:100_B_cache:10_B-4 200000000 9.14 ns/op BenchmarkMmap/file:10.0MiB_record:1.0KiB_cache:10_B-4 200000000 8.49 ns/op BenchmarkMmap/file:10.0MiB_record:2.0KiB_cache:10_B-4 200000000 8.36 ns/op BenchmarkMmap/file:10.0MiB_record:4.0KiB_cache:10_B-4 200000000 8.61 ns/op BenchmarkMmap/file:10.0MiB_record:8.0KiB_cache:10_B-4 200000000 9.29 ns/op BenchmarkMmap/file:10.0MiB_record:12.0KiB_cache:10_B-4 100000000 10.2 ns/op BenchmarkMmap/file:10.0MiB_record:16.0KiB_cache:10_B-4 100000000 10.5 ns/op BenchmarkMmap/file:10.0MiB_record:1.0MiB_cache:10_B-4 200000000 9.10 ns/op BenchmarkMmap/file:10.0MiB_record:0_B_cache:100_B-4 50000000 20.4 ns/op BenchmarkMmap/file:10.0MiB_record:100_B_cache:100_B-4 100000000 20.8 ns/op BenchmarkMmap/file:10.0MiB_record:1.0KiB_cache:100_B-4 100000000 15.7 ns/op BenchmarkMmap/file:10.0MiB_record:2.0KiB_cache:100_B-4 100000000 15.4 ns/op BenchmarkMmap/file:10.0MiB_record:4.0KiB_cache:100_B-4 100000000 15.1 ns/op BenchmarkMmap/file:10.0MiB_record:8.0KiB_cache:100_B-4 100000000 15.8 ns/op BenchmarkMmap/file:10.0MiB_record:12.0KiB_cache:100_B-4 100000000 15.3 ns/op BenchmarkMmap/file:10.0MiB_record:16.0KiB_cache:100_B-4 100000000 16.2 ns/op BenchmarkMmap/file:10.0MiB_record:1.0MiB_cache:100_B-4 100000000 23.2 ns/op BenchmarkMmap/file:10.0MiB_record:0_B_cache:1.0KiB-4 10000000 162 ns/op BenchmarkMmap/file:10.0MiB_record:100_B_cache:1.0KiB-4 10000000 154 ns/op BenchmarkMmap/file:10.0MiB_record:1.0KiB_cache:1.0KiB-4 10000000 158 ns/op BenchmarkMmap/file:10.0MiB_record:2.0KiB_cache:1.0KiB-4 20000000 103 ns/op BenchmarkMmap/file:10.0MiB_record:4.0KiB_cache:1.0KiB-4 20000000 76.7 ns/op BenchmarkMmap/file:10.0MiB_record:8.0KiB_cache:1.0KiB-4 20000000 68.4 ns/op BenchmarkMmap/file:10.0MiB_record:12.0KiB_cache:1.0KiB-4 20000000 65.6 ns/op BenchmarkMmap/file:10.0MiB_record:16.0KiB_cache:1.0KiB-4 20000000 67.4 ns/op BenchmarkMmap/file:10.0MiB_record:1.0MiB_cache:1.0KiB-4 10000000 157 ns/op BenchmarkMmap/file:10.0MiB_record:0_B_cache:2.0KiB-4 5000000 297 ns/op BenchmarkMmap/file:10.0MiB_record:100_B_cache:2.0KiB-4 5000000 289 ns/op BenchmarkMmap/file:10.0MiB_record:1.0KiB_cache:2.0KiB-4 5000000 291 ns/op BenchmarkMmap/file:10.0MiB_record:2.0KiB_cache:2.0KiB-4 5000000 313 ns/op BenchmarkMmap/file:10.0MiB_record:4.0KiB_cache:2.0KiB-4 10000000 208 ns/op BenchmarkMmap/file:10.0MiB_record:8.0KiB_cache:2.0KiB-4 10000000 154 ns/op BenchmarkMmap/file:10.0MiB_record:12.0KiB_cache:2.0KiB-4 10000000 144 ns/op BenchmarkMmap/file:10.0MiB_record:16.0KiB_cache:2.0KiB-4 10000000 157 ns/op BenchmarkMmap/file:10.0MiB_record:1.0MiB_cache:2.0KiB-4 5000000 298 ns/op BenchmarkMmap/file:100.0MiB_record:0_B_cache:10_B-4 20000000 60.1 ns/op BenchmarkMmap/file:100.0MiB_record:100_B_cache:10_B-4 200000000 9.06 ns/op BenchmarkMmap/file:100.0MiB_record:1.0KiB_cache:10_B-4 200000000 8.43 ns/op BenchmarkMmap/file:100.0MiB_record:2.0KiB_cache:10_B-4 200000000 8.37 ns/op BenchmarkMmap/file:100.0MiB_record:4.0KiB_cache:10_B-4 200000000 8.70 ns/op BenchmarkMmap/file:100.0MiB_record:8.0KiB_cache:10_B-4 200000000 8.72 ns/op BenchmarkMmap/file:100.0MiB_record:12.0KiB_cache:10_B-4 200000000 8.23 ns/op BenchmarkMmap/file:100.0MiB_record:16.0KiB_cache:10_B-4 200000000 8.25 ns/op BenchmarkMmap/file:100.0MiB_record:1.0MiB_cache:10_B-4 200000000 8.99 ns/op BenchmarkMmap/file:100.0MiB_record:0_B_cache:100_B-4 50000000 22.7 ns/op BenchmarkMmap/file:100.0MiB_record:100_B_cache:100_B-4 50000000 22.0 ns/op BenchmarkMmap/file:100.0MiB_record:1.0KiB_cache:100_B-4 100000000 15.6 ns/op BenchmarkMmap/file:100.0MiB_record:2.0KiB_cache:100_B-4 100000000 15.9 ns/op BenchmarkMmap/file:100.0MiB_record:4.0KiB_cache:100_B-4 100000000 15.8 ns/op BenchmarkMmap/file:100.0MiB_record:8.0KiB_cache:100_B-4 100000000 16.0 ns/op BenchmarkMmap/file:100.0MiB_record:12.0KiB_cache:100_B-4 100000000 15.9 ns/op BenchmarkMmap/file:100.0MiB_record:16.0KiB_cache:100_B-4 100000000 16.2 ns/op BenchmarkMmap/file:100.0MiB_record:1.0MiB_cache:100_B-4 50000000 23.3 ns/op BenchmarkMmap/file:100.0MiB_record:0_B_cache:1.0KiB-4 10000000 162 ns/op BenchmarkMmap/file:100.0MiB_record:100_B_cache:1.0KiB-4 10000000 162 ns/op BenchmarkMmap/file:100.0MiB_record:1.0KiB_cache:1.0KiB-4 10000000 165 ns/op BenchmarkMmap/file:100.0MiB_record:2.0KiB_cache:1.0KiB-4 10000000 107 ns/op BenchmarkMmap/file:100.0MiB_record:4.0KiB_cache:1.0KiB-4 20000000 77.0 ns/op BenchmarkMmap/file:100.0MiB_record:8.0KiB_cache:1.0KiB-4 20000000 68.7 ns/op BenchmarkMmap/file:100.0MiB_record:12.0KiB_cache:1.0KiB-4 20000000 67.3 ns/op BenchmarkMmap/file:100.0MiB_record:16.0KiB_cache:1.0KiB-4 20000000 71.9 ns/op BenchmarkMmap/file:100.0MiB_record:1.0MiB_cache:1.0KiB-4 10000000 155 ns/op BenchmarkMmap/file:100.0MiB_record:0_B_cache:2.0KiB-4 5000000 313 ns/op BenchmarkMmap/file:100.0MiB_record:100_B_cache:2.0KiB-4 5000000 316 ns/op BenchmarkMmap/file:100.0MiB_record:1.0KiB_cache:2.0KiB-4 5000000 312 ns/op BenchmarkMmap/file:100.0MiB_record:2.0KiB_cache:2.0KiB-4 5000000 307 ns/op BenchmarkMmap/file:100.0MiB_record:4.0KiB_cache:2.0KiB-4 5000000 206 ns/op BenchmarkMmap/file:100.0MiB_record:8.0KiB_cache:2.0KiB-4 10000000 156 ns/op BenchmarkMmap/file:100.0MiB_record:12.0KiB_cache:2.0KiB-4 10000000 145 ns/op BenchmarkMmap/file:100.0MiB_record:16.0KiB_cache:2.0KiB-4 10000000 161 ns/op BenchmarkMmap/file:100.0MiB_record:1.0MiB_cache:2.0KiB-4 5000000 304 ns/op BenchmarkMmap/file:1.0GiB_record:0_B_cache:10_B-4 10000000 120 ns/op BenchmarkMmap/file:1.0GiB_record:100_B_cache:10_B-4 200000000 14.3 ns/op BenchmarkMmap/file:1.0GiB_record:1.0KiB_cache:10_B-4 200000000 8.39 ns/op BenchmarkMmap/file:1.0GiB_record:2.0KiB_cache:10_B-4 200000000 8.22 ns/op BenchmarkMmap/file:1.0GiB_record:4.0KiB_cache:10_B-4 200000000 8.70 ns/op BenchmarkMmap/file:1.0GiB_record:8.0KiB_cache:10_B-4 200000000 8.32 ns/op BenchmarkMmap/file:1.0GiB_record:12.0KiB_cache:10_B-4 200000000 8.28 ns/op BenchmarkMmap/file:1.0GiB_record:16.0KiB_cache:10_B-4 200000000 8.28 ns/op BenchmarkMmap/file:1.0GiB_record:1.0MiB_cache:10_B-4 200000000 8.83 ns/op BenchmarkMmap/file:1.0GiB_record:0_B_cache:100_B-4 20000000 551 ns/op BenchmarkMmap/file:1.0GiB_record:100_B_cache:100_B-4 30000000 39.1 ns/op BenchmarkMmap/file:1.0GiB_record:1.0KiB_cache:100_B-4 100000000 19.9 ns/op BenchmarkMmap/file:1.0GiB_record:2.0KiB_cache:100_B-4 100000000 17.8 ns/op BenchmarkMmap/file:1.0GiB_record:4.0KiB_cache:100_B-4 100000000 16.4 ns/op BenchmarkMmap/file:1.0GiB_record:8.0KiB_cache:100_B-4 100000000 16.2 ns/op BenchmarkMmap/file:1.0GiB_record:12.0KiB_cache:100_B-4 100000000 16.2 ns/op BenchmarkMmap/file:1.0GiB_record:16.0KiB_cache:100_B-4 100000000 16.0 ns/op BenchmarkMmap/file:1.0GiB_record:1.0MiB_cache:100_B-4 50000000 22.3 ns/op BenchmarkMmap/file:1.0GiB_record:0_B_cache:1.0KiB-4 5000000 262 ns/op BenchmarkMmap/file:1.0GiB_record:100_B_cache:1.0KiB-4 5000000 265 ns/op BenchmarkMmap/file:1.0GiB_record:1.0KiB_cache:1.0KiB-4 5000000 258 ns/op BenchmarkMmap/file:1.0GiB_record:2.0KiB_cache:1.0KiB-4 5000000 212 ns/op BenchmarkMmap/file:1.0GiB_record:4.0KiB_cache:1.0KiB-4 10000000 133 ns/op BenchmarkMmap/file:1.0GiB_record:8.0KiB_cache:1.0KiB-4 10000000 129 ns/op BenchmarkMmap/file:1.0GiB_record:12.0KiB_cache:1.0KiB-4 10000000 121 ns/op BenchmarkMmap/file:1.0GiB_record:16.0KiB_cache:1.0KiB-4 10000000 117 ns/op BenchmarkMmap/file:1.0GiB_record:1.0MiB_cache:1.0KiB-4 10000000 160 ns/op BenchmarkMmap/file:1.0GiB_record:0_B_cache:2.0KiB-4 2000000 558 ns/op BenchmarkMmap/file:1.0GiB_record:100_B_cache:2.0KiB-4 2000000 561 ns/op BenchmarkMmap/file:1.0GiB_record:1.0KiB_cache:2.0KiB-4 2000000 556 ns/op BenchmarkMmap/file:1.0GiB_record:2.0KiB_cache:2.0KiB-4 2000000 585 ns/op BenchmarkMmap/file:1.0GiB_record:4.0KiB_cache:2.0KiB-4 2000000 625 ns/op BenchmarkMmap/file:1.0GiB_record:8.0KiB_cache:2.0KiB-4 2000000 525 ns/op BenchmarkMmap/file:1.0GiB_record:12.0KiB_cache:2.0KiB-4 3000000 336 ns/op BenchmarkMmap/file:1.0GiB_record:16.0KiB_cache:2.0KiB-4 5000000 289 ns/op BenchmarkMmap/file:1.0GiB_record:1.0MiB_cache:2.0KiB-4 5000000 297 ns/op --- BENCH: BenchmarkMmap bench_test.go:45: host page size : 4096 PASS ok _/Volumes/Transcend 247.663s ```Ubuntu(ec2) + SSD
| 項目 | 内容 |
|---|---|
| AMI | Ubuntu Server 18.04 LTS (HVM), SSD Volume Type - ami-07ad4b1c3af1ea214 |
| Instance | t2.micro |
| Disk | SSD(gp2) |
| Page Size | 4096 B |
| Go Version | 1.10.1 |
Source
``` $ go test -bench=. -v goos: linux goarch: amd64 BenchmarkMmap/file:10.0MiB_record:0_B_cache:10_B 50000000 26.1 ns/op BenchmarkMmap/file:10.0MiB_record:100_B_cache:10_B 50000000 25.2 ns/op BenchmarkMmap/file:10.0MiB_record:1.0KiB_cache:10_B 100000000 23.6 ns/op BenchmarkMmap/file:10.0MiB_record:2.0KiB_cache:10_B 100000000 23.8 ns/op BenchmarkMmap/file:10.0MiB_record:4.0KiB_cache:10_B 50000000 24.1 ns/op BenchmarkMmap/file:10.0MiB_record:8.0KiB_cache:10_B 50000000 24.0 ns/op BenchmarkMmap/file:10.0MiB_record:12.0KiB_cache:10_B 50000000 23.9 ns/op BenchmarkMmap/file:10.0MiB_record:16.0KiB_cache:10_B 50000000 24.2 ns/op BenchmarkMmap/file:10.0MiB_record:1.0MiB_cache:10_B 50000000 24.3 ns/op BenchmarkMmap/file:10.0MiB_record:0_B_cache:100_B 50000000 31.5 ns/op BenchmarkMmap/file:10.0MiB_record:100_B_cache:100_B 50000000 31.4 ns/op BenchmarkMmap/file:10.0MiB_record:1.0KiB_cache:100_B 50000000 34.0 ns/op BenchmarkMmap/file:10.0MiB_record:2.0KiB_cache:100_B 50000000 35.4 ns/op BenchmarkMmap/file:10.0MiB_record:4.0KiB_cache:100_B 50000000 32.9 ns/op BenchmarkMmap/file:10.0MiB_record:8.0KiB_cache:100_B 50000000 32.9 ns/op BenchmarkMmap/file:10.0MiB_record:12.0KiB_cache:100_B 50000000 34.1 ns/op BenchmarkMmap/file:10.0MiB_record:16.0KiB_cache:100_B 50000000 35.4 ns/op BenchmarkMmap/file:10.0MiB_record:1.0MiB_cache:100_B 50000000 37.9 ns/op BenchmarkMmap/file:10.0MiB_record:0_B_cache:1.0KiB 10000000 118 ns/op BenchmarkMmap/file:10.0MiB_record:100_B_cache:1.0KiB 20000000 119 ns/op BenchmarkMmap/file:10.0MiB_record:1.0KiB_cache:1.0KiB 10000000 119 ns/op BenchmarkMmap/file:10.0MiB_record:2.0KiB_cache:1.0KiB 20000000 113 ns/op BenchmarkMmap/file:10.0MiB_record:4.0KiB_cache:1.0KiB 20000000 105 ns/op BenchmarkMmap/file:10.0MiB_record:8.0KiB_cache:1.0KiB 20000000 109 ns/op BenchmarkMmap/file:10.0MiB_record:12.0KiB_cache:1.0KiB 10000000 113 ns/op BenchmarkMmap/file:10.0MiB_record:16.0KiB_cache:1.0KiB 10000000 121 ns/op BenchmarkMmap/file:10.0MiB_record:1.0MiB_cache:1.0KiB 10000000 203 ns/op BenchmarkMmap/file:10.0MiB_record:0_B_cache:2.0KiB 10000000 222 ns/op BenchmarkMmap/file:10.0MiB_record:100_B_cache:2.0KiB 10000000 218 ns/op BenchmarkMmap/file:10.0MiB_record:1.0KiB_cache:2.0KiB 10000000 221 ns/op BenchmarkMmap/file:10.0MiB_record:2.0KiB_cache:2.0KiB 10000000 222 ns/op BenchmarkMmap/file:10.0MiB_record:4.0KiB_cache:2.0KiB 10000000 186 ns/op BenchmarkMmap/file:10.0MiB_record:8.0KiB_cache:2.0KiB 10000000 186 ns/op BenchmarkMmap/file:10.0MiB_record:12.0KiB_cache:2.0KiB 10000000 196 ns/op BenchmarkMmap/file:10.0MiB_record:16.0KiB_cache:2.0KiB 10000000 212 ns/op BenchmarkMmap/file:10.0MiB_record:1.0MiB_cache:2.0KiB 5000000 390 ns/op BenchmarkMmap/file:100.0MiB_record:0_B_cache:10_B 50000000 35.3 ns/op BenchmarkMmap/file:100.0MiB_record:100_B_cache:10_B 50000000 26.2 ns/op BenchmarkMmap/file:100.0MiB_record:1.0KiB_cache:10_B 100000000 23.5 ns/op BenchmarkMmap/file:100.0MiB_record:2.0KiB_cache:10_B 100000000 23.9 ns/op BenchmarkMmap/file:100.0MiB_record:4.0KiB_cache:10_B 50000000 23.6 ns/op BenchmarkMmap/file:100.0MiB_record:8.0KiB_cache:10_B 100000000 23.9 ns/op BenchmarkMmap/file:100.0MiB_record:12.0KiB_cache:10_B 50000000 23.9 ns/op BenchmarkMmap/file:100.0MiB_record:16.0KiB_cache:10_B 50000000 24.1 ns/op BenchmarkMmap/file:100.0MiB_record:1.0MiB_cache:10_B 50000000 24.1 ns/op BenchmarkMmap/file:100.0MiB_record:0_B_cache:100_B 20000000 276 ns/op BenchmarkMmap/file:100.0MiB_record:100_B_cache:100_B 10000000 224 ns/op BenchmarkMmap/file:100.0MiB_record:1.0KiB_cache:100_B 50000000 45.4 ns/op BenchmarkMmap/file:100.0MiB_record:2.0KiB_cache:100_B 50000000 45.0 ns/op BenchmarkMmap/file:100.0MiB_record:4.0KiB_cache:100_B 50000000 35.8 ns/op BenchmarkMmap/file:100.0MiB_record:8.0KiB_cache:100_B 50000000 34.7 ns/op BenchmarkMmap/file:100.0MiB_record:12.0KiB_cache:100_B 50000000 34.8 ns/op BenchmarkMmap/file:100.0MiB_record:16.0KiB_cache:100_B 50000000 35.3 ns/op BenchmarkMmap/file:100.0MiB_record:1.0MiB_cache:100_B 50000000 38.3 ns/op BenchmarkMmap/file:100.0MiB_record:0_B_cache:1.0KiB 1000000 1397 ns/op BenchmarkMmap/file:100.0MiB_record:100_B_cache:1.0KiB 1000000 1951 ns/op BenchmarkMmap/file:100.0MiB_record:1.0KiB_cache:1.0KiB 1000000 1612 ns/op BenchmarkMmap/file:100.0MiB_record:2.0KiB_cache:1.0KiB 1000000 1317 ns/op BenchmarkMmap/file:100.0MiB_record:4.0KiB_cache:1.0KiB 2000000 646 ns/op BenchmarkMmap/file:100.0MiB_record:8.0KiB_cache:1.0KiB 5000000 384 ns/op BenchmarkMmap/file:100.0MiB_record:12.0KiB_cache:1.0KiB 10000000 222 ns/op BenchmarkMmap/file:100.0MiB_record:16.0KiB_cache:1.0KiB 10000000 176 ns/op BenchmarkMmap/file:100.0MiB_record:1.0MiB_cache:1.0KiB 10000000 208 ns/op BenchmarkMmap/file:100.0MiB_record:0_B_cache:2.0KiB 1000000 2457 ns/op BenchmarkMmap/file:100.0MiB_record:100_B_cache:2.0KiB 1000000 2969 ns/op BenchmarkMmap/file:100.0MiB_record:1.0KiB_cache:2.0KiB 1000000 2918 ns/op BenchmarkMmap/file:100.0MiB_record:2.0KiB_cache:2.0KiB 1000000 2965 ns/op BenchmarkMmap/file:100.0MiB_record:4.0KiB_cache:2.0KiB 1000000 1909 ns/op BenchmarkMmap/file:100.0MiB_record:8.0KiB_cache:2.0KiB 2000000 1317 ns/op BenchmarkMmap/file:100.0MiB_record:12.0KiB_cache:2.0KiB 1000000 1018 ns/op BenchmarkMmap/file:100.0MiB_record:16.0KiB_cache:2.0KiB 2000000 642 ns/op BenchmarkMmap/file:100.0MiB_record:1.0MiB_cache:2.0KiB 3000000 405 ns/op BenchmarkMmap/file:1.0GiB_record:0_B_cache:10_B 50000000 118 ns/op BenchmarkMmap/file:1.0GiB_record:100_B_cache:10_B 50000000 26.1 ns/op BenchmarkMmap/file:1.0GiB_record:1.0KiB_cache:10_B 50000000 23.4 ns/op BenchmarkMmap/file:1.0GiB_record:2.0KiB_cache:10_B 100000000 23.5 ns/op BenchmarkMmap/file:1.0GiB_record:4.0KiB_cache:10_B 100000000 24.0 ns/op BenchmarkMmap/file:1.0GiB_record:8.0KiB_cache:10_B 100000000 23.4 ns/op BenchmarkMmap/file:1.0GiB_record:12.0KiB_cache:10_B 100000000 23.7 ns/op BenchmarkMmap/file:1.0GiB_record:16.0KiB_cache:10_B 50000000 24.1 ns/op BenchmarkMmap/file:1.0GiB_record:1.0MiB_cache:10_B 50000000 24.2 ns/op BenchmarkMmap/file:1.0GiB_record:0_B_cache:100_B 20000000 1408 ns/op BenchmarkMmap/file:1.0GiB_record:100_B_cache:100_B 1000000 1528 ns/op BenchmarkMmap/file:1.0GiB_record:1.0KiB_cache:100_B 50000000 110 ns/op BenchmarkMmap/file:1.0GiB_record:2.0KiB_cache:100_B 30000000 72.6 ns/op BenchmarkMmap/file:1.0GiB_record:4.0KiB_cache:100_B 50000000 35.2 ns/op BenchmarkMmap/file:1.0GiB_record:8.0KiB_cache:100_B 50000000 34.7 ns/op BenchmarkMmap/file:1.0GiB_record:12.0KiB_cache:100_B 50000000 36.1 ns/op BenchmarkMmap/file:1.0GiB_record:16.0KiB_cache:100_B 50000000 35.4 ns/op BenchmarkMmap/file:1.0GiB_record:1.0MiB_cache:100_B 50000000 38.6 ns/op BenchmarkMmap/file:1.0GiB_record:0_B_cache:1.0KiB 1000000 13028 ns/op BenchmarkMmap/file:1.0GiB_record:100_B_cache:1.0KiB 100000 14377 ns/op BenchmarkMmap/file:1.0GiB_record:1.0KiB_cache:1.0KiB 1000000 14410 ns/op BenchmarkMmap/file:1.0GiB_record:2.0KiB_cache:1.0KiB 300000 7779 ns/op BenchmarkMmap/file:1.0GiB_record:4.0KiB_cache:1.0KiB 500000 3922 ns/op BenchmarkMmap/file:1.0GiB_record:8.0KiB_cache:1.0KiB 1000000 1994 ns/op BenchmarkMmap/file:1.0GiB_record:12.0KiB_cache:1.0KiB 1000000 1301 ns/op BenchmarkMmap/file:1.0GiB_record:16.0KiB_cache:1.0KiB 10000000 852 ns/op BenchmarkMmap/file:1.0GiB_record:1.0MiB_cache:1.0KiB 10000000 209 ns/op BenchmarkMmap/file:1.0GiB_record:0_B_cache:2.0KiB 1000000 29845 ns/op BenchmarkMmap/file:1.0GiB_record:100_B_cache:2.0KiB 50000 25536 ns/op BenchmarkMmap/file:1.0GiB_record:1.0KiB_cache:2.0KiB 1000000 30416 ns/op BenchmarkMmap/file:1.0GiB_record:2.0KiB_cache:2.0KiB 50000 25693 ns/op BenchmarkMmap/file:1.0GiB_record:4.0KiB_cache:2.0KiB 1000000 14409 ns/op BenchmarkMmap/file:1.0GiB_record:8.0KiB_cache:2.0KiB 500000 7886 ns/op BenchmarkMmap/file:1.0GiB_record:12.0KiB_cache:2.0KiB 500000 5283 ns/op BenchmarkMmap/file:1.0GiB_record:16.0KiB_cache:2.0KiB 500000 3957 ns/op BenchmarkMmap/file:1.0GiB_record:1.0MiB_cache:2.0KiB 3000000 399 ns/op --- BENCH: BenchmarkMmap bench_test.go:45: host page size : 4096 PASS ok _/home/ubuntu/bench 351.013s ```Ubuntu(ec2) + HDD
| 項目 | 内容 |
|---|---|
| AMI | Ubuntu Server 18.04 LTS (HVM), SSD Volume Type - ami-07ad4b1c3af1ea214 |
| Instance | t2.micro |
| Disk | HDD(standard) |
| Page Size | 4096 B |
| Go Version | 1.10.1 |
レコードを追記する場合に、１件ごと Mmap された領域へコピーする場合と、バッファリングして複数件を一度にコピーする場合でどの程度パフォーマンスに開きがあるのか調査する