Dieterbe
commented
9 years ago +1
Closed woodsaj closed 9 years ago
Dieterbe
commented
9 years ago +1
vrecan
commented
9 years ago this looks good but I don't see any test that enables refresh and validates the request is using it.
vrecan
commented
9 years ago :+1:
woodsaj
commented
9 years ago Any specific reason this PR cant be merged? Not being able to use the BulkAPI is a pretty big issue for us.
mattbaird
commented
9 years ago +1
weberr13
commented
9 years ago I will merge this but keep in mind that
go test -race ./...
both before and after your changes finds several race conditions in corebulk.go:
WARNING: DATA RACE Write by goroutine 18: github.com/mattbaird/elastigo/lib.func·025() /home/weberr/go/src/github.com/mattbaird/elastigo/lib/corebulk_test.go:66 +0x232 github.com/mattbaird/elastigo/lib.func·003() /home/weberr/go/src/github.com/mattbaird/elastigo/lib/corebulk.go:198 +0x2ba
Previous read by goroutine 6: github.com/mattbaird/elastigo/lib.func·026() /home/weberr/go/src/github.com/mattbaird/elastigo/lib/corebulk_test.go:81 +0x3a github.com/mattbaird/elastigo/lib.waitFor() /home/weberr/go/src/github.com/mattbaird/elastigo/lib/coretest_test.go:96 +0xb1 github.com/mattbaird/elastigo/lib.TestBulkIndexerBasic() /home/weberr/go/src/github.com/mattbaird/elastigo/lib/corebulk_test.go:82 +0x779 testing.tRunner() /usr/local/go/src/testing/testing.go:447 +0x133
Goroutine 18 (running) created at: github.com/mattbaird/elastigo/lib.(*BulkIndexer).startHttpSender() /home/weberr/go/src/github.com/mattbaird/elastigo/lib/corebulk.go:220 +0x108 github.com/mattbaird/elastigo/lib.func·001() /home/weberr/go/src/github.com/mattbaird/elastigo/lib/corebulk.go:140 +0x14e
Goroutine 6 (running) created at: testing.RunTests() /usr/local/go/src/testing/testing.go:555 +0xd4e testing.(*M).Run() /usr/local/go/src/testing/testing.go:485 +0xe0 main.main()
WARNING: DATA RACE Read by goroutine 6: github.com/mattbaird/elastigo/lib.TestBulkIndexerBasic() /home/weberr/go/src/github.com/mattbaird/elastigo/lib/corebulk_test.go:89 +0xd4f testing.tRunner() /usr/local/go/src/testing/testing.go:447 +0x133
Previous write by goroutine 18: github.com/mattbaird/elastigo/lib.func·025() /home/weberr/go/src/github.com/mattbaird/elastigo/lib/corebulk_test.go:65 +0x10f github.com/mattbaird/elastigo/lib.func·003() /home/weberr/go/src/github.com/mattbaird/elastigo/lib/corebulk.go:198 +0x2ba
Goroutine 6 (running) created at: testing.RunTests() /usr/local/go/src/testing/testing.go:555 +0xd4e testing.(*M).Run() /usr/local/go/src/testing/testing.go:485 +0xe0 main.main() github.com/mattbaird/elastigo/lib/_test/_testmain.go:106 +0x28c
Goroutine 18 (running) created at: github.com/mattbaird/elastigo/lib.(*BulkIndexer).startHttpSender() /home/weberr/go/src/github.com/mattbaird/elastigo/lib/corebulk.go:220 +0x108 github.com/mattbaird/elastigo/lib.func·001()
WARNING: DATA RACE Read by goroutine 19: github.com/mattbaird/elastigo/lib.func·025() /home/weberr/go/src/github.com/mattbaird/elastigo/lib/corebulk_test.go:64 +0x78 github.com/mattbaird/elastigo/lib.func·003() /home/weberr/go/src/github.com/mattbaird/elastigo/lib/corebulk.go:198 +0x2ba
Previous write by goroutine 18: github.com/mattbaird/elastigo/lib.func·025() /home/weberr/go/src/github.com/mattbaird/elastigo/lib/corebulk_test.go:64 +0x8e github.com/mattbaird/elastigo/lib.func·003() /home/weberr/go/src/github.com/mattbaird/elastigo/lib/corebulk.go:198 +0x2ba
Goroutine 19 (running) created at: github.com/mattbaird/elastigo/lib.(*BulkIndexer).startHttpSender() /home/weberr/go/src/github.com/mattbaird/elastigo/lib/corebulk.go:220 +0x108 github.com/mattbaird/elastigo/lib.func·001() /home/weberr/go/src/github.com/mattbaird/elastigo/lib/corebulk.go:140 +0x14e
Goroutine 18 (running) created at: github.com/mattbaird/elastigo/lib.(*BulkIndexer).startHttpSender() /home/weberr/go/src/github.com/mattbaird/elastigo/lib/corebulk.go:220 +0x108 github.com/mattbaird/elastigo/lib.func·001()
WARNING: DATA RACE Read by goroutine 19: runtime.growslice() /usr/local/go/src/runtime/slice.go:37 +0x0 github.com/mattbaird/elastigo/lib.func·025() /home/weberr/go/src/github.com/mattbaird/elastigo/lib/corebulk_test.go:66 +0x18d github.com/mattbaird/elastigo/lib.func·003() /home/weberr/go/src/github.com/mattbaird/elastigo/lib/corebulk.go:198 +0x2ba
Previous write by goroutine 18: github.com/mattbaird/elastigo/lib.func·025() /home/weberr/go/src/github.com/mattbaird/elastigo/lib/corebulk_test.go:66 +0x1cd github.com/mattbaird/elastigo/lib.func·003() /home/weberr/go/src/github.com/mattbaird/elastigo/lib/corebulk.go:198 +0x2ba
Goroutine 19 (running) created at: github.com/mattbaird/elastigo/lib.(*BulkIndexer).startHttpSender() /home/weberr/go/src/github.com/mattbaird/elastigo/lib/corebulk.go:220 +0x108 github.com/mattbaird/elastigo/lib.func·001() /home/weberr/go/src/github.com/mattbaird/elastigo/lib/corebulk.go:140 +0x14e
Goroutine 18 (running) created at: github.com/mattbaird/elastigo/lib.(*BulkIndexer).startHttpSender() /home/weberr/go/src/github.com/mattbaird/elastigo/lib/corebulk.go:220 +0x108 github.com/mattbaird/elastigo/lib.func·001()
WARNING: DATA RACE Write by goroutine 30: sync.raceWrite() /usr/local/go/src/sync/race.go:41 +0x36 sync.(*WaitGroup).Wait() /usr/local/go/src/sync/waitgroup.go:125 +0x177 github.com/mattbaird/elastigo/lib.func·002() /home/weberr/go/src/github.com/mattbaird/elastigo/lib/corebulk.go:167 +0x53
Previous read by goroutine 27: sync.raceRead() /usr/local/go/src/sync/race.go:37 +0x36 sync.(*WaitGroup).Add() /usr/local/go/src/sync/waitgroup.go:63 +0xc1 github.com/mattbaird/elastigo/lib.func·003() /home/weberr/go/src/github.com/mattbaird/elastigo/lib/corebulk.go:195 +0xfa
Goroutine 30 (running) created at: github.com/mattbaird/elastigo/lib.wgChan() /home/weberr/go/src/github.com/mattbaird/elastigo/lib/corebulk.go:169 +0x159 github.com/mattbaird/elastigo/lib.(*BulkIndexer).shutdown() /home/weberr/go/src/github.com/mattbaird/elastigo/lib/corebulk.go:283 +0xc1 github.com/mattbaird/elastigo/lib.func·001() /home/weberr/go/src/github.com/mattbaird/elastigo/lib/corebulk.go:146 +0x239
Goroutine 27 (running) created at: github.com/mattbaird/elastigo/lib.(*BulkIndexer).startHttpSender() /home/weberr/go/src/github.com/mattbaird/elastigo/lib/corebulk.go:220 +0x108 github.com/mattbaird/elastigo/lib.func·001() /home/weberr/go/src/github.com/mattbaird/elastigo/lib/corebulk.go:140 +0x14e
I don't see a reason that your code could have caused this, but you should be aware of these race conditions if you are relying on the bulk API. I will track this issue here:
This commit moves the refresh property to be an attribute of the BulkIndexer struct. The refresh attribute is included as a query param when the request is sent to the Elasticsearch API.