miku / esbulk

Bulk indexing command line tool for elasticsearch.
GNU General Public License v3.0
278 stars 41 forks source link

Show Error reasons when bulk fails #7

Closed shadyabhi closed 7 years ago

shadyabhi commented 7 years ago

The current version is misleading when it comes to reporting errors. For ex, if I try to push a malformed json, it reports this error:-

$ ./esbulk  -index rum-hello -server http://vimehta-ld1:9200  -w 1 docs  
2017/06/06 04:13:45 error during bulk operation, try less workers (lower -w value) or increase thread_pool.bulk.queue_size in your nodes
$ 

Now, a new user will be confused seeing this error and will think that it's actually due to finishing up bulk threads at Elasticsearch side. This PR makes the error a little more informant.

% ./esbulk -index test_index -server http://arastogi-ld1:9200/ ~/docs.json
2017/06/06 05:59:05 error during bulk operation, try less workers (lower -w value) or increase thread_pool.bulk.queue_size in your nodes or check error on server. Errors: [failed to parse]
%

What do you think? Is there a better way to show errors?

miku commented 7 years ago

Thanks a lot for this PR. You are definitely right, that error reporting needs to be better.

However, if I run:

$ cat fixtures/pr-7.json
{"x: "doc-1", "db": "mysql"}
{"x": "doc-2", "db": "mongo"}

$ go run cmd/esbulk/esbulk.go -index throwaway fixtures/pr-7.json
2017/06/07 09:31:23 error during bulk operation, try less workers (lower -w value)
or increase thread_pool.bulk.queue_size in your nodes or check error on server. Errors: []

the error list is empty.

shadyabhi commented 7 years ago

Hi @miku,

That's probably because cmd/esbulk/esbulk.go is importing github.com/miku/esbulk. Are you sure that your code in go workspace (echo $GOPATH/src/github.com/miku/esbulk) is modified as per the patch?

miku commented 7 years ago

This is resolved by #9.