ES 6.1t

[gransy]

After upgrade ES to 6.1, the esbulk works incorrectly for me.

I start command with processes by count of cores, and with verbose i see how all processes doing bulk insert .. in few time a lot of processes ended without any error (i try put some logging) and thee is keep just one or two who fill finish insert. After last inserted data, the program is not finished and keep on this unknown state without any logging.

In total in ES is inserted total_docs-(10000*(processes-1)). In 24 processes i lost about 230.000 records.

[miku]

@gransy, if you overload the server by using too many workers, failures are expected - as the elasticsearch bulk API will return an error and this error let's esbulk stop the import process (see: caveat).

Can you try to indexing with fewer workers? If it still fails, can you share a log - e.g. output with -verbose flag?

[gransy]

Hi,

i use a command:

esbulk -server http://server:9200 -index logs-2018-02-04 -verbose 2018-02-04_.log.json >> esb.log 2> esb2.log

But command is no ever ending. I was kill it (Ctrl+C) after some hours.

The log is attached. As you see, each workers finish early during the process.

esb2.log

[miku]

@gransy, thanks for providing more details.

But command is no ever ending.

That's unfortunate and something I haven't encountered yet. As a first measure, I will try to add more logging information to esbulk, so debugging might become easier.

As a sidenote: Please make sure, you are always using the newest version, at this moment, this would be esbulk 0.4.10.

[gransy]

Hi,

finally my issue is not about your processing data, but about your handling errors. I found the problem in my cluster - one data node is not worked correctly. After correct my cluster, esbulk works again perfect. For this reasson i think, that there is something wrong in error detection.

Jan

[miku]

@gransy, glad it worked out.