search

miku / esbulk

Bulk indexing command line tool for elasticsearch.
GNU General Public License v3.0
281 stars 40 forks source link

Workers #33

Open claytondukes opened 4 years ago

claytondukes commented 4 years ago

I'm curious about the -w option. I have a 40GB file to import but it doesn't seem to matter how much CPU or mem I give the system. I have 192 cores and ~300GB ram but can't get -w to go past -w 7. I have 6 nodes in this cluster.

2020/06/22 16:20:36 error during bulk operation, check error details; maybe try fewer workers (-w) or increase thread_pool.bulk.queue_size in your nodes

FWIW, htop shows barely any utilization while I am running the import. Am I doing something wrong?

miku commented 4 years ago

Thanks for the bug report! I'll prioritize the improved error reporting, which I meant to implement for some time now.

Apart from that, I do not see any error (and usage should be more or less self-explanatory).