With a large cluster, using NodeClient it is possible to encounter 'too many open files' errors due to the full-mesh connections between each elasticsearch node (including hadoop workers).
To counter that, I added the possibility to use TransportClient and define the desire entry-point nodes, thus limiting the number of connections each hadoop workers has. Using this type of client introduce 'double-hop' request for indexing. So far, I didn't saw much decrease in performance, but there is certainly one, which is counterbalanced by the ability to use all the nodes on the cluster.
Feel free to accept the pull request, but I am using it in production for 1-2 weeks without any issues.
With a large cluster, using NodeClient it is possible to encounter 'too many open files' errors due to the full-mesh connections between each elasticsearch node (including hadoop workers).
To counter that, I added the possibility to use TransportClient and define the desire entry-point nodes, thus limiting the number of connections each hadoop workers has. Using this type of client introduce 'double-hop' request for indexing. So far, I didn't saw much decrease in performance, but there is certainly one, which is counterbalanced by the ability to use all the nodes on the cluster.
Feel free to accept the pull request, but I am using it in production for 1-2 weeks without any issues.