Enable bulk mode - Githubissues

hcira commented 8 years ago

I want to enable bulk mode for elastic search output:

The way that I find to do that is to update the nodeJS source file: FILE: lib/outputs/output_elasticsearch.js function OutputElasticSearch() { abstract_http.AbstractHttp.call(this); this.mergeConfig({ name: 'Elastic search', optional_params: ['bulk_limit', 'bulk_timeout', 'index_prefix', 'data_type', 'index_name'], default_values: { 'index_prefix': 'logstash', 'data_type': 'logs', 'bulk': 'true', /* UPDARTED FROM 'false' TO 'true' */ 'bulk_limit': '500', /* ADDED THIS LINE */ 'bulk_timeout': '1000' }, start_hook: this.start, }); }

By doing that, I get rid of ElasticSearch error 429 (too many requests)

BUT, I got another error 400 (bad request), after Investigation, I find that it happens when the batch contain one request! So, after reading some docs related to ElasticSearch bulk API, I added a new line in the request body to resolve this issue: FILE: lib/outputs/abstract_http.js AbstractHttp.prototype.sendHttpRequest = function(http_options, body) { body = body + '\n'; /* UPDATED THIS LINE BY ADDING ( + '\n') */ if (this.proxying_agent) { http_options.agent = this.proxying_agent; } if (this.path_prefix) { http_options.path = this.path_prefix + http_options.path; }

Please let me know if there is a proper way to do that, thanks!

bpaquet commented 8 years ago

Hi Hcira,

The bulk_mode is not enabled by default. You have to set the bulk_limit param. See doc. Your code modification will enable it by default.
I'm not able to reproduce your issue with one line in bulk mode with ES 2.1.1. Which version of ES are you using ? Which line are you trying to inject ?

Regards,

hcira commented 8 years ago

Thank you for your response! My ES is: { "status" : 200, "name" : "Lady Lark", "cluster_name" : "elasticsearch", "version" : { "number" : "1.5.0", "build_hash" : "544816042d40151d3ce4ba4f95399d7860dc2e92", "build_timestamp" : "2015-03-23T14:30:58Z", "build_snapshot" : false, "lucene_version" : "4.10.4" }, "tagline" : "You Know, for Search" }

The line that I added to get rid of error 400 is in FILE: lib/outputs/abstract_http.js: `AbstractHttp.prototype.sendHttpRequest

.....

body = body + '\n'; /I inject this line/`

Thanks!

bpaquet commented 8 years ago

Hi,

I'm not able to reproduce the problem with ES 1.5.0. Can you provide your node-logstash version ? Your node-logstash config ? And a tcpdump beteween node-logstash and ES (sudo tcpdump -A -s0 -i lo0 port 9200 on my mac).

Regards,

Bertrand

On Wed, Feb 3, 2016 at 3:25 PM, hcira notifications@github.com wrote:

Thank you for your response! My ES is: { "status" : 200, "name" : "Lady Lark", "cluster_name" : "elasticsearch", "version" : { "number" : "1.5.0", "build_hash" : "544816042d40151d3ce4ba4f95399d7860dc2e92", "build_timestamp" : "2015-03-23T14:30:58Z", "build_snapshot" : false, "lucene_version" : "4.10.4" }, "tagline" : "You Know, for Search" }

The line that I added to get rid of error 400 is in FILE: lib/outputs/abstract_http.js: `AbstractHttp.prototype.sendHttpRequest

.....

body = body + '\n'; /I inject this line/`

Thanks!

— Reply to this email directly or view it on GitHub https://github.com/bpaquet/node-logstash/issues/119#issuecomment-179264386 .

hcira commented 8 years ago

Hi,

My node-logstash is from this github repository...

the sample of failed request is:

4SPOST /hdfslogs-2016.01.29/logs/_bulk HTTP/1.1 Host: elasticsealan.sociablelabs.net:9200 Connection: close Transfer-Encoding: chunked {"index": {}} {"message":"2016-01-29 20:15:05,280 INFO datanode.BlockPoolSliceScanner (BlockPoolSliceScanner.java:verifyBlock(444)) - Verification succeeded for BP-205801576-10.0.100.126-1422487659507:blk_1075529110_1788306","path":"/var/log/hadoop/hdfs/hadoop-hdfs-datanode-ip-10-0-100-136.log","type":"hdfslog","host":"ip-10-0-100-136","@timestamp":"2016-01-29T20:15:05.280Z","@version":"1","timestamp":"2016-01-29 20:15:05,280","severity":"INFO","msg":" datanode.BlockPoolSliceScanner (BlockPoolSliceScanner.java:verifyBlock(444)) - Verification succeeded for BP-205801576-10.0.100.126-1422487659507:blk_1075529110_1788306"} 9j@@ HTTP/1.1 400 Bad Request Content-Type: application/json; charset=UTF-8 Content-Length: 99 {"error":"ActionRequestValidationException[Validation Failed: 1: no requests added;]","status":400}J

Thanks

bpaquet commented 8 years ago

Hi,

I do not understand how your node logstash can send this request

See https://github.com/bpaquet/node-logstash/blob/master/lib/outputs/output_elasticsearch.js#L63 : the {index:{}} line should not be empty
The request send to ES is incorrect because {index:{}} is empty.

What is your node-logstash config ?

Bertrand

On Wed, Feb 3, 2016 at 5:43 PM, hcira notifications@github.com wrote:

Hi,

My node-logstash is from this github repository...

the sample of failed request is:

4SPOST /hdfslogs-2016.01.29/logs/_bulk HTTP/1.1 Host: elasticsealan.sociablelabs.net:9200 Connection: close Transfer-Encoding: chunked {"index": {}} {"message":"2016-01-29 20:15:05,280 INFO datanode.BlockPoolSliceScanner (BlockPoolSliceScanner.java:verifyBlock(444)) - Verification succeeded for BP-205801576-10.0.100.126-1422487659507:blk_1075529110_1788306","path":"/var/log/hadoop/hdfs/hadoop-hdfs-datanode-ip-10-0-100-136.log","type":"hdfslog","host":"ip-10-0-100-136"," @timestamp https://github.com/timestamp":"2016-01-29T20:15:05.280Z"," @version https://github.com/version":"1","timestamp":"2016-01-29 20:15:05,280","severity":"INFO","msg":" datanode.BlockPoolSliceScanner (BlockPoolSliceScanner.java:verifyBlock(444)) - Verification succeeded for BP-205801576-10.0.100.126-1422487659507:blk_1075529110_1788306"} 9j@@ HTTP/1.1 400 Bad Request Content-Type: application/json; charset=UTF-8 Content-Length: 99 {"error":"ActionRequestValidationException[Validation Failed: 1: no requests added;]","status":400}J

Thanks

— Reply to this email directly or view it on GitHub https://github.com/bpaquet/node-logstash/issues/119#issuecomment-179335760 .

hcira commented 8 years ago

Hi Bertrand,

{index:{}} is empty as all informations are sent within the url /hdfslogs-2016.01.29/logs/_bulk which is the same as { "index" : { "_index" : "hdfslogs-2016.01.29", "_type" : "log" } with url /_bulk

almost all requests passed correctly except the one request (two lines, metadata and source)

but by adding '\n' at the end of the request body (see previous comments) every things is fine, no error.

my config related to the previous example is:

output://elasticsearch://elasticsealan.sociablelabs.net:9200?index_prefix=hdfslogs&only_type=hdfslog&bulk_limit=500

Thanks my friends!

bpaquet commented 8 years ago

Hi,

The elasticsearch output plugins only push to /_bulk, not to /hdfslogs-2016.01.29/logs. I do not understand how do you manage to have such trace with node-logstash. https://github.com/bpaquet/node-logstash/blob/master/lib/outputs/output_elasticsearch.js#L46

Anyway, you are right, your request is ok (with the final \n).

Bertrand

On Wed, Feb 3, 2016 at 11:04 PM, hcira notifications@github.com wrote:

Hi Bertrand,

{index:{}} is empty as all informations are sent within the url /hdfslogs-2016.01.29/logs/_bulk which is the same as { "index" : { "_index" : "hdfslogs-2016.01.29", "_type" : "log" } with url /_bulk

almost all requests passed correctly except the one request (two lines, metadata and source)

but by adding '\n' at the end of the request body (see previous comments) every things is fine, no error.

Thanks my friends!

— Reply to this email directly or view it on GitHub https://github.com/bpaquet/node-logstash/issues/119#issuecomment-179494002 .

bpaquet / node-logstash

Enable bulk mode #119