Closed hcira closed 8 years ago
Hi Hcira,
Regards,
Thank you for your response!
My ES is:
{ "status" : 200, "name" : "Lady Lark", "cluster_name" : "elasticsearch", "version" : { "number" : "1.5.0", "build_hash" : "544816042d40151d3ce4ba4f95399d7860dc2e92", "build_timestamp" : "2015-03-23T14:30:58Z", "build_snapshot" : false, "lucene_version" : "4.10.4" }, "tagline" : "You Know, for Search" }
The line that I added to get rid of error 400 is in FILE: lib/outputs/abstract_http.js: `AbstractHttp.prototype.sendHttpRequest
.....
body = body + '\n'; /I inject this line/`
Thanks!
Hi,
I'm not able to reproduce the problem with ES 1.5.0. Can you provide your node-logstash version ? Your node-logstash config ? And a tcpdump beteween node-logstash and ES (sudo tcpdump -A -s0 -i lo0 port 9200 on my mac).
Regards,
Bertrand
On Wed, Feb 3, 2016 at 3:25 PM, hcira notifications@github.com wrote:
Thank you for your response! My ES is: { "status" : 200, "name" : "Lady Lark", "cluster_name" : "elasticsearch", "version" : { "number" : "1.5.0", "build_hash" : "544816042d40151d3ce4ba4f95399d7860dc2e92", "build_timestamp" : "2015-03-23T14:30:58Z", "build_snapshot" : false, "lucene_version" : "4.10.4" }, "tagline" : "You Know, for Search" }
The line that I added to get rid of error 400 is in FILE: lib/outputs/abstract_http.js: `AbstractHttp.prototype.sendHttpRequest
.....
body = body + '\n'; /I inject this line/`
Thanks!
— Reply to this email directly or view it on GitHub https://github.com/bpaquet/node-logstash/issues/119#issuecomment-179264386 .
Hi,
My node-logstash is from this github repository...
the sample of failed request is:
4SPOST /hdfslogs-2016.01.29/logs/_bulk HTTP/1.1 Host: elasticsealan.sociablelabs.net:9200 Connection: close Transfer-Encoding: chunked {"index": {}} {"message":"2016-01-29 20:15:05,280 INFO datanode.BlockPoolSliceScanner (BlockPoolSliceScanner.java:verifyBlock(444)) - Verification succeeded for BP-205801576-10.0.100.126-1422487659507:blk_1075529110_1788306","path":"/var/log/hadoop/hdfs/hadoop-hdfs-datanode-ip-10-0-100-136.log","type":"hdfslog","host":"ip-10-0-100-136","@timestamp":"2016-01-29T20:15:05.280Z","@version":"1","timestamp":"2016-01-29 20:15:05,280","severity":"INFO","msg":" datanode.BlockPoolSliceScanner (BlockPoolSliceScanner.java:verifyBlock(444)) - Verification succeeded for BP-205801576-10.0.100.126-1422487659507:blk_1075529110_1788306"} 9j@@ HTTP/1.1 400 Bad Request Content-Type: application/json; charset=UTF-8 Content-Length: 99 {"error":"ActionRequestValidationException[Validation Failed: 1: no requests added;]","status":400}J
Thanks
Hi,
I do not understand how your node logstash can send this request
What is your node-logstash config ?
Bertrand
On Wed, Feb 3, 2016 at 5:43 PM, hcira notifications@github.com wrote:
Hi,
My node-logstash is from this github repository...
the sample of failed request is:
4SPOST /hdfslogs-2016.01.29/logs/_bulk HTTP/1.1 Host: elasticsealan.sociablelabs.net:9200 Connection: close Transfer-Encoding: chunked {"index": {}} {"message":"2016-01-29 20:15:05,280 INFO datanode.BlockPoolSliceScanner (BlockPoolSliceScanner.java:verifyBlock(444)) - Verification succeeded for BP-205801576-10.0.100.126-1422487659507:blk_1075529110_1788306","path":"/var/log/hadoop/hdfs/hadoop-hdfs-datanode-ip-10-0-100-136.log","type":"hdfslog","host":"ip-10-0-100-136"," @timestamp https://github.com/timestamp":"2016-01-29T20:15:05.280Z"," @version https://github.com/version":"1","timestamp":"2016-01-29 20:15:05,280","severity":"INFO","msg":" datanode.BlockPoolSliceScanner (BlockPoolSliceScanner.java:verifyBlock(444)) - Verification succeeded for BP-205801576-10.0.100.126-1422487659507:blk_1075529110_1788306"} 9j@@ HTTP/1.1 400 Bad Request Content-Type: application/json; charset=UTF-8 Content-Length: 99 {"error":"ActionRequestValidationException[Validation Failed: 1: no requests added;]","status":400}J
Thanks
— Reply to this email directly or view it on GitHub https://github.com/bpaquet/node-logstash/issues/119#issuecomment-179335760 .
Hi Bertrand,
{index:{}} is empty as all informations are sent within the url /hdfslogs-2016.01.29/logs/_bulk which is the same as { "index" : { "_index" : "hdfslogs-2016.01.29", "_type" : "log" } with url /_bulk
almost all requests passed correctly except the one request (two lines, metadata and source)
but by adding '\n' at the end of the request body (see previous comments) every things is fine, no error.
my config related to the previous example is:
output://elasticsearch://elasticsealan.sociablelabs.net:9200?index_prefix=hdfslogs&only_type=hdfslog&bulk_limit=500
Thanks my friends!
Hi,
The elasticsearch output plugins only push to /_bulk, not to /hdfslogs-2016.01.29/logs. I do not understand how do you manage to have such trace with node-logstash. https://github.com/bpaquet/node-logstash/blob/master/lib/outputs/output_elasticsearch.js#L46
Anyway, you are right, your request is ok (with the final \n).
Bertrand
On Wed, Feb 3, 2016 at 11:04 PM, hcira notifications@github.com wrote:
Hi Bertrand,
{index:{}} is empty as all informations are sent within the url /hdfslogs-2016.01.29/logs/_bulk which is the same as { "index" : { "_index" : "hdfslogs-2016.01.29", "_type" : "log" } with url /_bulk
almost all requests passed correctly except the one request (two lines, metadata and source)
but by adding '\n' at the end of the request body (see previous comments) every things is fine, no error.
Thanks my friends!
— Reply to this email directly or view it on GitHub https://github.com/bpaquet/node-logstash/issues/119#issuecomment-179494002 .
I want to enable bulk mode for elastic search output:
The way that I find to do that is to update the nodeJS source file: FILE: lib/outputs/output_elasticsearch.js
function OutputElasticSearch() { abstract_http.AbstractHttp.call(this); this.mergeConfig({ name: 'Elastic search', optional_params: ['bulk_limit', 'bulk_timeout', 'index_prefix', 'data_type', 'index_name'], default_values: { 'index_prefix': 'logstash', 'data_type': 'logs', 'bulk': 'true', /* UPDARTED FROM 'false' TO 'true' */ 'bulk_limit': '500', /* ADDED THIS LINE */ 'bulk_timeout': '1000' }, start_hook: this.start, }); }
By doing that, I get rid of ElasticSearch error 429 (too many requests)
BUT, I got another error 400 (bad request), after Investigation, I find that it happens when the batch contain one request! So, after reading some docs related to ElasticSearch bulk API, I added a new line in the request body to resolve this issue: FILE: lib/outputs/abstract_http.js
AbstractHttp.prototype.sendHttpRequest = function(http_options, body) { body = body + '\n'; /* UPDATED THIS LINE BY ADDING ( + '\n') */ if (this.proxying_agent) { http_options.agent = this.proxying_agent; } if (this.path_prefix) { http_options.path = this.path_prefix + http_options.path; }
Please let me know if there is a proper way to do that, thanks!