Cant send data to elasticsearch 5

[paveldpua]

Hi, I have installed stagemonitos as a widget and it can not send data to ES

logs: 16-12-12 09:58:59.357 {WARN} [elasticsearch-tasks] org.stagemonitor.core.util.HttpClient$ErrorLoggingResponseHandler.handleResponse() @169 https://user:password@dev-elk-stack.simplex.tv/elasticsearch/stagemonitor-requests-*,-stagemonitor-requests-2016.12.12,-stagemonitor-requests-2016.12.11,-stagemonitor-requests-2016.12.10,-stagemonitor-requests-2016.12.09,-stagemonitor-requests-2016.12.08,-stagemonitor-requests-2016.12.07,-stagemonitor-requests-2016.12.06,-stagemonitor-requests-2016.12.05?timeout=20m&ignore_unavailable=true: 404 {"statusCode":404,"error":"Not Found"} 16-12-12 09:58:59.358 {WARN} [async-elasticsearch] org.stagemonitor.core.util.HttpClient$ErrorLoggingResponseHandler.handleResponse() @169 https://user:password@dev-elk-stack.simplex.tv/elasticsearch/stagemonitor: 404 {"statusCode":404,"error":"Not Found"} 16-12-12 09:58:59.410 {WARN} [elasticsearch-tasks] org.stagemonitor.core.util.HttpClient$ErrorLoggingResponseHandler.handleResponse() @169 https://user:password@dev-elk-stack.simplex.tv/elasticsearch/stagemonitor-external-requests-*,-stagemonitor-external-requests-2016.12.12,-stagemonitor-external-requests-2016.12.11,-stagemonitor-external-requests-2016.12.10,-stagemonitor-external-requests-2016.12.09,-stagemonitor-external-requests-2016.12.08,-stagemonitor-external-requests-2016.12.07,-stagemonitor-external-requests-2016.12.06,-stagemonitor-external-requests-2016.12.05?timeout=20m&ignore_unavailable=true: 404 {"statusCode":404,"error":"Not Found"} 16-12-12 09:58:59.419 {WARN} [async-elasticsearch] org.stagemonitor.core.elasticsearch.ElasticsearchClient$BulkErrorReportingResponseHandler.handleResponse() @350 Error(s) while sending a _bulk request to elasticsearch: {"statusCode":404,"error":"Not Found"} 16-12-12 09:58:59.468 {WARN} [async-elasticsearch] org.stagemonitor.core.util.HttpClient$ErrorLoggingResponseHandler.handleResponse() @169 https://user:password@dev-elk-stack.simplex.tv/elasticsearch/_template/stagemonitor-metrics: 404 {"statusCode":404,"error":"Not Found"} 16-12-12 09:58:59.512 {WARN} [async-elasticsearch] org.stagemonitor.core.elasticsearch.ElasticsearchClient$BulkErrorReportingResponseHandler.handleResponse() @350 Error(s) while sending a _bulk request to elasticsearch: {"statusCode":404,"error":"Not Found"} 16-12-12 09:58:59.555 {WARN} [async-elasticsearch] org.stagemonitor.core.elasticsearch.ElasticsearchClient$BulkErrorReportingResponseHandler.handleResponse() @350 Error(s) while sending a _bulk request to elasticsearch: {"statusCode":404,"error":"Not Found"} 16-12-12 09:58:59.603 {WARN} [async-elasticsearch] org.stagemonitor.core.util.HttpClient$ErrorLoggingResponseHandler.handleResponse() @169 https://user:password@dev-elk-stack.simplex.tv/elasticsearch/_template/stagemonitor-requests: 404 {"statusCode":404,"error":"Not Found"}

[felixbarny]

That might be because stagemonitor does not yet support basic authentication. There is already a pull request for that: https://github.com/stagemonitor/stagemonitor/pull/223

[paveldpua]

Thank you very much with quick reply. So can i get this changes to make stagemonitor working with ES with authentication?

[felixbarny]

You can check out @tetha's branch and build stagemonitor yourself to see if it's working for you: https://github.com/Tetha/stagemonitor/tree/basic-authentication#build-locally

Or you can wait until the change is officially released. It will be released soon after @tetha has implemented my remark.

[paveldpua]

OK, Thank you very much. Going to try basic-authentication branch

[paveldpua]

I have built new stagemonitor-core-0.31.0-SNAPSHOT.jar and replaced it in WEB-INF/lib dir. Restarted my APP, but have same errors... BTW, i use JDK 1.7. can it be the issue?

[paveldpua]

I have installed JDK 1.8 but nothing changed....

[paveldpua]

curl https://user:pass@dev-elk-stack.simplex.tv/elasticsearch/_template/stagemonitor-metrics?pretty { "stagemonitor-metrics" : { "order" : 0, "template" : "stagemonitor-metrics-", "settings" : { "index" : { "refresh_interval" : "5s" } }, "mappings" : { "default" : { "_source" : { "enabled" : false }, "dynamic_templates" : [ { "strings" : { "mapping" : { "index" : "not_analyzed", "type" : "string", "doc_values" : true }, "match_mapping_type" : "string", "match" : "" } } ], "_all" : { "enabled" : false }, "properties" : { "mean_rate" : { "index" : "no", "type" : "float", "doc_values" : true }, "measurement_start" : { "type" : "date", "doc_values" : true }, "std" : { "index" : "no", "type" : "float", "doc_values" : true }, "max" : { "index" : "no", "type" : "integer", "doc_values" : true }, "count" : { "index" : "no", "type" : "integer", "doc_values" : true }, "p95" : { "index" : "no", "type" : "float", "doc_values" : true }, "p98" : { "index" : "no", "type" : "float", "doc_values" : true }, "m1_rate" : { "index" : "no", "type" : "float", "doc_values" : true }, "p75" : { "index" : "no", "type" : "float", "doc_values" : true }, "p99" : { "index" : "no", "type" : "float", "doc_values" : true }, "value_boolean" : { "index" : "no", "type" : "boolean", "doc_values" : true }, "p25" : { "index" : "no", "type" : "float", "doc_values" : true }, "@timestamp" : { "type" : "date", "doc_values" : true }, "min" : { "index" : "no", "type" : "float", "doc_values" : true }, "median" : { "index" : "no", "type" : "float", "doc_values" : true }, "mean" : { "index" : "no", "type" : "integer", "doc_values" : true }, "value_string" : { "index" : "no", "type" : "string", "doc_values" : true }, "p999" : { "index" : "no", "type" : "float", "doc_values" : true }, "m15_rate" : { "index" : "no", "type" : "float", "doc_values" : true }, "value" : { "index" : "no", "type" : "float", "doc_values" : true }, "m5_rate" : { "index" : "no", "type" : "float", "doc_values" : true } } } }, "aliases" : { } } }

[felixbarny]

The response {"statusCode":404,"error":"Not Found"} indicates that stagemonitor could reach elasticsearch but something else went wrong.

Could you try

curl -XPOST  https://user:pass@dev-elk-stack.simplex.tv/elasticsearch/_bulk -d '{ "index" : { "_index" : "test", "_type" : "type1", "_id" : "1" } }
{ "field1" : "value1" }
'
curl localhost:9200/test/type1/1 

Please post the output.

If that also fails, there might be something wrong in you proxy configuration.

[paveldpua]

curl -XPOST https://user:pass@dev-elk-stack.simplex.tv/elasticsearch/_bulk -d '{ "index" : { "_index" : "test", "_type" : "type1", "_id" : "1" } }

{ "field1" : "value1" } ' {"statusCode":404,"error":"Not Found"}

[paveldpua]

cat requests { "index" : { "_index" : "test", "_type" : "type1", "_id" : "1" } } { "field1" : "value1" } { "delete" : { "_index" : "test", "_type" : "type1", "_id" : "2" } } { "create" : { "_index" : "test", "_type" : "type1", "_id" : "3" } } { "field1" : "value3" } { "update" : {"_id" : "1", "_type" : "type1", "_index" : "test"} } { "doc" : {"field2" : "value2"} }

curl -s -XPOST user:pass@localhost:9200/_bulk?pretty --data-binary "@requests"; echo { "took" : 117, "errors" : true, "items" : [ { "index" : { "_index" : "test", "_type" : "type1", "_id" : "1", "_version" : 6, "result" : "updated", "_shards" : { "total" : 2, "successful" : 1, "failed" : 0 }, "created" : false, "status" : 200 } }, { "delete" : { "found" : false, "_index" : "test", "_type" : "type1", "_id" : "2", "_version" : 1, "result" : "not_found", "_shards" : { "total" : 2, "successful" : 1, "failed" : 0 }, "status" : 404 } }, { "create" : { "_index" : "test", "_type" : "type1", "_id" : "3", "status" : 409, "error" : { "type" : "version_conflict_engine_exception", "reason" : "[type1][3]: version conflict, document already exists (current version [1])", "index_uuid" : "lShHoeEVS7GeVXQslEHKBA", "shard" : "4", "index" : "test" } } }, { "update" : { "_index" : "test", "_type" : "type1", "_id" : "1", "_version" : 7, "result" : "updated", "_shards" : { "total" : 2, "successful" : 1, "failed" : 0 }, "status" : 200 } } ] }

[paveldpua]

you are quite right. It seems issue is with my proxy... I still dont know exactly where but I use gninx and kibana to access it as it can be seen from link https://user:pass@dev-elk-stack.simplex.tv/elasticsearch/_bulk (even with https) When i have opened port 9200 directly - it seem working... Thank you for your time. I will try to locate issue in nginx.