Add Support for Elasticsearch 7

[gangsterrapper22]

I have installed Statusengine on CentOS 8 and I tried to exports the Metrics to Elasticsearch 7. I have read that only Elasticsearch 6 is supported but I tried it anyway, if it maybe will work, but this wasn't the case. I get this Output when it tries to export to Elasticsearch 7 "{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Setting index.mapper.dynamic was removed after version 6.0.0"}],"type":"illegal_argument_exception","reason":"Setting index.mapper.dynamic was removed after version 6.0.0"},"status":400}"

Is there anything planned to add Elasticsearch 7 Support or maybe you can fix it? Please let me know, if you need more information from me.

I use the Version 3.6.0

Thank you.

Edit: It seems that the Data is get written to Elasticsearch, also with this Errror.

[nook24]

Have you installed the elasticsearch 7.0 driver for PHP?

apt-get install php-curl php-json php-mbstring
cd /opt/statusengine/worker
composer require elasticsearch/elasticsearch:~7.0 

Could you try to remove the following two lines and check if the issue is gone: https://github.com/statusengine/worker/blob/01ab62614c03b1ad5793989c5284c0f4db6b6c7f/src/Backends/Elasticsearch/Template.php#L61 https://github.com/statusengine/worker/blob/01ab62614c03b1ad5793989c5284c0f4db6b6c7f/lib/Elasticsearch/template.json#L9

[gangsterrapper22]

Edit: Maybe I have fixed the issue. I will do a reinstall of the System, to see if this has been really fixed, or if it was just luck. When I get it working again, then I will post the solution.

https://github.com/statusengine/worker/blob/01ab62614c03b1ad5793989c5284c0f4db6b6c7f/lib/Elasticsearch/template.json#L9

I have installed the Elasticsearch 7.0 Driver for PHP and removed also the Lines "mapper.dynamic": false. Then I get this Error Message. I couldn't figure it out yet what has to be exaclty changed in the Template, that it is Elasticsearch 7 compatible.

{"error":{"root_cause":[{"type":"mapper_parsing_exception","reason":"Root mapping definition has unsupported parameters: [_default_ : {_source={enabled=true}, _all={enabled=false}}] [metric : {properties={hostname={type=keyword}, service_description={type=keyword}, @timestamp={type=date}, metric={type=keyword}, value={index=false, type=double}}}]"}],"type":"mapper_parsing_exception","reason":"Failed to parse mapping [_doc]: Root mapping definition has unsupported parameters: [_default_ : {_source={enabled=true}, _all={enabled=false}}] [metric : {properties={hostname={type=keyword}, service_description={type=keyword}, @timestamp={type=date}, metric={type=keyword}, value={index=false, type=double}}}]","caused_by":{"type":"mapper_parsing_exception","reason":"Root mapping definition has unsupported parameters: [_default_ : {_source={enabled=true}, _all={enabled=false}}] [metric : {properties={hostname={type=keyword}, service_description={type=keyword}, @timestamp={type=date}, metric={type=keyword}, value={index=false, type=double}}}]"}},"status":400}

[gangsterrapper22]

I will write a new commend, because this will get then confusing. I did now now a complete reinstall of Centos 8, Naemon and Statusengine. The only thing which I have now modified was removing '_type' => 'metric', and then it has worked. https://github.com/statusengine/worker/blob/fb69e99f240b383db5c0bbaaba8599d86c5aba90/src/Backends/PerfdataBackends/ElasticsearchPerfdata.php#L96

This is the Output on Elasticsearch, when the Index was created: [statusengine-metric-2020.04] creating index, cause [auto(bulk api)], templates [statusengine-metric], shards [2]/[0], mappings [_doc]

If OK, I can create a Pull Request or you can fix it, but I don't know if this will break older Elasticsearch Installations.

[nook24]

Thanks for your investigation. According to the docs the _type field got deprecated in 6.0.0 and removed in 7.0.0

Initially, we spoke about an “index” being similar to a “database” in an SQL database, and a “type” being equivalent to a “table”.

This was a bad analogy that led to incorrect assumptions...

I guess due to the fact that Statusengine always creates a own index to store performance date, it never it never makes use of the _type field.

If OK, I can create a Pull Request or you can fix it, but I don't know if this will break older Elasticsearch Installations.

Yes of course. I'm not an expert when it comes to ES but I think it should not break older versions of elasticsearch. I guess it could only break someone's queries?

[gangsterrapper22]

I will try to make a Pull Request, but I am not a expert to do this. I think I will do this tomorrow or on the Weekend. On older ES Versions I can't tell you, if the queries can get broken or not, because I started with Version 7. In Grafana I use Lucene to do the Query for ES.

In the Statusengine UI I have seen, that the Performance Graph, display also no Graphs. I don't know, if this is also caused by this or if this another issue, or is there maybe a File which has the query in it for ES?

In the Statusengine-UI I have set: display_perfdata: 1 perfdata_backend: elasticsearch elasticsearch_pattern: monthly elasticsearch_address: IP of ES Server elasticsearch_port: 9200

Thanks.

Edit: I have also fixed the issue that the Performancegraphs are not showing any Data in the Webinterface for ES 7. In https://github.com/statusengine/interface/blob/9da0070026b553082ee10f0b12384106c00bbf7f/src/Loader/Elasticsearch/ServicePerfdataLoader.php#L115

I have also removed the type' => 'metric', and then it was working.

[nook24]

Thanks for your feedback. If i find some time over the weekend i will try this with older ES versions and patch the code.

[duylong]

Hi,

Here is my template for working with ES7:

{
  "template": "statusengine-metric-*",
  "order": 0,
  "settings": {
    "index": {
      "number_of_shards": 1,
      "refresh_interval": "60s",
      "codec": "best_compression"
    }
  },
  "mappings": {
    "_source": {
      "enabled": "true"
    },
    "properties": {
      "@timestamp": {
        "type": "date"
      },
      "value": {
        "type": "double",
        "index": "false"
      },
      "hostname": {
        "type": "string",
        "index": "not_analyzed"
      },
      "service_description": {
        "type": "string",
        "index": "not_analyzed"
      },
      "metric": {
        "type": "string",
        "index": "not_analyzed"
      }
    }
  }
}

Files modified in order to adapt with new template:

• worker/src/Backends/Elasticsearch/Template.php
• worker/lib/Elasticsearch/template.json

Others files:

• worker/etc/config.yml (remove enable_all)
• worker/src/Backends/PerfdataBackends/ElasticsearchPerfdata.php (remove "'_type' => 'metric'")
• interface/src/Loader/Elasticsearch/ServicePerfdataLoader.php (remove "'_type' => 'metric'")

[duylong]

I can still load my olds data (created in ES6) ;)

[nook24]

I added ES7 support in the ES7 branch. Would be great if you guys could test this. :)

I will also patch the master branch of https://github.com/statusengine/interface

[duylong]

Everything seems correct to me :)

[jairoko]

hi, i am working in statusEngine with ElasticSearch 7.10 and composer require elasticsearch/elasticsearch:~7.0. I need to change the field @timestamp, because is long (unix time) and i need a date format (yyyy-MM-dd)

¿How can i change this type? I change template.json located in /opt/statusengine/worker/lib/Elasticsearch and the type is being long.

Furthermore, I do the steps indicated here whitout success. I need this format to do some graphics.

Attach the pictures with my configuration and problem to complete the description.

Thanks

Grafana image: ElasticSearch index: Error to create in kibana: StatusEngine UI: Format unix time in milliseconds:

My template json:

{ "template": "statusengine-metric-*", "order": 0, "settings": { "index": { "number_of_shards": 1, "refresh_interval": "60s", "codec": "best_compression" } }, "mappings": { "default": { "_all": { "enabled": "false" }, "_source": { "enabled": "true" } }, "metric": { "properties": { "@timestamp": { "type": "date" }, "value": { "type": "double", "index": "no" }, "hostname": { "type": "string", "index": "not_analyzed" }, "service_description": { "type": "string", "index": "not_analyzed" }, "metric": { "type": "string", "index": "not_analyzed" } } } }

[duylong]

"@timestamp" must be "type" => "date"

[nook24]

Sorry for my late response to this one. I will build the 3.7 Release over the weekend which also adds support for Elasticsearch 7. It's already available in the ES7 branch. Tbh, i don't know why this isn't released already :(

[nook24]

Version 3.7 is released. Please see the changelog for update instructions: https://github.com/statusengine/worker/releases/tag/3.7.0