bug: ElasticSearch logger not working with error Action/metadata line [1] contains an unknown parameter [_type]

[gaoxingliang]

Current Behavior

The plugin eleasticsearch-logger is not working with elasticseachr-8.x and apisix - 3.0.0 and in apisix it has below error log:

2023/01/10 07:51:51 [error] 48#48: *231989 [lua] batch-processor.lua:95: Batch Processor[elasticsearch-logger] failed to process entries: elasticsearch server returned status: 400,
 body: {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Action/metadata line [1] contains an unknown parameter [_type]"}],"type":"illegal_argument_exception",
"reason":"Action/metadata line [1] contains an unknown parameter [_type]"},"status":400}, context: ngx.timer, client: , server: 0.0.0.0:9080

the plugin conf is referenced to the doc url: https://apisix.apache.org/docs/apisix/plugins/elasticsearch-logger/ and my conf is:

{
  "auth": {
    "password": "changeme",
    "username": "elastic"
  },
  "batch_max_size": 1,
  "buffer_duration": 60,
  "disable": false,
  "endpoint_addr": "http://xxxx:9200",
  "field": {
    "index": "services",
    "type": "collector"
  },
  "inactive_timeout": 5,
  "max_retry_count": 10,
  "name": "elasticsearch-logger",
  "retry_delay": 1,
  "ssl_verify": false,
  "timeout": 60
}

I also tried with es 6.0 and same conf as above. it shows error:

{"error":{"root_cause":[{"type":"action_request_validation_exception","reason":"Validation Failed: 1: an id must be provided if version type or value are set;"}],"type":"action_request_validation_exception","reason":"Validation Failed: 1: an id must be provided if version type or value are set;"},"status":400}, context: ngx.timer

Updates:

It seems the _type is deprecated in es: https://www.elastic.co/guide/en/elasticsearch/reference/7.17/mapping-type-field.html#mapping-type-field After removed the type in the conf. It worked.

{
  "auth": {
    "password": "changeme",
    "username": "elastic"
  },
  "batch_max_size": 1,
  "buffer_duration": 60,
  "disable": false,
  "endpoint_addr": "http://xxxx:9200",
  "field": {
    "index": "services"
  },
  "inactive_timeout": 5,
  "max_retry_count": 10,
  "name": "elasticsearch-logger",
  "retry_delay": 1,
  "ssl_verify": false,
  "timeout": 60
}

Expected Behavior

The elasticsearch logger plugin should work with es 8.x version.

Error Logs

See above.

Steps to Reproduce

1. create a elk cluster using https://github.com/deviantony/docker-elk#bringing-up-the-stack
2. add the elasticsearch logger plugin
3. check the apisix logs.

Environment

• APISIX version 3.0.0
• Operating system: centos7

[An-DJ]

It is a break change introduced by es 8.0+ due to removal of mapping types.

Ref: https://www.elastic.co/guide/en/elasticsearch/reference/8.0/release-highlights.html#_7_x_rest_api_compatibility

The plugin eleasticsearch-logger is tested based on es 7.17.1. For now it may only support the version 7.x and the earlier.

@gaoxingliang Would you like to fix that issue?

[An-DJ]

We should ensure that the plugin supports all versions of es as much as possible.

If version 8.x of the elasticsearch REST API is not compatible with earlier versions (6.0+), 7.x compatibility headers maybe a better solution.

CC @spacewander

[gaoxingliang]

It is a break change introduced by es 8.0+ due to removal of mapping types.

Ref: https://www.elastic.co/guide/en/elasticsearch/reference/8.0/release-highlights.html#_7_x_rest_api_compatibility

The plugin eleasticsearch-logger is tested based on es 7.17.1. For now it may only support the version 7.x and the earlier.

@gaoxingliang Would you like to fix that issue?

Sorry. I don't know about the lua scripts.

[spacewander]

@An-DJ Would you give it a try?

[Revolyssup]

@An-DJ Are you planning to work on this?

[pottekkat]

I was using this plugin and ran into this issue. It works with the 7.17 version of Elasticsearch. If this is not being worked on now, can we mention this in the documentation? Add a note that says only 7.x is supported.

cc: @Revolyssup

[lakshya8066]

I can look into this @shreemaan-abhishek can you assign me?

[lakshya8066]

@gaoxingliang @An-DJ @pottekkat I am trying to implement the REST API compatibility headers. Should we also change the elasticsearch used for testing to 8.x ? Or would it be better to run tests on both the versions?

[Vacant2333]

@gaoxingliang @An-DJ @pottekkat I am trying to implement the REST API compatibility headers. Should we also change the elasticsearch used for testing to 8.x ? Or would it be better to run tests on both the versions?

if we can run on the 2 version may better , how do u think ? @shreemaan-abhishek

[pottekkat]

@lakshya8066 I think we can only run tests on 8.x. It does not seem to me that Elasticsearch is maintaining multiple versions.

[lakshya8066]

Okay, I think we can also implement 7.x compatibility headers and then test the plugin on version 8.x. This will prevent error logs in case someone uses version 7.x of elasticsearch.

[pottekkat]

Nice catch. Sounds like a good solution.

[lakshya8066]

Sure, will work on a PR for this.

[membphis]

Okay, I think we can also implement 7.x compatibility headers and then test the plugin on version 8.x. This will prevent error logs in case someone uses version 7.x of elasticsearch.

agree with this way, it LGTM

[lakshya8066]

@pottekkat I am not sure why the CI is not passing for me. As I understand the, the failing test cases do not contain field.type parameter in the config, so it should be working fine on version 8 as well.

[pottekkat]

@lakshya8066 Let me rerun the CI and check.

[lakshya8066]

@pottekkat any update on this?