sematext / logsene-js

Log shipping library for sending logs to Sematext from JavaScript apps
https://sematext.com/cloud
Apache License 2.0
10 stars 9 forks source link

Elasticsearch 8: Action/metadata line [1] contains an unknown parameter [_type] #42

Open am-zq opened 1 year ago

am-zq commented 1 year ago

I'm trying to use logagent with a newly-installed Elastic 8.5.3 and getting the following error:

logagent_1  | 2023-01-23T11:24:38.977Z v3.1.3 pid[7] Error in Elasticsearch request:  source="logsene-js" err={"message":"HTTP status code:400","httpStatus":400,"httpBody":"{\"error\":{\"root_cause\":[{\"type\":\"illegal_argument_exception\",\"reason\":\"Action/metadata line [1] contains an unknown parameter [_type]\"}],\"type\":\"illegal_argument_exception\",\"reason\":\"Action/metadata line [1] contains an unknown parameter [_type]\"},\"status\":400}","url":"http://192.168.0.10:9001/\"docker\"/_bulk"} /  message="HTTP status code:400" httpStatus=400 httpBody="{\"error\":{\"root_cause\":[{\"type\":\"illegal_argument_exception\",\"reason\":\"Action/metadata line [1] contains an unknown parameter [_type]\"}],\"type\":\"illegal_argument_exception\",\"reason\":\"Action/metadata line [1] contains an unknown parameter [_type]\"},\"status\":400}" url="http://192.168.0.10:9001/\"docker\"/_bulk"

The error indicates source logsene-js, hence posting the issue here rather than in logagent-js.

The "_type" parameter seems to be used at: https://github.com/sematext/logsene-js/blob/master/index.js#L419

  if (elasticsearchDocId !== null) {
    this.bulkReq.write(stringifySafe({ 'index': { '_index': _index, '_id': String(elasticsearchDocId), '_type': type || this.type } }) + '\n')
  } else {
    this.bulkReq.write(stringifySafe({ 'index': { '_index': _index, '_type': type || this.type } }) + '\n')
  }

The "_type" parameter was deprecated for bulk operations as of Elasticsearch 8.0: https://www.elastic.co/guide/en/elasticsearch/reference/7.17/removal-of-types.html

If my diagnosis is correct, could this be updated to be compatible with Elasticsearch 8.0+?

Environment Running latest docker image: docker pull sematext/logagent

My docker-compose.yml:

logagent:
  image: 'sematext/logagent:latest'
  environment:
    - LOGS_TOKEN="docker"
    - LOGS_RECEIVER_URL="http://192.168.0.10:9001"
  cap_add:
    - SYS_ADMIN
  restart: always
  volumes:
    - '/var/run/docker.sock:/var/run/docker.sock'