search

uken / fluent-plugin-elasticsearch

Apache License 2.0
891 stars 310 forks source link

Getting error while passing multilevel JSON to ES #902

Closed fit-dhairya closed 3 years ago

fit-dhairya commented 3 years ago

Problem

Hello, I'm facing an issue while passing multilevel JSON log to ES .

Steps to replicate

I'm using Winston for logging and I can log using below syntax,

logger.info({message:'this is a message', data: "this is data"});

This log can be passed to ES without any error and viewable inside Kibana. But when I pass multilevel JSON, it causes error. 

//Log syntax
logger.info({message:'this is a message', data: {function: "adminDashboard", file: "dashboard"}});

// Error in Kibana
{
  "_index": "fluentd-20210623",
  "_type": "_doc",
  "_id": "-JZTOHoBZUsthc_svLUa",
  "_score": 1,
  "fields": {
    "record.data.file.keyword": [
      "dashboard"
    ],
    "error.keyword": [
      "#<Fluent::Plugin::ElasticsearchErrorHandler::ElasticsearchError: 400 - Rejected by Elasticsearch>"
    ],
    "record.message": [
      "this is a message"
    ],
    "record.level.keyword": [
      "info"
    ],
    "error": [
      "#<Fluent::Plugin::ElasticsearchErrorHandler::ElasticsearchError: 400 - Rejected by Elasticsearch>"
    ],
    "message": [
      "dump an error event: error_class=Fluent::Plugin::ElasticsearchErrorHandler::ElasticsearchError error=\"400 - Rejected by Elasticsearch\" location=nil tag=\"test\" time=1624442647 record={\"message\"=>\"this is a message\", \"data\"=>{\"function\"=>\"adminDashboard\", \"file\"=>\"dashboard\"}, \"level\"=>\"info\", \"module\"=>\"dashboard_logs\", \"timestamp\"=>\"2021-06-23T10:04:07.262Z\"}"
    ],
    "@log_name.keyword": [
      "fluent.warn"
    ],
    "record.data.function": [
      "adminDashboard"
    ],
    "@timestamp": [
      "2021-06-23T10:04:08.623Z"
    ],
    "record.data.function.keyword": [
      "adminDashboard"
    ],
    "record.module": [
      "dashboard_logs"
    ],
    "record.level": [
      "info"
    ],
    "@log_name": [
      "fluent.warn"
    ],
    "tag": [
      "test"
    ],
    "time": [
      1624442647
    ],
    "record.module.keyword": [
      "dashboard_logs"
    ],
    "record.message.keyword": [
      "this is a message"
    ],
    "tag.keyword": [
      "test"
    ],
    "record.timestamp": [
      "2021-06-23T10:04:07.262Z"
    ],
    "record.data.file": [
      "dashboard"
    ]
  }
}

Using Fluentd and ES plugin versions

fit-dhairya commented 3 years ago

Issue solved, whenever we change the content type from string to complex object, we need to clear the current index, so that next time ES will take the field as complex object and not as string.