markmcdowell / NLog.Targets.ElasticSearch

NLog target for Elasticsearch
MIT License
176 stars 89 forks source link

After adding mapping template no docs indexed #123

Closed Rassi closed 4 years ago

Rassi commented 4 years ago

I attempted to fix mapping conflicts by adding a mapping template, but after doing that no logs were added to new indexes.

I got no exceptions, and the internal nlog log file didn't show anything. After getting the NLog.Targets.ElasticSearch code, I could finally set a breakpoint in ElasticSearchTarget.SendBatch and saw the following at https://github.com/markmcdowell/NLog.Targets.ElasticSearch/blob/master/src/NLog.Targets.ElasticSearch/ElasticSearchTarget.cs#L290.

result.DebugInformation:

Successful (200) low level call on POST: /_bulk
# Server indicated deprecations:
- 299 Elasticsearch-7.5.0-e9ccaed468e2fac2275a3761849cbee64b39519f "[types removal] Specifying types in bulk requests is deprecated."
# Audit trail of this API call:
 - [1] PingSuccess: Node: https://www.example.com:9243/ Took: 00:00:00.2835193
 - [2] HealthyResponse: Node: https://www.example.com:9243/ Took: 00:00:00.4522220
# Request:
<Request stream not captured or already read to completion by serializer. Set DisableDirectStreaming() on ConnectionSettings to force it to be set on the response.>
# Response:
{"took":2,"errors":true,"items":[{"index":{"_index":"testindex-03","_type":"logevent","_id":"JVDaA3IBc9a3A5n7JPnq","status":400,"error":{"type":"illegal_argument_exception","reason":"Rejecting mapping update to [testindex-03] as the final mapping would have more than 1 type: [_doc, logevent]"}}}]}

So it is caused by Elastic deprecating types, and me having specified documentType as logevent (don't ask me why. Must be copy/pasta from somewhere.)

nlog.config:

<nlog xmlns="http://nlog-project.org/schemas/NLog.xsd"
      xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
      throwExceptions="true"
      internalLogFile="c:\temp\nlog_internal_log_file.txt">
  <extensions>
    <add assembly="NLog.Targets.ElasticSearch"/>
  </extensions>

  <targets async="true">
    <target xsi:type="ElasticSearch"
            name="elastic"
            uri="${configsetting:name=Logging.ElasticUri}"
            index="testindex-03"
            documentType="logevent"
            includeAllProperties="True"
            requireAuth="True"
            username="${configsetting:name=Logging.ElasticUser}"
            password="${configsetting:name=Logging.ElasticPassword}">
      <field name="app" layoutType="System.Object">
        <layout xsi:type="JsonLayout" IncludeGdc="true" />
      </field>
      <field name="logger" layout="${logger}" />
      <field name="request" layoutType="System.Object">
        <layout xsi:type="JsonLayout" >
          <attribute name="client-ip" layout="${mdlc:client-ip}" />
          <attribute name="client-id" layout="${mdlc:client-id}" />
          <attribute name="client-name" layout="${mdlc:client-name}" />
          <attribute name="duration-ms" layout="${mdlc:duration-ms}" encode="false" />
          <attribute name="status-code" layout="${mdlc:status-code}" encode="false" />
          <attribute name="url" layout="${mdlc:url}" />
          <attribute name="request-id" layout="${mdlc:RequestId}" />
          <attribute name="request-path" layout="${mdlc:RequestPath}" />
        </layout>
      </field>
    </target>
  </targets>

  <rules>
    <logger name="*" minLevel="Trace" appendTo="elastic" />
  </rules>
</nlog>

So document type logevent conflicts with the default _doc when index/mapping is created from a template. I can see that logevent is default in https://github.com/markmcdowell/NLog.Targets.ElasticSearch/blob/master/src/NLog.Targets.ElasticSearch/ElasticSearchTarget.cs#L149, and when I tried removing documentType from my nlog.config it did indeed still fail with the same error.

So I fixed it by changing to the same document type (_doc), but I would still like to point out a couple of suggestions:

  1. Default document type to null or _doc (seems like others have set it to null https://issues.apache.org/jira/browse/KARAF-6539 and worked for me when I set ElasticSearchTarget.DocumentType to null runtime)
  2. Raise exception or log internal error when receiving these kinds of "nested" errors.
snakefoot commented 4 years ago

Created #124 to improve on this.