Unable to use API key authentication to connect to Elasticsearch

Currently we use Basic Authentication to create document logs in elastic which works fine with NLog.Targets.ElasticSearch v7.7.0. On switching to API Key authentication I keep getting Unauthorized error on authentication. Our ElasticSearch is on version 8.7.0.

Here is the exception stack trace Exception: Elasticsearch.Net.ElasticsearchClientException: Could not authenticate with the specified node. Try verifying your credentials or check your Shield configuration. Call: Status code 401 from: POST /_bulk ---> Elasticsearch.Net.PipelineException: Could not authenticate with the specified node. Try verifying your credentials or check your Shield configuration. ---> System.Net.WebException: The remote server returned an error: (401) Unauthorized. at System.Net.HttpWebRequest.GetResponse() at Elasticsearch.Net.HttpWebRequestConnection.Request[TResponse](RequestData requestData) --- End of inner exception stack trace --- at Elasticsearch.Net.RequestPipeline.ThrowBadAuthPipelineExceptionWhenNeeded(IApiCallDetails details, IElasticsearchResponse response) at Elasticsearch.Net.RequestPipeline.CallElasticsearch[TResponse](RequestData requestData) at Elasticsearch.Net.Transport1.Request[TResponse](HttpMethod method, String path, PostData data, IRequestParameters requestParameters)`

The API Key is working with curl request to fetch the document based on elastic uri and the index. I am passing the same API Key in the Authorization header. So permission shouldn't be the problem. Here is the API Key information and you can see privileges are set correctly.

{ "api_keys": [ { "username": "*****", "realm": "cloud-saml", "name": "*****", "creation": 1693470140150, "invalidated": false, "role_descriptors": { "appuser": { "applications": [], "transient_metadata": { "enabled": true }, "run_as": [], "cluster": [], "indices": [ { "privileges": [ "write", "create_index", "create_doc", "create", "all" ], "allow_restricted_indices": false, "names": [ "logs-<index1>-default-*", "logs-<index2>-*" ] } ], "metadata": {} } }, "id": "APIKEYID", "metadata": {} } ] }

Here is my NLog.Config

<?xml version="1.0" encoding="utf-8"?>

<nlog xmlns="http://www.nlog-project.org/schemas/NLog.xsd"
      xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
      xsi:schemaLocation="http://www.nlog-project.org/schemas/NLog.xsd NLog.xsd"
      autoReload="true"
      throwExceptions="false"
      internalLogLevel="Error" internalLogFile="${basedir}/logs/error.log">

    <variable name="defaultLayout" value="${longdate} | ${level:uppercase=true} | ${logger} | ${message}${onexception:${newline}${exception:format=tostring}}"/>
    <extensions>
        <add assembly="NLog.Web" />
        <add assembly="NLog.Targets.ElasticSearch" />
        <add assembly="Elastic.Apm.NLog"/>
        <add assembly="Elastic.CommonSchema.NLog"/>
    </extensions>
    <targets>
        <target name="elastic" xsi:type="BufferingWrapper" flushTimeout="5000">
            <target xsi:type="ElasticSearch"
                    name="elastic"
                    enableJsonLayout="true"
                    opCodeCreate="true"
                    includeDefaultFields="true"
                    includeAllProperties="true"
                    documentType="">
                <layout xsi:type="EcsLayout">
                    <metadata name="data_stream.dataset" layout="${appsetting:Log.DataStream.Dataset}" />
                    <metadata name="application.title" layout="${appsetting:Log.Application.Title}" />
                </layout>
            </target>
        </target>
    </targets>

    <rules>
        <logger name="*" minlevel="Trace" writeTo="elastic" />
    </rules>
</nlog>

And my helper class to initialize the ElasticSearchTarget

public static class ElasticSearchTargetSetup
    {
        public static bool SetupElasticTarget()
        {
            bool isElasticActive = false;

            var elasticTargetWrapper = LogManager.LogFactory.Configuration.AllTargets.SingleOrDefault(x => x is BufferingTargetWrapper) as BufferingTargetWrapper;

            if (elasticTargetWrapper != null && elasticTargetWrapper.WrappedTarget is ElasticSearchTarget)
            {
                var elasticTarget = elasticTargetWrapper.WrappedTarget as ElasticSearchTarget;
                elasticTarget.Index = ConfigurationManager.AppSettings["Log.Elastic.Index"];
                elasticTarget.CloudId = ConfigurationManager.AppSettings["Log.Elastic.CloudId"];
                elasticTarget.Uri = ConfigurationManager.AppSettings["Log.Elastic.Url"];
                elasticTarget.ApiKeyId = ConfigurationManager.AppSettings["Log.Elastic.ApiKeyId"];
                elasticTarget.ApiKey = ConfigurationManager.AppSettings["Log.Elastic.ApiKey"];
            }            

            return isElasticActive;
        }
    }

Here is my web.config to read appsettings

<appSettings>
    <add key="Log.Application.Title" value="My App 1" />
    <add key="Log.DataStream.Dataset" value="my data stream" />
    <add key="Log.Elastic.CloudId" value="my-cluster:<Base64string>" />
    <add key="Log.Elastic.Index" value="logs-<index1>-default" />
    <add key="Log.Elastic.Url" value="https://my-elastic-cloud:9243" />
    <add key="Log.Elastic.ApiKeyId" value="APIKEYID"/>
    <add key="Log.Elastic.ApiKey" value="APIKEY"/>
</appSettings>

Has anyone faced this issue or were able to use the API Key authentication successfully with NLog.Targets.ElasticSearch 7.7.0?

markmcdowell / NLog.Targets.ElasticSearch

Unable to use API key authentication to connect to Elasticsearch #178