search

serilog-contrib / serilog-sinks-splunk

A Serilog sink that writes to Splunk
https://splunk.com
Apache License 2.0
46 stars 47 forks source link

Feature.customfields #59

Closed patriklindstrom closed 7 years ago

patriklindstrom commented 7 years ago

Add CustomField splunk feature in serilog-sink log constructor

Splunk has a feature for adding customfields that you want to be associated with the event and can be used in filters but not shown as part of the event. See http://dev.splunk.com/view/event-collector/SP-CAAAFB6 .

These index for CustomFields can be used in splunk filters like eg: role::backend role::rest channel::verification .

These CustomFields can be string key value pairs or key array of values. This feature (afaik) was missing in the current splunk sink.

I have made these two new classes: CustomField and CustomFieldsList also added constructor that takes fields as parameter and logic for adding this to _suffix

I added test to SplunkJsonFormatterTests. However I am not that good with Xunit test soo also I added project with NUnit test in style of Arrange,Act, Assert style. I think these sort of filter capabilites is great at larger organisations that will make an abstract log layer around serilog and set these fields at deploy time with a config file and a tool like Octopus Deply or TFS Release Manager. That is nothing that the developer will care about. They will just focus on the what will be in de event. I think better big corporate Splunk Dasboard can be build with this feature where you want eg to search for all event for the last release on all dotnet core rest backendservices deployed to Linux machines to the Acceptance Test Channel.

patriklindstrom commented 7 years ago

Hi Matthew Thanks for feedback Ill start fixing issues you have mentioned. And Ill move my test test to Xunit.

I will probably be done after this weekend.

Cheers Patrik

2017-06-14 10:30 GMT+02:00 Matthew Erbs notifications@github.com:

@merbla requested changes on this pull request.

Hi @patriklindstrom https://github.com/patriklindstrom, I think this is a great addition. Thanks so much for taking the time to create a PR!

A have noted a few items, also we are eager to stay with XUnit due to DotNet core concerns and consistency across the Serilog Organisation.

In Notes.md.saved.bak https://github.com/serilog/serilog-sinks-splunk/pull/59#discussion_r121882391 :

@@ -0,0 +1,7 @@ +Getting error.

Nit: Remove this file

In Notes.md https://github.com/serilog/serilog-sinks-splunk/pull/59#discussion_r121882412 :

@@ -0,0 +1,4 @@ +Getting error.

Nit: Remove this file

In result.json https://github.com/serilog/serilog-sinks-splunk/pull/59#discussion_r121882561 :

@@ -0,0 +1,19 @@ +{

Is this an example event? Not sure of its purpose

In sample/Sample/Program.cs https://github.com/serilog/serilog-sinks-splunk/pull/59#discussion_r121882834 :

using Serilog; using Serilog.Sinks.Splunk;

namespace Sample { public class Program {

@patriklindstrom https://github.com/patriklindstrom FYI your VM info is here. Would the Sample work better with some args? e.g. HEC token and host?

In src/Serilog.Sinks.Splunk/Sinks/Splunk/SplunkJsonFormatter.cs https://github.com/serilog/serilog-sinks-splunk/pull/59#discussion_r121883404 :

  • ///
  • /// Supplies culture-specific formatting information, or null.
  • /// If true, the template used will be rendered and written to the output as a property named MessageTemplate
  • /// The Splunk index to log to
  • /// The source of the event
  • /// The source type of the event
  • /// The host of the event
  • /// Object that describes extra splunk fields that should be indexed with event see: http://dev.splunk.com/view/event-collector/SP-CAAAFB6
  • public SplunkJsonFormatter(
  • bool renderTemplate,
  • IFormatProvider formatProvider,
  • string source,
  • string sourceType,
  • string host,
  • string index,
  • CustomFields customFields)

Should CustomFields customFields be an optional param. That seems to represent the HEC implementation.

In src/Serilog.Sinks.Splunk/SplunkLoggingConfigurationExtensions.cs https://github.com/serilog/serilog-sinks-splunk/pull/59#discussion_r121883429 :

@@ -49,7 +49,7 @@ public static class SplunkLoggingConfigurationExtensions /// The minimum log event level required in order to write an event to the sink. /// The output template to be used when logging /// Supplies culture-specific formatting information, or null.

  • /// If ture, the message template will be rendered
  • /// If true, the message template will be rendered

Nice pickup!

In test/Serilog.Sinks.Splunk.CustomFieldsTests/Serilog.Sinks.Splunk. CustomFieldsTests/SplunkCustomFieldsTests.cs https://github.com/serilog/serilog-sinks-splunk/pull/59#discussion_r121884773 :

  • public string Version { get; set; }
  • public string Rel { get; set; }
  • public List Role { get; set; }
  • }
  • public class TestEventResultObject
  • {
  • public string Time { get; set; }
  • public Event @Event { get; set; }
  • public string Source { get; set; }
  • public string Sourcetype { get; set; }
  • public string Host { get; set; }
  • public string Index { get; set; }
  • public TestCustomFields Fields { get; set; }
  • }
  • [TestFixture]
  • class SplunkCustomFieldsTests

There area some great use cases here, can we move to the existing test project. We are keen to stay with XUnit due to a number of issues with NUnit & DotNet core.

As for the Arrange/Act/Assert. We should be able to achieve in XUnit

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/serilog/serilog-sinks-splunk/pull/59#pullrequestreview-43945244, or mute the thread https://github.com/notifications/unsubscribe-auth/ABv8Vhn-7pm62JhA0GJVT9MnM2aZ6Dg6ks5sD5ofgaJpZM4N5ACf .

--

=========================================================== I welcome VSRE emails. Learn more at http://vsre.info/

patriklindstrom commented 7 years ago

Hi @merbla
Have I fixed everything that you wanted now? I am a bit new to the github pullrequest. I belive only issue left is about wether the fields parameter is optional or not.

merbla commented 7 years ago

@patriklindstrom a dev package should be available for this feature. Serilog.Sinks.Splunk.2.3.0-dev-00175.nupkg. https://www.nuget.org/packages/Serilog.Sinks.Splunk/2.3.0-dev-00176

It will go out with #61

patriklindstrom commented 7 years ago

Great thanks

Skickat från min iPhone

17 juni 2017 kl. 00:27 skrev Matthew Erbs notifications@github.com:

@patriklindstrom a dev package should be available for this feature. Serilog.Sinks.Splunk.2.3.0-dev-00175.nupkg. https://www.nuget.org/packages/Serilog.Sinks.Splunk/2.3.0-dev-00176

It will go out with #61

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

patriklindstrom commented 7 years ago

@merbla Is it ok if I update the wiki with CustomFields examples? https://github.com/serilog/serilog-sinks-splunk/wiki/Configuring-the-sink ?

merbla commented 7 years ago

Yep! I have not had a chance to test the dev package.

merbla commented 7 years ago

Let me know if you do not have access.

dhaval-mistry commented 3 years ago

I have tried to add customfield into splunk using below code. But I am failed. 

           //convert RPA JSON log into object 
         var SplunkLog = ConvertJsonToObject(LogJson);
        // add custom field
        var metaData = new CustomFields(new List<CustomField>
        {
            new CustomField("logger", SplunkLog.logger),
            new CustomField("timestamp",  SplunkLog.timestamp.ToString())
        });

        Log.Logger = new LoggerConfiguration()
            .MinimumLevel.Debug()
            .WriteTo.Console()
            .WriteTo.EventCollector(
                splunkHost: SPLUNK_ENDPOINT
                , eventCollectorToken: SPLUNK_HEC_TOKEN
                , fields: metaData  //// adding customField into logger object.
                , host: "RPA BOT Runner"
                , source: "BOT"
                , sourceType: "BOT")
            //SPLUNK_FULL_ENDPOINT,EventCollectorToken)
            .Enrich.WithProperty("CorrelationId", SplunkLog.BOTProperties.CorrelationId)
            .Enrich.WithProperty("BusinessUnit", SplunkLog.BOTProperties.BusinessUnit)
            .Enrich.WithProperty("ApplicationName", SplunkLog.BOTProperties.ApplicationName)
            .Enrich.WithProperty("ApplicationFeatureName", SplunkLog.BOTProperties.ApplicationFeatureName)
            .Enrich.WithProperty("EnvironmentName", SplunkLog.BOTProperties.EnvironmentName)
            .Enrich.WithProperty("BOTMachine", SplunkLog.BOTProperties.BOTMachine)
            .Enrich.WithProperty("CallType", SplunkLog.BOTProperties.CallType)
            .CreateLogger();

        //Log.Information("Running vanilla loop with full uri {Counter}", i);
        Log.Information("logger: {logger} timestamp: {timestamp}", SplunkLog.logger, SplunkLog.timestamp.ToString() );

I could able to create log in splunk but custom fields are not visible in splunk. below is my Splunk JSON we design.

{ "severity": "INFO", "logger": "RPA", "timestamp": "09/17/2021", "RPAJobName": "<>", "BOTProperties": { "CorrelationId": "<>", "BusinessUnit": "<>", "ApplicationName": "<>", "ApplicationFeatureName": "<>", "EnvironmentName": "<>", "BOTMachine": "<<BOT Machine Name/IP>>", "CallType": "BOT", "RequestBody": [ { "companyId": "gtstest" }, { "userName": "qaestatement" }, { "RquestData1": "RquestData1" }, { "RquestData2": "RquestData2" } ], "BOT Step": "<< BOT Step Detail >>", "Status": "PASSED", "ResponseBody": [ { "companyId": "gtstest" }, { "userName": "qaestatement" }, { "executionStartTime": "09/16/2021 16:25:36.466" }, { "executionEndTime": "09/16/2021 16:30:27.312" }, { "Response Sample": "<< Response Sample >>" }, { "Response Sample 2": "<< Response Sample 2 >>" } ] } }