lumberjack endpoint, rename @source.host to @shipper.host

[mrdavidlaing]

logstash/config/lumberjack_to_redis.conf.erb#L14 sets @source.host to %{host}

filter {
  if [_logstash_input] == "lumberjack" {
    mutate {
      add_field => [ "@source.host", "%{host}" ]
      add_field => [ "@source.path", "%{file}" ]
      add_field => [ "@source.offset", "%{offset}" ]
      remove_field => [ "host", "file", "offset", "_logstash_input" ]
    }
  }
}

%{host} is actually the name of the shipper, not the machine where the source log file comes from.

we should rather have add_field => [ "@shipper.host", "%{host}" ]

[mrdavidlaing]

@dpb587; thoughts? I'm trying to get @source to be about where the log file comes from, and @shipper to be about the thing that is shipping the log file.

[mrdavidlaing]

See https://github.com/logsearch/logsearch-shipper.NET/commit/e0eb1f3a9005c100d70e1dcb79f7ee202ec99d3e - currently this results in parsed log entries like (see this dashboard:

"@version": "1",
    "@timestamp": "2014-02-26T18:52:08.123+00:00",
    "@type": "csv_caret",
    "@shipper.host": "LON-WS01186",
    "@source.service": "ActiveMQStatsLoggerConsole",
    "@source.host": [
      "PKH-PPE-APP34",
      "LON-WS01186"
    ],
    "@environment": "PPE",
    "@source_tz": "+00:00",
    "@source.path": "\\\\PKH-PPE-APP34\\Logs\\Apps\\ActiveMQStatsLoggerConsole\\ActiveMqStats_20140225.log",
    "@source.offset": "211702295",
    "@message":  ...

[dpb587]

Good thought. Instead, I think we should remove the following:

add_field => [ "@source.host", "%{host}" ]

Then create a corrected conditional:

if "" == [@shipper.host] {
  mutate {
    add_field => [ '@shipper.host', "%{host}" ]
  }
}

Ultimately, I'd prefer that the shipper is the only thing adding shipper.* (not logstash), but I don't know that that's practical at this point (your .NET is a good example of what I think we should be doing), so the conditional will continue to support what we've been doing.

You've assigned this issue to yourself, but let me know if I should create a PR.

[mrdavidlaing]

I like the principal of only having the shipper add @shipper tags. Let's go with the conditional, with the intention of removing it eventually

----- Reply message ----- From: "Danny Berger" notifications@github.com To: "cityindex/logsearch" logsearch@noreply.github.com Cc: "David Laing" david.laing@cityindex.co.uk Subject: [logsearch] lumberjack endpoint, rename @source.host to @shipper.host (#350) Date: Thu, Feb 27, 2014 17:37

Good thought. Instead, I think we should remove the following:

add_field => [ "@source.host", "%{host}" ]

Then create a corrected conditional:

if "" == [@shipper.host] { mutate { add_field => [ '@shipper.host', "%{host}" ] } }

Ultimately, I'd prefer that the shipper is the only thing adding shipper.* (not logstash), but I don't know that that's practical at this point (your .NET is a good example of what I think we should be doing), so the conditional will continue to support what we've been doing.

You've assigned this issue to yourself, but let me know if I should create a PR.

— Reply to this email directly or view it on GitHub.

This email is intended only for the use of the individual or entity to which it is addressed and may contain information that is confidential, subject to copyright or which constitutes a trade secret. You should not copy it for any purpose or disclose its contents to any person. City Index Ltd has taken reasonable precautions to minimise the risk of transmitting software viruses but we advise you to carry out your own virus checks on any attachment to this message. We cannot accept any liability for any loss or damage resulting from any software viruses. If you receive this email in error, please telephone our postmaster on +44 20 7550 8500 or email postmaster@cityindex.co.uk and delete all copies on your system.

City Index Limited is authorised and regulated by the Financial Conduct Authority. FCA Register Number: 113942. Registered Office: Park House, 16 Finsbury Circus, London, EC2M 7EB. Registered in England and Wales, number: 1761813.