Filter out "Empty" information messages

JonnyBooker commented 2 years ago

I am currently using Gelf to forward logs from a Gitlab instance I am running under docker. I am running this under Seq itself as a NuGet package with the following settings:

{
  "gelfAddress": "udp://172.18.0.11:12202",
  "certificatePath": "",
  "certificatePrivateKeyPath": "",
  "enableDiagnostics": "False"
}

The logs are being received just fine and the docker logging driver looks to be configured correctly, however I am getting some logs where there is no information in them:

Which looks like this when I export as JSON for a "empty" row with a \r value in it:

{
  "@t": "2022-11-09T00:10:53.4760000Z",
  "@mt": "\r",
  "@m": "\r",
  "@i": "e1221a99",
  "container_id": "5923b89cf3bbc470cdcb0134907aaed7d9099787907b5a2b9acac7549aadb32a",
  "created": "2022-11-08T23:43:01.931459734Z",
  "image_id": "sha256:eec20347402c7c4395f066925eb4a92702de8b12c781c75542f0ad17dec4f13a",
  "tag": "5923b89cf3bb",
  "image_name": "gitlab/gitlab-ce:latest",
  "command": "/assets/wrapper",
  "host": "xxxx",
  "container_name": "gitlab",
  "Application": "GitLab",
  "IsDocker": "true"
}

I was hoping I could potentially leverage the "filter" field however applying this filter seems to stop logs coming through at all, which I have tried using <> and != and just = just in case to try all options:

And if I enable diagnostics, I can see the messages still coming in but they don't show in the UI:

{
  "@t": "2022-11-09T00:24:22.5582479Z",
  "@mt": "Collected GELF server metrics",
  "@m": "Collected GELF server metrics",
  "@i": "1fcf524e",
  "@l": "DEBUG",
  "process": {
    "msg": 168
  },
  "receive": {
    "chunk": 0,
    "msg_chunked": 0,
    "msg_incomplete_chunk_overflow": 0,
    "msg_unchunked": 168
  },
  "server": {
    "process_err": 0,
    "process_ok": 168,
    "receive_err": 0,
    "receive_ok": 168,
    "tcp_conn_accept": 0,
    "tcp_conn_close": 0,
    "tcp_conn_timeout": 0,
    "tcp_msg_overflow": 0
  },
  "AppId": "hostedapp-36",
  "AppInstanceId": "appinstance-66"
}

Was wondering if could advise on how to filter out these empty logs at all?

nblumhardt commented 2 years ago

Hi @JonnyBooker,

The \r escape sequence isn't supported in Seq's query language strings; I think the best you can do for your filter is to use a regex with the \s whitespace matcher:

@Message <> /\s/

(Filters use the Seq names for properties like @Message, rather than the shorthand @m.)

Let me know if this helps! Nick

JonnyBooker commented 2 years ago

@nblumhardt - That makes sense now, thank you for the clarity. I've added the Filter as you've written above and that works great, immediately no more blank logs coming through, much cleaner log history.

Thanks for the help and quick response.

datalust / seq-input-gelf

Filter out "Empty" information messages #114