uselagoon / lagoon-charts

A collection of Helm charts for Lagoon and associated services.
Apache License 2.0
11 stars 10 forks source link

Support TCP application logs in lagoon-logging #661

Open tobybellwood opened 1 month ago

tobybellwood commented 1 month ago

lagoon-logging currently supports UDP for application logs.

In order to support more logging transports more easily (such as Winston, Pino etc), we should also accept TCP logs on a separate port

smlx commented 1 month ago

I took a look at the docs. fluentd JSON parser expects one object per line (newline delimited). This is configurable but the default is simplest.

So adding a port on the statefulset:

        - containerPort: 5140
          protocol: TCP
          name: application_tcp

The service:

  - name: application_tcp
    port: 5140
    protocol: TCP
    targetPort: application_tcp

And a snippet like this to the fluent config should do the trick (totally untested):

    <source>
      @type  tcp
      @id    in_application_tcp
      tag    "lagoon.#{ENV['CLUSTER_NAME']}.application"
      port   5140
      <parse>
        @type json
      </parse>
    </source>

And then we need to document how to get logs there in the right format. And note that if you don't escape newlines (so a line has half a JSON object), the log message will be dropped.