search

fluent / fluent-bit-go

Fluent Bit Golang package to build plugins
Apache License 2.0
189 stars 52 forks source link

Decoder don't support new timestamp + metadata log record format #66

Closed edsiper closed 1 year ago

edsiper commented 1 year ago

In Fluent Bit we introduced a new format for log records:

[[timestamp, {METADATA}], {EVENT}]

Fluent Bit code base has been updated to work with the old format and the new one. But the golang decoder is not updated and needs to be fixed.

cosmo0920 commented 1 year ago

I tested this for in_dummy and gstdout that is built by this Golang interface. But, it works. Is there a fluent-bit configuration any containing for this issue? Or, minimal reproducible steps? IIRC, there are a few cases to add metadata on fluent-bit. One of the main scenarios is using multiline feature.

edsiper commented 1 year ago

How do you extract metadata by using the golang output decoder?

On Fri, Jul 14, 2023, 04:51 Hiroshi Hatake @.***> wrote:

I tested this for in_dummy and gstdout that is built by this Golang interface. But, it works. Is there a fluent-bit configuration any containing for this issue? Or, minimal reproducible steps?

— Reply to this email directly, view it on GitHub https://github.com/fluent/fluent-bit-go/issues/66#issuecomment-1635681914, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAC2INVW76O5YFPX6V46ZX3XQEQCNANCNFSM6AAAAAA2JGA2KY . You are receiving this because you authored the thread.Message ID: @.***>

cosmo0920 commented 1 year ago

Ah, sorry for confusing. I misunderstood for this issue and I wrote down a patch for handling this issue.

randomizedcoder commented 1 year ago

Tried to add a little more to the tests, but it fails. https://github.com/fluent/fluent-bit-go/pull/69

rasturic commented 1 year ago

I'm looking forward to the fix as our company depends on https://github.com/logzio/fluent-bit-logzio-output which uses this.

randomizedcoder commented 1 year ago

I realized today that Fluentbit can send an array messages, rather than single message at a time. This is probably good for performance (batching), but there are no tests with this type. I only noticed because I increased the message rate of my test.

I just added a few more test data files here https://github.com/randomizedcoder/fluent-bit-go/tree/tweaks/output/testdata

Decoding the msgpack to yaml, you can see the data1 file turns into data1.yaml. The second record starts here, for example https://github.com/randomizedcoder/fluent-bit-go/blob/tweaks/output/testdata/data1.yaml#L527