fluent / fluent-bit-go

Fluent Bit Golang package to build plugins
Apache License 2.0
188 stars 52 forks source link

can not get specific log from records #44

Open wexgjduv opened 3 years ago

wexgjduv commented 3 years ago

I have fluent-bit deployed in k8s with the go plugin, but we noticed some specific logs are lost.

Here is the pod's log:

{"level":"debug","ts":1625627314150.1467,"caller":"xservice/callback.go:179","msg":"api call done","app":"test","appBuildHost":"runner-x","appBuildTime":"2021-07-06 10:14:06","appGoVersion":"go1.15","appGitTag":"v0.6.0_2021-07-06_10-14-06","appGitVersion":"30333326","Resp":"task_id:1628888882647513000 data:{x_id:6108888888888787 xList:{rn:\"rn:xt:x/64888888889070\" timestamp:{seconds:1625627313 nanos:21238000} x_id:2020092191 status:FINISHED amount:{amount:1} x_list:{amount:{amount:1} x:{rn:\"rn:xx:y/59578888888885733\" x_owner:{user_rn:\"rn:user:legacy-user/1688850\" xx_qq_rn:\"rn:xx:xx/58033951876352146:qq/59600811175680000\" display_name:{language:\"zh-CN\" text:\"张\"}} type:EXTERNAL display_name:{language:\"zh-CN\" text:\"测试 Mall\"}} external_xx_info:\"{\\"Data\\":{\\"XInCent\\":1,\\"CreateTime\\":1625627313,\\"Cent\\":1,\\"ETime\\":1625627313,\\"ErrorCode\\":null,\\"ErrorDescription\\":null,\\"Status\\":\\"Success\\"},\\"ID\\":\\"73b7cb2ad93f\\",\\"NotifyTime\\":1625627313,\\"NotifyType\\":\\"XResult\\",\\"XServiceID\\":1712788888888888889,\\"XNumber\\":46766078888888888875}\"}}}","x-trace-id":"da488888888888758893429","x-span-id":"7188888888833","x-parent-id":"388888888886"}

In FLBPluginFlushCtx, I tried to print all records but found no messages like api call done in fluent-bit's output.

//export FLBPluginFlushCtx
func FLBPluginFlushCtx(ctx, data unsafe.Pointer, length C.int, tag *C.char) int {
    // Type assert context back into the original type for the Go variable
    // cfgContext := output.FLBPluginGetContext(ctx).(map[string]string)

    dec := output.NewDecoder(data, int(length))

    messages := make(map[string][]string)

    for {
        ret, _, record := output.GetRecord(dec)
        if ret != 0 {
            break
        }

        fbTag := fmt.Sprintf("%s", C.GoString(tag))

        for k, v := range record {
            fmt.Printf("tag: %s, record: \"%s\": %v, \n", fbTag, k, v)
        }
    }

    return output.FLB_OK
}
wexgjduv commented 3 years ago

After inspecting the file contents in running pod's log file, some logs had wrong encoding like \u0008˵\ufffd or \ufffd<90>\ufffd\u or \ufffdʁ\ufffd\. I disabled outputting of these logs, other logs were collected by our plugin successfully. I think the encoding of the pod's log may affect the collectiong of other logs.