fluent / fluent-bit-go

Fluent Bit Golang package to build plugins

Apache License 2.0

189 stars 52 forks source link

Add support for Workers #45

Open kishaningithub opened 3 years ago

kishaningithub commented 3 years ago

Fluent bit added support for workers in v.1.7.0

https://fluentbit.io/announcements/v1.7.0/

It would be really valuble go plugins can also have worker support

Impact

Because of the single threaded nature if one output plugin is "stuck", then the subsequent outputs / pipelines are not getting executed https://github.com/fluent/fluent-bit/issues/3088

I believe workers can solve this problem. Basically setting workers to 1 should ensure there is a separate thread for processing

kishaningithub commented 3 years ago

From what i observed setting

Workers 1

did not spawn a new thread in the engine for data delivery.

IMO, full workers need not be supported as go lang has goroutines. But what is needed in an ability for the go plugins to run in a separate delivery thread.

nokute78 commented 3 years ago

I merged the patch https://github.com/fluent/fluent-bit/pull/3908 @gautampunhani Thank you.

l2dy commented 3 years ago

Should we add the RWMutex mentioned in https://github.com/fluent/fluent-bit-go/commit/b191f58f228c29d1df67ee75ecbee7eb413854ef to protect the contexts map?

nokute78 commented 3 years ago

I think it is better to add RWMutex for prevention.

In my understanding, a lock is taken care on C side. https://fluentbit.io/announcements/v1.7.0/

The multithread implementation is lock-free at runtime.

On the other hand, the API FLBPluginSetContext and FLBPluginGetContext are for golang side API. We can't know when they are called.

l2dy commented 3 years ago

I think "lock-free" means that there is no lock on the C side, so contexts in Go must be protected with a mutex or sync.Map.