[Core feature] Add Kafka support for workflow notifications

[devictr]

Motivation: Why do you think this is important?

Notifications only support AWS and GCP for the moment. My company is on Azure and would benefit from having notifications as well. It seems like Flyte uses gizmo for this, which only supports AWS, GCP, Kafka and HTTP.

Adding Kafka support would benefit Azure deployments, but also on-premise or other deployments that already have Kafka setups internally.

Goal: What should the final outcome look like, ideally?

I want to be able to configure Kafka for workflow notifications.

Describe alternatives you've considered

Alternatives: HTTP support from gizmo. This feature doesn't exclude HTTP support from being added in the future.

Propose: Link/Inline OR Additional context

Code: https://github.com/flyteorg/flyte/blob/b30cb723afc7c966516c8460f7a82246b068bea4/flyteadmin/pkg/async/notifications/factory.go#L95-L147

Are you sure this issue hasn't been raised already?

• [X] Yes

Have you read the Code of Conduct?

• [X] Yes

[Future-Outlier]

I added the sandbox email publisher 3 months ago. If you need help, feel free to collaborate with me. PR Link: https://github.com/flyteorg/flyteadmin/pull/595

[devictr]

Thanks @Future-Outlier!

[devictr]

@pingsutw, any preference as to where to store Kafka offsets? Since this is flyteadmin, maybe I can create a table in postgres for that?

[devictr]

Actually, seems like we use Sarama behind the scenes, which supports auto commit and is enabled by default: https://github.com/IBM/sarama/blob/main/config.go#L427

[davidmirror-ops]

with gizmo on maintenance mode do we still consider following their http route?

[eapolinario]

We talked offline about this and the conclusion is that we're not married to gizmo. We'd welcome a PR that brings in another dependency to implement Azure workflow notifications.

[Sovietaced]

We have an implementation of this internally that we could potentially upstream.