Publish function result as kafka message

[migmartri]

Currently, it is possible to make a function to be triggered by listening to a Kafka topic by providing the -trigger-topic <topic_name> flag on creation.

It might be interesting to be able to tell the result of the function to be published into an specific topic as well. This will enable chaining/pipeline capabilities.

From the user perspective this might mean to add an extra flag like for example trigger-topic-result (I do not like this specific naming).

kubeless function create test --runtime python27 \
                              --handler test.foobar \
                              --from-file test.py \
                              --trigger-topic <topic_name>

[stigsb]

I like this idea. A good flag name could be simply --result-topic.

[sebgoa]

we need to look more into function chaining in other solutions. Not sure how it is done in them (e.g lambda, azure functions etc).

[arjunrao87]

I wanted to follow up on this discussion. We have a use case where we are looking to chain a bunch of kubeless functions and I was wondering what the community approach was to solve this problem.

From what I see for Azure they seem to use the Azure Service Bus for this ( https://docs.microsoft.com/en-us/azure/azure-functions/functions-bindings-service-bus )

Amazon has 2 approaches

• Simpler approach to use Amazon SNS as the queue to publish to and read from ( http://docs.aws.amazon.com/sns/latest/dg/sns-lambda.html )
• The new approach of creating state machines which they call Step Functions ( https://aws.amazon.com/blogs/aws/new-aws-step-functions-build-distributed-applications-using-visual-workflows/ )

In Kubeless today, do we have the following capabilities? ( I couldn't glean it from the documentation/open issues )

1. Ability to publish to a Kafka queue topic from a serverless function that is triggered via pub-sub ( so that the next function can be triggered via pub sub )
2. External process ( outside of kubeless but inside the same k8s cluster ) to populate a kubeless topic so it can trigger a pub-sub function.

[sebgoa]

In Kubeless today, do we have the following capabilities? ( I couldn't glean it from the >documentation/open issues )

• Ability to publish to a Kafka queue topic from a serverless function that is triggered via pub-sub ( >so that the next function can be triggered via pub sub )

This is not automated, but your function could also contain a kafka producer to publish any other message to a separate topic.

• External process ( outside of kubeless but inside the same k8s cluster ) to populate a kubeless >topic so it can trigger a pub-sub function.

the kafka that you get with the default kubeless is totally generic. The only thing is that it lives in the kubeless namespace. So you can definitely publish to it from anywhere, just point to the broker at the dns name kafka.kubeless

[sebgoa]

Also you can do basic chaining with the serverless plugin, check out this example:

https://github.com/serverless/serverless-kubeless/tree/master/examples/node-chaining-functions

[pplavetzki]

Is this issue/feature still in development or in consideration for development? I'm looking to do some non-trivial workflow with kubeless and it does require a bit more logic than simple function chaining.

[sebgoa]

this is definitely needed even though there is a need to publish back to various types of eventing system.

@deissnerk had a proposal in the works maybe he can share ?

[LudovitVarga]

Any update on this? Missing of this one feature in Kubeless is currently big plus for Fission.

[andresmgot]

Not right now @LudovitVarga, what you can do is to add the code in your function to send messages to Kafka. See this conversation: https://github.com/kubeless/kubeless/issues/881#issuecomment-421224305

[LudovitVarga]

I know about this workaround but with direct support in trigger, functions can be less coupled to Kafka itself. Also if you have consumer and publisher in same place, you can use Kafka transactions what is big plus.