Funtion configured with kafka trigger not receiving all messages

[gemanilkashyap]

BUG REPORT

What happened: Python code function configured with Kafka trigger was not receiving all the messages (I could say 1 out of 5 received).

What you expected to happen: Function to receive all message

How to reproduce it (as minimally and precisely as possible): I couldnt suggest anything apart from me left them running for few days (random behaviour)

Anything else we need to know?: I did a function delete and redeploy after which it started behaving normal.

Environment:

• Kubernetes version: v1.15.3

• Kubeless version (use kubeless version): v1.0.3

• Cloud provider or physical cluster: AWS

[andresmgot]

There are two possible points of failure:

• Kafka/Zookeeper are not processing the message. You can check that in those services logs.
• Kafka-controller is not processing the message. You can check that in the logs from the controller in the kubeless namespace.

Can you check if in those logs you are able to see any error?

[gemanilkashyap]

@andresmgot thanks for the reply and sorry for not responding. I got the issue yesterday and found no error within kafka controller logs. Though i suspect that it could be because i do produce kafka message operation with in function call. Do you have the best practice to do kafka and database operations within function configured against kafka trigger?

[andresmgot]

There is no a specific good practice I can point you to. I would recommend you to log in your function if it's able to reach Kafka and properly publish the message. That way you can narrow down the possible causes of the issue. If you have a timestamp of the failed message it will be easier to debug what happened.

[gemanilkashyap]

@andresmgot thanks for the reply and sorry for not responding. I got the issue yesterday and found no error within kafka controller logs. Though i suspect that it could be because i do produce kafka message operation with in function call. Do you have the best practice to do kafka and database operations within function configured against kafka trigger?

[gemanilkashyap]

I was looking into any specific way of handling connections to kafka or database when working with kubeless? Currently closing kafka connection with each function call

[andresmgot]

Depending on the runtime you may be able to open the connection outside the function code. This improves performance but it keeps the connection open for the time the pod is running.

[gemanilkashyap]

Sure i will give it a go

[gemanilkashyap]

@andresmgot I created the database connection and kafka connection to publish outside the function to reuse the connection but it is not giving the performance i was expecting and also it stops after inserting few records, my runtime is python.

[andresmgot]

There are other bottlenecks regarding Kafka, see https://github.com/kubeless/kubeless/issues/826 for more details. It's unresolved at this moment.

[gemanilkashyap]

Hmm, will that reuse the kafka and database connections across multiple function invocations?

[gemanilkashyap]

Hmm, will that reuse the kafka and database connections across multiple function invocations?

[andresmgot]

In the case of python, if I recall correctly, a new thread is created to execute the function every time it's invoked so I think it's not reusing the database connection. You can double check that if you print some message in the code that is creating the connection and check if that message is printed every time.

[gemanilkashyap]

Ahh ok it seem to be a costly and performance hit, have you got a way to get connections within python functions in the past?

[andresmgot]

Unfortunately no, I have no personal experience trying to do so. Happy to know your findings if you work on this!