[Questions] Java based Kafka Producer to trigger OpenWhisk action

[patpatpat123]

Hello,

I wanted to contribute with a Java based Kafka end to end flow for serverless OpenWhisk.

By that, I mean to take the basic hello world and have a java Kafka based producer on one side, with a Kafka based consumer on the other. (That will mimic real life scenario where micro services call each other, or an OpenWhisk action responding to live Kafka event).

public class Hello { public static JsonObject main(JsonObject args) { String name = "stranger"; if (args.has("name")) name = args.getAsJsonPrimitive("name").getAsString(); JsonObject response = new JsonObject(); response.addProperty("greeting", "Hello " + name + "!"); return response; } }

Can it be possible to trigger/invoke this OpenWhisk deployed function with a plain java Producer, like this?

public class Producer { public static void main(String[] args) throws Exception{ Properties kafkaParams = new Properties(); String topicName = "mytopic"; kafkaParams.put("bootstrap.servers", "some-kafka:9092"); [and some other Kafka properties] Producer<String, String> producer = new KafkaProducer<>(kafkaParams); producer.send(new ProducerRecord<>(topicName,"somekey", "{\"name\":\"world\"})); }

1. First of all, is it something supported? Feasible? If yes (which I really hope!), what is the correct way to deploy the hello world function? Still with wsk -i action create helloJava pof-1.1.jar --main Hello ?

2. If we use a local docker development first, shall we do the docker pull of openwhisk/kafkaprovider? Shall we incorporate it to the docker-compose file of incubator-openwhisk-devtools for the docker run?

3. For Kafka, but with a local docker, shall we start with the package binding? wsk package bind /whisk.system/messaging myMessageHub -p kafka_brokers_sasl "[\"some-kaka:9092\"]" without user, password and admin?

4. I believe the trigger is the most important step, linking the producer to the java function (OpenWhisk action), is that correct? if yes, shall I run wsk trigger create MyKafkaTrigger -f /whisk.system/messaging/kafkaFeed -p brokers "[\"some-kafka:9092\", \"some-kafka:9093\"]" -p topic mytopic -p isJSONData true

5. About the kafkaParams, which one shall be altered for OpenWhisk? The key/value serializer can be left with StringSerializer? How about he security protocols?

6. Finally, what will the payload looks like? For example, if the isJSONData is true, shall I send "{\"name\":\"world\"}? {"value": "{\"name\":\"world\"}"}? Or the entire message payload with partition, key, offset, topic and value those shall be manually added? Or just to focus on the value of the payload?

I hope not to trouble with those questions, as I believe it is an interesting case and usage of OpenWhisk package Kafka and OpenWhisk in general.

Thank you a lot

[csantanapr]

Yes is possible to use java for a kafka producer, you can use any kafka client there are many in other languages also. The producer is independent from OpenWhisk. Yes create the function/action in any language including java, then create the trigger and rule to the action usinbthe action feed like you have it.

For the producer side if you are getting started with kafka technology just focus on the value for now not the key.

[patpatpat123]

Hello Carlos,

Thanks a lot for your quick response.

I believe to get my answer for question 1, I believe it is yes, which is quite motivating!

As for question 2, what is the relation between this repo, the docker container and kafkaprovider? Shall I docker run the container? Link it to controller or invoker? Shall I run one of the shell script from this repo?

For question 3, can you suggest the package binding command as written in the readme of this repo? You did not mention this step, does it mean it is an optional step?

For question 4, is it ok to run it like this based on the fact that I am running the local dev version with Docker.

I believe to understand for question 5 and 6. I can leave all default value for the producer and input some random string for the key. As for the value, just to focus on the payload itself.

Again, thank you for your help.

[csantanapr]

Take a look here on how to run the kafkaprovider container https://github.com/apache/incubator-openwhisk-package-kafka/blob/master/devGuide.md

[patpatpat123]

Hello,

Thank you for this reference. May I ask a quick follow up question?

Quoting the doc: "Setting up a Message Hub package outside Bluemix

If you're not using OpenWhisk in Bluemix or if you want to set up your Message Hub outside of Bluemix, you must manually create a package binding for your Message Hub service. You need the Message Hub service credentials and connection information.

Create a package binding that is configured for your Message Hub service. $ wsk package bind /whisk.system/messaging myMessageHub -p kafka_brokers_sasl "[\"kafka01-prod01.messagehub.services.us-south.bluemix.net:9093\", \"kafka02-prod01.messagehub.services.us-south.bluemix.net:9093\", \"kafka03-prod01.messagehub.services.us-south.bluemix.net:9093\"]" -p user -p password -p kafka_admin_url https://kafka-admin-prod01.messagehub.services.us-south.bluemix.net:443 "

Is it possible to have the equivalent for localhost testing? Like from the docker-compose file? Could you help suggest a command? wsk package bind /whisk.system/messaging myMessageHub -p kafka_brokers_sasl "[\"localhost???:9093\"]" -p user how-can-I-find-this-user-for-local? -p how-can-I-find-this-password-for-local? -p how-can-I-find-this-url-for-local? https://localhost???:443

Thank you a lot for your help.

[csantanapr]

How are you setting up couchdb container? The problem is that the kafkaprovider can’t reach couchdb container.

You can try docker inspect couchdb and get the internal IP from couchdb and use that IP for the kafkaprovider to reach to it via internal network.

It would be the same IP that container can controller and invoker uses to connect to couchdb

[patpatpat123]

Hello Carlos,

Thank you for your advices. I started with docker run -e LOCAL_DEV=true -e DB_URL=http://xx.xx.xxx.xxx:5984 -e DB_USER=whisk_admin -e DB_PASS=some_passw0rd -p 81:5000 openwhisk/kafkaprovider where xx.xx.xxx.xxx was given by make quick-start ok: whisk auth set. Run 'wsk property get --auth' to see the new value. ok: whisk API host set to xx.xx.xxx.xxx ok: whisk namespace set to guest localhost and 127.0.0.1 will not work, even with port forwarding version: '3' services: db: image: apache/couchdb:2.1 ports:

• "5984:5984" environment: COUCHDB_USER: whisk_admin COUCHDB_PASSWORD: some_passw0rd volumes:
• ~/tmp/openwhisk/couchdb:/usr/local/var/lib/couchdb:rw Is it expected?

Then, I have a question for the package bind. What would be the

wsk package bind comment for a local docker environment?

Thank you so much.

[patpatpat123]

Or if anyone have the localhost version with the make quick-start + docker-compose from OpenWhisk repo, it will be very helpful

[patpatpat123]

Thank you Carlos, closing