search

apache / openwhisk-package-kafka

Apache OpenWhisk package for communicating with Kafka or Message Hub
https://openwhisk.apache.org/
Apache License 2.0
32 stars 43 forks source link

How to use message feed #210

Closed zhy989 closed 7 years ago

zhy989 commented 7 years ago

I installed the message package successfully

-wsk -i package list packages /whisk.system/messagingWeb private /whisk.system/messaging shared /whisk.system/alarmsWeb private /whisk.system/alarms shared /whisk.system/combinators shared /whisk.system/watson-speechToText shared /whisk.system/samples shared /whisk.system/watson-translator shared /whisk.system/slack shared /whisk.system/websocket shared /whisk.system/github shared /whisk.system/utils shared /whisk.system/weather shared /whisk.system/watson-textToSpeech shared

why the CLI "wsk -i package refresh" failed ?

zhy989 commented 7 years ago

I can not create trigger MyMessageHubTrigger successfully.

jberstler commented 7 years ago

@zhy989 For interactive support, please join us on Slack: https://openwhisk.incubator.apache.org/slack.html

In the meantime, you will be able to find more details about why you cannot create a trigger successfully by looking at the logs for activation 69c08183c6c74241ba96c4ff6512f549. To do this, run wsk activation get 69c08183c6c74241ba96c4ff6512f549 and examine the logs field.

zhy989 commented 7 years ago

@bjustin-ibm thanks

jberstler commented 7 years ago

Ah ok... the most likely problem is that localhost does not resolve to what you think it does. Keep in mind that actions run inside docker containers, and name resolution for things like localhost and 127.0.0.1 get a little tricky.

I suggest that instead of using localhost as your edgehost and apihost argument values when installing the actions, you use a real IP address that can be used for cross-container communication: https://github.com/bjustin-ibm/openwhisk-package-kafka/blob/5f9c08daa845b7c376f236e6cc8ede9a32243138/devGuide.md#install-actions

zhy989 commented 7 years ago

@bjustin-ibm thanks ! I update the ip address from localhost to 172.17.0.1, but there is another failure about "Could not authenticate with Message Hub. Please check your credentials" with messageHub and "connection error" with kafkaMessage

The log is as below:

the error log is as below:

jberstler commented 7 years ago

@zhy989 I think the first case is pretty clear... it appears that the Message Hub credentials you are trying to use are not correct. Please verify them.

In the second case, it seems most likely that you are not escaping quotes properly in the value of your brokers parameter. As it turns out, you can also use a comma-separated list of brokers, rather than trying to get your shell to properly interpret a JSON array. For example: 

-p brokers "mykafkahost:9092,mykafkahost:9093"

Of course this assumes that mykafkahost can really be resolved, and that it represents a valid Kafka cluster at those ports, and that you have tested and verified that it is working properly using something other than the feed provider service (for example, using a Kafka client).

zhy989 commented 7 years ago

@bjustin-ibm thanks in case 1: Should I register a new account on the local host for message hub? in case 2: I tried this CLI "wsk trigger create MyKafkaTrigger -f /whisk.system/messaging/kafkaFeed -p brokers "mykafkahost:9092,mykafkahost:9093" -p topic mytopic -p isJSONData true -i", and got the same error

Should I start up the catalog_kafkatrigger container before create the trigger ? if do and how to start up that container?

jberstler commented 7 years ago

Please keep in mind that this feed provider (the code in this repo) interacts with an external service - either a Message Hub instance or generic Kafka cluster. This feed provider cannot work without having the external service created and in working order. Following the README and running the example commands (which only have example parameter values) will almost definitely not work without being changed so that the parameter values match your specific instance of Kafka or Message Hub. I suggest you first become familiar with Kafka and/or Message Hub before attempting to run this software.

Should I register a new account on the local host for message hub?

Message Hub is the name of a service hosted in IBM Bluemix. It is essentially Bluemix-hosted Kafka, and is not something you can run locally. Please see the following for details: https://developer.ibm.com/messaging/message-hub/

Should I start up the catalog_kafkatrigger container before create the trigger ? if do and how to start up that container?

Yes, if you are trying to build and run your own instance of the feed provider service, please carefully follow this development guide: https://github.com/bjustin-ibm/openwhisk-package-kafka/blob/1aa6ae1684930d5175fa0ce36886b2e7db123248/devGuide.md However, please be sure you have a working Kafka or Message Hub instance created, configured, tested, and working before attempting this.

zhy989 commented 7 years ago

@bjustin-ibm thanks in case 1

I can sign in Bluemix with my username and password, why can not authenticate with Message Hub with the same username and password when I run the CLI "wsk -i trigger create MyMessageHubTrigger -f myMessageHub/messageHubFeed -p topic mytopic -p isJSONData true -p"?

in case 2 I try to run the example for "https://github.com/bjustin-ibm/openwhisk-package-kafka/blob/1aa6ae1684930d5175fa0ce36886b2e7db123248/devGuide.md"

I can create docker image kafkafeedprovider successfully but when I run script docker_run.sh, there is an error:

In here it must be CLOUDANT_USER ? couchdb_user can not ?

jberstler commented 7 years ago

Setting up a Message Hub package outside Bluemix Is this install Message Hub package on the local host outside Bluemix?

No. This only installs the related actions into your OpenWhisk. It does not create a Message Hub or Kafka instance that you can use.

I can sign in Bluemix with my username and password, why can not authenticate with Message Hub with the same username and password

You have to first provision a Message Hub instance in Bluemix. When you do this, a set of Message Hub credentials (that are different from your Bluemix username and password) will be generated. You will need to use these credentials when creating the trigger. Please refer to the Getting Started with Message Hub documentation.

I can create docker image kafkafeedprovider successfully but when I run script docker_run.sh, there is an error

It appears that you are using an older version of the feed provider that used to require a CLOUDANT_USER environment variable. Please update your repository to the latest in the master branch, then rebuild the image, and start a new container.

zhy989 commented 7 years ago

@bjustin-ibm thanks! Can Kafka instance run on the local host outside the Bluemix?

jberstler commented 7 years ago

Yes, please refer to the following Apache Kafka guides: https://kafka.apache.org/intro https://kafka.apache.org/quickstart

zhy989 commented 7 years ago

@bjustin-ibm thanks

I had run Kafka instance successfully follow "https://kafka.apache.org/quickstart".

I re download the code from "https://github.com/bjustin-ibm/openwhisk-package-kafka" and update to the master branch .

run the docker image and got the same error as before:

I download code from "https://github.com/apache/incubator-openwhisk-package-kafka" run the docker image and got the error as below

jberstler commented 7 years ago

I re download the code from "https://github.com/bjustin-ibm/openwhisk-package-kafka" and update to the master branch .

Ok, you definitely don't want the code from my fork, especially not from the master branch as I never bother to update it.

I download code from "https://github.com/apache/incubator-openwhisk-package-kafka" run the docker image and got the error as below

While this is the code you want, it seems it is unable to connect to your CouchDB or Cloudant instance. Please verify that your DB is running and that the parameters you have supplied as environment variables regarding your DB details are correct: https://github.com/bjustin-ibm/openwhisk-package-kafka/blob/1aa6ae1684930d5175fa0ce36886b2e7db123248/devGuide.md#mandatory-environment-variables

Keep in mind that the environment variable values shown in the example command are just that, examples. They almost certainly will not work in your environment without editing them to contain the correct information (i.e. your DB URL and credentials).

zhy989 commented 7 years ago

@bjustin-ibm thanks! I had run the docker image(kafkafeedprovider) successfully.

then I try to create kafka trigger with CLI :

It failed , the logs is as below:

Is the variable values about brokers (-p brokers "mykafkahost:9092, mykafkahost:9093" ) right? Should I set the brokers variable to the service port of the docker image(kafkafeedprovider) or set to localhost:9092 ?

I tried both borkers , and got the same error ""Error: error happened in your connection".

jberstler commented 7 years ago

Is the variable values about brokers (-p brokers "mykafkahost:9092, mykafkahost:9093" ) right?

Almost certainly not. These are just example values that need to be replaced with the hostname and port of the brokers in your Kafka cluster.

zhy989 commented 7 years ago

@bjustin-ibm thanks

Is docker image( kafkafeedprovider) a kafka server ?
When I run the docker image (kafkafeedprovider), the service port is set to 8081 on the host and 5000 in the container ( -p 8081:5000) .

so when I create the kafka trigger , the brokers is set to localhost:8081, is right? I have tried brokers (-p localhost:8081), there is also connect error as before.

jberstler commented 7 years ago

@zhy989 No, the kafkaFeedProvider image does not contain a Kafka server. However, the OpenWhisk core system does use one. This is the one from which you managed to list topics (e.g. "health", "completed"). I highly recommend you do not use this kafka for anything, and let it stay dedicated to the needs of the OpenWhisk core system. Instead I suggest you set up your own separate kafka server.

Also, please remember that "localhost" only resolves relative to the environment where the action runs. As such, when the feed action runs in a docker container, "localhost" very likely does not resolve to the IP address you think it does. It is always best to use specific IP addresses (that are not 127.0.0.1).

zhy989 commented 7 years ago

@bjustin-ibm thanks I have created the kafka trigger successfully, the reason is the couchdb url should be http not https.

I follow below CLI

There is a 's' . I remove the 's', the kafka trigger can be created successfully.

Then I create rule iot1 related MyKafkaTrigger1 and hello action , when I produce a message to kafka, but the helloworld action did not run . When kafka received a message how to trigger the hello action ?

jberstler commented 7 years ago

@zhy989 First verify that the message was indeed produced. Yes, the action should fail if the produce was not successful, but it helps to be sure. You can verify by doing wsk activation get 5e5ed2151ecd4a79ac2576ee38f0bab3 and examining the result. It should say something like "Successfully sent message to {topic}:{partition} at offset {offset}"

Next, verify whether or not the trigger has fired in response to the message. You can do this by looking for trigger activations with wsk activation list MyKafkaTrigger1. If this returns nothing, then your trigger is not firing, and you should examine the container logs to figure out why.

If the trigger is firing, then similarly verify whether or not the rule is being invoked. This can be done with wsk activation list iot1. If this list is empty then your rule has likely not been created properly. However, given the set of commands you show above, this seems very unlikely.

If the rule is firing, then check to see if your action is being invoked with wsk activation list hello. If this list is empty, then your action is not being invoked. If everything up to this point has been working, this seems very unlikely as well.

In all, you need to use the procedure above to see where in the chain of expected events things are not happening. Once you find that point, look at the activation for the last successful part of the chain to examine the result and logs to look for clues about why the next step is failing. If it seems that the trigger is not firing at all, use the docker logs command to examine the kafkaFeedProvider container to see why the trigger isn't working.

zhy989 commented 7 years ago

@bjustin-ibm thanks！ The action can be invoked successfully when I set verify to False from line "response = requests.post(self.triggerURL, json=payload, timeout=10.0, verify=False)" in file "incubator-openwhisk-package-kafka/provider/consumer.py"

jberstler commented 7 years ago

@zhy989 ah very nice! Any easier way to handle this situation is to set the LOCAL_DEV environment variable to true when starting the container: https://github.com/apache/incubator-openwhisk-package-kafka/blob/master/devGuide.md#optional-environment-variables

Doing this will allow you to use the code unchanged, while still allowing the service to POST to trigger URLs in your local OpenWhisk.

zhy989 commented 7 years ago

@bjustin-ibm thanks! I have tried to set the LOCAL_DEV environment variable to true, it works well without modify code, thanks!