Feed Management Enhancements : READ and UPDATE lifecycleEvent

[csantanapr]

Feed Management Enhancements for Kafka feed

This issue is an Epic to collect all the related tasks to implement the final solution for kafka

For some background on feeds and lifecycles refer to https://github.com/apache/incubator-openwhisk/blob/master/docs/feeds.md

This features are based on user feedback

Feature Requests

• Allow the user retrieve trigger information from the feed provider
  • This allows the user to get back the original config
  • This allows the user to get the current status of the trigger
• Allow the user to update the trigger configuration on the feed provider without the need to delete and re-create the trigger with a new configuration

Lifecycle Events

Today this feed implements the lifecycle events CREATE and DETELETE To implement the features requested above the this feed would need to implement the lifecycle events READ, UPDATE

READ lifecycle

The feed action should accept the parameter lifecycleEvent with the value of READ.

It should return the configuration and status for the trigger, for example:

{
  "config":{
    "triggerName": "_/myKafkaTrigger",
    "isMessageHub": true,
    "topic": "mytopic",
    "isJSONData": true,
    "isBinaryValue": false,
    "isBinaryKey": false,
    "kafka_brokers_sasl": [
      "kafka01-prod01.messagehub.services.us-south.bluemix.net:9093",
      "kafka02-prod01.messagehub.services.us-south.bluemix.net:9093",
      "kafka03-prod01.messagehub.services.us-south.bluemix.net:9093",
      "kafka04-prod01.messagehub.services.us-south.bluemix.net:9093",
      "kafka05-prod01.messagehub.services.us-south.bluemix.net:9093"
    ],
    "user": "admin",
    "password": "123",
    "kafka_admin_url": "https://kafka-admin-prod01.messagehub.services.us-south.bluemix.net:443"
  },
  "status":{
    "active": false,
    "dateChanged":1509321736699,
    "dateChangedISO":"2017-10-30T00:02:16.699Z",
    "reason":{
      "kind": "AUTO",
      "statusCode": 403,
      "message": "Automatically disabled after receiving a 403 status code when firing the trigger."
    }
  }
}

Notes: The field status.reason not present when status.active is true

The clients would be able to invoke the feed action with this lifecycleEvent For example using the CLI directly invoking the feed action.

wsk action invoke /whisk.system/messaging/messageHubFeed -p triggerName _/myKafkaTrigger -p authKey guest:123 -p lifecycleEvent READ -r

The go-CLI would be able to call the feed action automatically on a trigger get. For example:

wsk trigger get myKafkaTrigger

ok: got trigger myCloudantTrigger
{
    "namespace": "guest",
    "name": "myKafkaTrigger",
    "version": "0.0.1",
    "annotations": [
        {
            "key": "feed",
            "value": "/whisk.system/messaging/messageHubFeed"
        }
    ]
}
ok: invoked feed action /whisk.system/messaging/messageHubFeed
{
  config:{
    triggerName: "_/myKafkaTrigger",
    isMessageHub: true,
    topic: "mytopic",
    isJSONData: true,
    isBinaryValue: false,
    isBinaryKey: false,
    kafka_brokers_sasl: [
      "kafka01-prod01.messagehub.services.us-south.bluemix.net:9093",
      "kafka02-prod01.messagehub.services.us-south.bluemix.net:9093",
      "kafka03-prod01.messagehub.services.us-south.bluemix.net:9093",
      "kafka04-prod01.messagehub.services.us-south.bluemix.net:9093",
      "kafka05-prod01.messagehub.services.us-south.bluemix.net:9093"
    ],
    user: "admin",
    password: "123",
    "kafka_admin_url": "https://kafka-admin-prod01.messagehub.services.us-south.bluemix.net:443"
  },
  status:{
    active: true
    }
  }
}

UPDATE lifecycle

The UPDATE lifecycle will allow users to update their feed without the need to delete and re-create their trigger. The feed action should accept the parameter lifecycleEvent with the value of UPDATE.

The user should be able to pass a partial set of the configuration values, allowing them to update their trigger feed.

The feed will be paused while it's being updated. For example if user wants to update the topic name, it will invoke the feed action with the topic parameter for the existing trigger.

For example invoking directly the feed action:

wsk action invoke /whisk.system/messaging/messageHubFeed -p topic newtopic -p triggerName _/myKafkaTrigger -p authKey guest:123 -p lifecycleEvent UPDATE -r

Using the cli using the integration embedded into the CLI

wsk trigger update myKafkaTrigger -p topic newtopic

[jberstler]

One wrinkle here is that, for Message Hub triggers, the properties recorded in the DB can have different names than the properties specified when creating the trigger. For example, I specify my brokers using the kafka_brokers_sasl property, but this gets recorded as brokers in the DB.

I would argue that when retrieving the trigger details, the user should always see the property names the same as what they set when creating the trigger. This may mean massaging the property name back to what the user expects to see (e.g. brokers -> kafka_brokers_sasl)

[csantanapr]

Agree we should not expose the implementation details and keep the abstraction of the interface to the user the same so massaging the keys is a good thing.

[csantanapr]

@abaruni Could you change READ config object to match the input parameters by user based on feedback from @jberstler above ^^ https://github.com/apache/incubator-openwhisk-package-kafka/issues/220#issuecomment-343330248

I would argue that when retrieving the trigger details, the user should always see the property names the same as what they set when creating the trigger. This may mean massaging the property name back to what the user expects to see (e.g. brokers -> kafka_brokers_sasl)

cc @jasonpet

[csantanapr]

I did a quick look it looks like there are 2 fields to change username -> user brokers -> kafka_brokers_sasl

{
"kafka_brokers_sasl": [
      "kafka01-prod01.messagehub.services.us-south.bluemix.net:9093",
      "kafka02-prod01.messagehub.services.us-south.bluemix.net:9093",
      "kafka03-prod01.messagehub.services.us-south.bluemix.net:9093",
      "kafka04-prod01.messagehub.services.us-south.bluemix.net:9093",
      "kafka05-prod01.messagehub.services.us-south.bluemix.net:9093"
    ],
"user": "admin"
}

@abaruni I will update description above, can you double check?

[csantanapr]

It's definitely user here in produce action https://github.com/apache/incubator-openwhisk-package-kafka/blob/master/action/messageHubProduce.py#L124

[csantanapr]

It's definitely user here in MessageHubFeedWeb Action https://github.com/apache/incubator-openwhisk-package-kafka/blob/master/action/messageHubFeedWeb.js#L145

But stored as username in db, so we need the get to change to user in the way out https://github.com/apache/incubator-openwhisk-package-kafka/blob/master/action/messageHubFeedWeb.js#L84-L86 it is

                            kafka_brokers_sasl: triggerDoc.brokers,
                            user: triggerDoc.username,

it should be

                            brokers: triggerDoc.brokers,
                            username: triggerDoc.username,