Add support of MQTT to Event Service

[devguyio]

Description

Add support of MQTT protocol in Kyma to allow external MQTT clients. First, a proposal needs to be created that answers following questions:

1. Do we need an extra API endpoints?
2. How does the final API look like?
3. How do you know if that's an MQTT application? And how to differentiate?
4. How will we figure out the Application name? What is the application path
5. Can we use Knative Eventing Source or update the Eventing Service ?

AC

• A proposal was created and approved.
• Kyma can receive MQTT events from outside and subscribers receive them successfully.
• Validation of MQTT was done using one of the standard MQTT clients.

[devguyio]

@brendandobbs can you take a look at this?

[brendandobbs]

@Abd4llA , will do!

[brendandobbs]

The source code for for the current Marketing MQTT implementation can be found here https://github.com/kyma-incubator/connector-tools/tree/master/mqtt-event-bridge. There is also a diagram that can be found here https://github.com/kyma-incubator/connector-tools.

Some important points:

• It's not a full blown MQTT broker. If we used a broker implementation we would need to manage subscriptions in 2 places.
• The broker also offers durability by persisting the messages (in memory or to disk) which we don't want either since we have nats doing that
• The Marketing MQTT client is using MQTT over websockets. A generic implementation should support direct MQTT directly also
• The Marketing implementation also requires an oauth2 server. We ship an oauth2 server with the preview connector but we could now use hydra

[brendandobbs]

Here is a sample event from Marketing

{ 
   "eventType":"BO.MarketingPermission.Changed",
   "cloudEventsVersion":"0.1",
   "source":"https://S4HANAOD.sap.com",
   "eventID":"+hY+CySbHtm5pLFENTnSjg==",
   "eventTime":"2019-10-02T15:01:41Z",
   "schemaURL":"https://S4HANAOD.sap.com/sap/opu/odata/IWXBE/BROWSER_SRV/",
   "contentType":"application/json",
   "data":{ 
      "KEY":[ 
         { 
            "MARKETINGPERMISSIONUUID":"FA163E0B249B1EE9B6C5E9FDB01EABA0"
         }
      ],
      "CONTACTPERMISSION":"Y"
   }
}

[marcobebway]

@brendandobbs, regarding the event payload coming from marketing, we have some questions:

• Are all remote applications publishing v0.1 cloudevents?
• Will the "eventTypeVersion" be present in the events coming from the remote applications as well? Because it was hardcoded to "v1" here.
• Can we uniquely identify a remote application by the source, eventType and eventTypeVersion?
• What if the publish request reached the MQTT bridge successfully but failed to reach the internal Kyma Event Bus (for example if publish service was not available), Can the MQTT clients at the remote application side handle the retry(s) to publish?

[brendandobbs]

@marcobebway

• Are all remote applications publishing v0.1 cloudevents?

Marketing Cloud and S/4HANA are both using v0.1 at this point in time. I don't know of any other SAP applications using MQTT

• Will the "eventTypeVersion" be present in the events coming from the remote applications as well?

There is no versioning of the event definitions from the source application (Marketing & S4H) so we need to hard code the version in our event bridge.

• Can we uniquely identify a remote application by the source, eventType and eventTypeVersion?

Unfortunately not. Source is a generic URL and doesn't identify the remote system. The only thing I can think of that we can use to unique identify the system sending the events is a unique event endpoint/path (maybe even a service?). Each unique path/endpoint would need to be secured with different credentials to ensure events are being published to the correct channel.

• What if the publish request reached the MQTT bridge successfully but failed to reach the internal Kyma Event Bus (for example if publish service was not available), Can the MQTT clients at the remote application side handle the retry(s) to publish?

I intentionally avoided using a full MQTT broker so that we only can do an ACK to the sending application if the event is able to be successfully published to the Kyma event bus. If the event is not published to the Kyma event bus, no ACK is received and the source system will retry.

[anishj0shi]

@brendandobbs when we say "source":"https://S4HANAOD.sap.com" wouldn't it imply that the event is coming from a unique marketing cloud tenant, coz currently every cloud tenant has it's own ondemand URL (for example: https://my301xxx.s4hana.ondemand.com), please correct me If I'm wrong.

[brendandobbs]

@anishj0shi , I think it could be used for that purpose but unfortunately it's used as a generic reference to the consumer. We could ask Stani to contact the responsible product team but it wouldn't be a short term solution assuming they agreed to making this change.

[brendandobbs]

AsyncAPI spec can be found here https://github.com/SAP/xf-addons/blob/master/bundles/marketing-connector-0.1.0/chart/marketing-connector/templates/api-registration-job-configmap.yaml

[brendandobbs]

The sample event provided above was only containing the event payload. The MQTT package also includes the topic. Example MQTT packet....

{ 
   "cmd":"publish",
   "retain":false,
   "qos":1,
   "dup":false,
   "length":438,
   "topic":"extfactory/BO/MarketingPermission/Changed",
   "payload":{ 
      "type":"Buffer",
      "data":[ 
         .......
      ]
   },
   "messageId":88
}

[stale[bot]]

This issue has been automatically marked as stale due to the lack of recent activity. It will soon be closed if no further activity occurs. Thank you for your contributions.

[stale[bot]]

This issue has been automatically closed due to the lack of recent activity.

[stale[bot]]

This issue has been automatically marked as stale due to the lack of recent activity. It will soon be closed if no further activity occurs. Thank you for your contributions.

[stale[bot]]

This issue has been automatically closed due to the lack of recent activity.