Integrate with Apache EventMesh

[qqeasonchen]

Apache EventMesh[1] can serve as Serverless Eventing infrastructure and also in CNCF Serverles Landscape, it might be a good choice to integrate with ServerlessWorkflow as orchestration engine.

[1]https://github.com/apache/incubator-eventmesh

[tsurdilo]

@qqeasonchen i think this would be definitely cool to have. I am personally new to EventMesh so don't feel comfortable proposing an integration until learn more about it. If you would like please share how you would integrate it with serverless workflow dsl, would help a lot.

[qqeasonchen]

@tsurdilo glad to hear your reply, i want to know if serverlessworkflow runtime can integrate with eventmesh runtime, or serverlessworkflow runtime only can runs itself and eventmesh using serverlessworkflow-java-sdk to communicate with serverlessworkflwo runtime, you can correct me if i am wrong, thanks.

[vongosling]

AFAIK, serverless workflow runtime is being designed and developed by @yzhao244

[mswiderski]

in general there are several serverless workflow runtime implementations in java in the works (https://github.com/kiegroup/kogito-runtimes, https://github.com/automatiko-io and maybe more that I am not aware of) so in theory they could all be integrated though it would have to be known more on what type of integration is expected. I am pretty much on the same page as @tsurdilo on Apache EventMesh so would be good to discuss and see how this could be made into the spec.

Maybe it could be done similar way as we have the AsyncAPI, OpenAPI etc or even better if it could already be integrated via CloudEvents or AsyncAPI... again just thinking out loud.

[qqeasonchen]

EventMesh supports CloudEvents already，and also can extend AsyncAPI too. so if there is no default runtime implementation of serverlessworkflow runtime, we can design it just as @vongosling @yzhao244 mentioned.

[tsurdilo]

@qqeasonchen

i want to know if serverlessworkflow runtime can integrate with eventmesh runtime, or serverlessworkflow runtime only can runs itself and eventmesh using serverlessworkflow-java-sdk to communicate with serverlessworkflwo runtime, you can correct me if i am wrong, thanks.

Our sdks are not runtime implementations, they provide help with parsing and validating the dsl (json/yaml), and ability to create a workflow definition using programming language. Some like java and typescript also allow for svg/png generation to visualize the workflow definition.

[tsurdilo]

@qqeasonchen Wanted to ask, looking at EventMesh docs they mention that orchestration is part of their project. Sorry if I am being ignorant, but wouldnt it make sense for EventMesh to integrate the Serverless Workflow DSL as part of their orchestration language? Imo SW dsl would be perfect for describing the control flow logic of whats in the "rule engine box" here:

[tsurdilo]

@qqeasonchen yeah, looking at this more it seems SW would indeed be a perfect solution for EventMesh as far as using it to define the orchestration, and invocation of "end systems" that are ontop of the image in last post. This is really what SW is about in the end.

As far as integrating of EventMesh inside the SW DSL, do you mean that a workflow could trigger EventMesh orchestration, or is it the other way around? The image in last post is unclear to me as the "Rule Engine" points to EM only.

[ricardozanini]

@qqeasonchen, as @tsurdilo just mentioned, I think EventMesh could be a candidate to implement the spec.

[yzhao244]

@tsurdilo @ricardozanini @qqeasonchen @vongosling I totally agree with @tsurdilo. Serverless Workflow defines the rules and regulations how event/data is orchestrated in a flow. I believe supporting event orchestration is definitely what EventMesh needs. :) As mentioned by @mswiderski , both projects support CloudEvents protocol which makes integration easier lol..

Additionally, SW supports rich rules towards Event related states, definitions and data filters. A quick example about event data filter from SW, I think this definitely is a good use case for EventMesh to support event orchestration when dealing with event data filtering lol.

[tsurdilo]

@qqeasonchen @vongosling et all, maybe it would be best if we discussed this on zoom. Think it would be great for EM and SW to connect and start some discussions around possible integrations. Will try to catch up on EM in the meantime so I'm not a total noob :)

Feel free to propose a good day and time and let's get together

[qqeasonchen]

@tsurdilo sure, we can discuss on zoom.

[qqeasonchen]

@tsurdilo @ricardozanini hey guys, shall we have a meeting schedule?

[tsurdilo]

@qqeasonchen definitely, let us know what works best for you this week

[qqeasonchen]

@tsurdilo how about 2022/01/13 10:00-12:00 (GMT+08:00)?

[ricardozanini]

@qqeasonchen, I'm in GMT-03:00, this time for me is like midnight :( I can't. I guess you're in Asia, right? I think the most reasonable time would be the end of the day for us, the beginning of the morning for you (7~8am)?

[qqeasonchen]

@ricardozanini yes, we are in Asia, and if midnight (9 ~ 12pm) is ok for us, time time for you would be in the morning(9-11am), I do't know if it is suitable for you, because 7 ~ 8am is too early for us, thanks.

[tsurdilo]

@qqeasonchen i am on EST time (GMT-5) but can make time you guys decide. Feel free to send us invite and let's go from there? "tsurdilovic at gmail dot com"

[qqeasonchen]

@qqeasonchen i am on EST time (GMT-5) but can make time you guys decide. Feel free to send us invite and let's go from there? "tsurdilovic at gmail dot com"

sure

[ricardozanini]

@qqeasonchen mine is ricardozanini at gmail dot com :)

[qqeasonchen]

@qqeasonchen mine is ricardozanini at gmail dot com :)

got it~

[jinrongluo]

@tsurdilo We have the design doc about EventMesh and Serverless Workflow integration, including the Spec DSL and SDK.

https://github.com/apache/incubator-eventmesh/blob/master/docs/en/features/eventmesh-workflow-design.md

Please review and provide some comments. Thanks so much.

[tsurdilo]

@jinrongluo very nice! read through it and have a question (also working on a PR for your doc to do some small updates):

The online store is defined by:

  asyncapi: 2.2.0
  info:
    title: Online Store application
    version: '0.1.0'
  channels:
    store/order:
      subscribe:
        operationId: receivedStoreOrder
        message:
          name: StoreOrder

so I think the first workflow state should be "event" state that receives the order event, and not an operation state that send an event to this service.

Have also asked AsyncAPI guys to take a look at provide input, and they seem to be faster than me: https://github.com/apache/incubator-eventmesh/issues/716 :)

[tsurdilo]

@jinrongluo i think the pr they opened is a little bit incorrect, and that you have the asyncapi definitions right...will add comment. (edit..ok they closed it, works as well)

[jinrongluo]

Hi @tsurdilo

- name: Received Store Order
  type: operation
  actions:
    - functionRef:
        refName: receivedStoreOrder

My reasoning of having the operation state as the Start State of the workflow is that, the workflow has to explicitly invoking receivedStoreOrder function in order to receive the StoreOrder events from the channel.

From the Spec, event state is assuming the event is already arrived at the workflow runtime, and the workflow reacts on it. But it didn't say which function to invoke to fetch (or subscribe) such event.

Maybe I am missing something else. Please clarify. Thanks.

[jinrongluo]

@jinrongluo i think the pr they opened is a little bit incorrect, and that you have the asyncapi definitions right...will add comment. (edit..ok they closed it, works as well)

Thanks for bring it up. I was having the same confusion when studying AsycnAPI spec as well. :)

[tsurdilo]

@jinrongluo since you are using asyncapi and event-driven communication, i think you can define your workflow completely using events, see https://github.com/apache/incubator-eventmesh/pull/717

you can link to even source for events your asyncapi definition as well...this is indeed something that we need to document better (and fix some small things), will work on it next week.

[tsurdilo]

@jinrongluo here is wf image in case you want to add to docs as well:

[jinrongluo]

@jinrongluo here is wf image in case you want to add to docs as well:

Thanks. I will add workflow diagram to my doc.

[jinrongluo]

@jinrongluo since you are using asyncapi and event-driven communication, i think you can define your workflow completely using events, see apache/incubator-eventmesh#717

you can link to even source for events your asyncapi definition as well...this is indeed something that we need to document better (and fix some small things), will work on it next week.

Thanks for the PR. This PR gives us much clarity of the event related elements in the workflow spec.

I think the Spec needs to mention that, event.soure not only refers to CloudEvents Source, but also AsyncAPI Operation

https://github.com/serverlessworkflow/specification/blob/main/specification.md#event-definition

events:
  - name: NewOrderEvent
    source: file://onlineStoreApp.yaml#newStoreOrder
    type: asyncapi
    kind: consumed

[tsurdilo]

@jinrongluo sure i will do that @ricardozanini @cdavernas @antmendoza can you guys please take a look at the pr as well (the workflow definition) to make sure we are all on the same page here before I make updates to spec for it. Basically, the idea is and its something we can do already w/o changes in dsl schema, just is not documented, defined full true event-driven communications with our DSL, when using AsyncAPI. The event definitions source can point to an AsyncApi operation which will give the runtime all the info it needs to subscribe to a channel of that operationId to receive events, as well as both the channel as well as the mesage type (payload of the CE) to where to publish an event to. I think this is really powerful feature of our dsl that we are not advertising currently. WDYT?

It would be also possible to use callback state for this use case, but since all services are full event-driven in their example architecture, i think there is no need. If the used services had for example a rest endpoint that receives a new payment, or shipping info, then yes a function for that would be more optimal to be used.

(here is gist with workflow json def for your convenience: https://gist.github.com/tsurdilo/18aea415d023f1ea92f79c28fc14c258)

[tsurdilo]

@jinrongluo can you please check and let me know if it's ok to put EventMesh (logo and text) on our website: https://serverlessworkflow.io/ (see section on bottom). Thanks.

[jinrongluo]

OK I will talk to @qqeasonchen

[qqeasonchen]

@jinrongluo can you please check and let me know if it's ok to put EventMesh (logo and text) on our website: https://serverlessworkflow.io/ (see section on bottom). Thanks.

@tsurdilo Sure ok, Thank You, the community would happy to see that :)

[qqeasonchen]

OK I will talk to @qqeasonchen

@jinrongluo Thank You for reminding me.

[yzhao244]

@jinrongluo can you please check and let me know if it's ok to put EventMesh (logo and text) on our website: https://serverlessworkflow.io/ (see section on bottom). Thanks.

@tsurdilo Sure ok, Thank You, the community would happy to see that :)

@tsurdilo Wow, that's great news. I can't wait to see eventmesh in serverless workflow website. @qqeasonchen Thanks :)

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[qqeasonchen]

Hi,
Do anybody know that the direktiv[1] follow serverlessworkflow spec or not now？

[1]https://github.com/direktiv/direktiv

[tsurdilo]

@qqeasonchen

No, they decided to go their own way and copy our spec and develop it on their own terms. Direktiv is not associated in any shape or form to our project (well except the copy part which imo was super dirty..but it is what it is and an unfortunate part of open source).

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[qqeasonchen]

Apache EventMesh has already realized a version: https://github.com/apache/incubator-eventmesh/tree/master/eventmesh-workflow-go