Present OpenMessaging to TOC

[vongosling]

OpenMessaging, a vendor-neutral open standard for distributed messaging and streaming, which includes the establishment of industry guidelines and messaging, streaming specifications to provide a common framework for finance, e-commerce, IoT and big-data area. The design principles are the cloud-oriented, simplicity, flexibility, and language independent in distributed heterogeneous environments. Conformance to these specifications will make it possible to develop a heterogeneous messaging applications across all major platforms and operating systems.

GitHub: https://github.com/openmessaging Website: http://openmessaging.cloud/ License: Apache 2.0

Alignment to CNCF:

Makes OpenMessaging to be a spec. in messaging and streaming field, making other spec. such as CloudEvents more widely adopted in messaging.

Connects different cloud platform (Alibaba Cloud, Azure, Google, and AWS) through OpenMessaging, users are free to choose according to OpenMessaging Benchmark, like DB-Engines in No-SQL field.

The broader Messaging community is actively exploring how to collaborate and improve the application developer experience, creating cutting-edge features, making messaging runtime schedule and deployment on Kubernates is a natural evolution of these efforts.

Consider OpenMessaging is a Linux Foundation Project, our founding members are including Alibaba Cloud (https://www.alibabacloud.com) Yahoo (https://www.yahoo.com/) DIDi Chuxing (http://www.didichuxing.com/) Streamlio (https://streaml.io/)

What do we want to get out of submitting to the CNCF?

Expanding the contributor base (we hope all existing messaging solutions could be involved in, not only including Apache RocketMQ, RabbitMQ, Apache Pulsar incubating). We'd love to get others to contribute.

Expanding the ecosystem of OpenMessaging to integrate with CNCF ecosystem projects, improving and standardizing the messaging and streaming field as a whole.

I've spoken with Dan Kohn, Chris Aniszczyk, Alexis Richardson and Keith, shall we dear TOC members schedule a presentation for OpenMessaging on May 15th, Thanks for any information you can provide!

[caniszczyk]

Thanks @vongosling, I'll check with the @cncf/toc but May 15th SGTM.

[vongosling]

Hi @caniszczyk. I am so sorry to delay the OpenMessaging report, Could we schedule it at Sep 4th :-)

[caniszczyk]

done

On Fri, Jun 29, 2018 at 7:33 AM, von gosling notifications@github.com wrote:

Hi @caniszczyk https://github.com/caniszczyk. I am so sorry to delay the OpenMessaging report, Could we schedule it at Sep 4th :-)

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/cncf/toc/issues/103#issuecomment-401340630, or mute the thread https://github.com/notifications/unsubscribe-auth/AAD5ITaQNk3K78vblCSNY2kOeDqKBh-Lks5uBh6FgaJpZM4S8oB9 .

-- Cheers,

Chris Aniszczyk http://aniszczyk.org +1 512 961 6719

[vongosling]

thanks @caniszczyk :-)

[monadic]

This is a great start. I would really like to see feedback from messaging implementations, if I am to support this wholeheartedly. eg Nats? @ColinSullivan1

[ColinSullivan1]

@monadic, sure. I'd like to dig into this further, and will comment in the next few days.

[qqeasonchen]

Webank uses different banking architectures, from distributed architecture to open source technologies, and fully utilizes on the benefits of messaging by implementing various messaging techniques in different scenarios, such as message exchanges and pub/sub models. We believe OpenMessaging can address our current challenges, and as a member of OpenMessaging, we hope more and more corporations can give their efforts on it.

[sijie]

This is Sijie from Apache Pulsar community. I think it is a good idea to have a open messaging standard for different messaging implementations. That would provides great interoperability between different messaging systems. It would be great to see OpenMessaging can grow into a vendor neutral standard with more and more organization involving in it.

[Jason918]

As a member of OpenMessaging, Didi Chuxing has its own messaging implementation with proprietary protocol. We strongly believe the benefits that OpenMessaging can bring to us. With the release of OpenMessaging 1.0, we are actively developing our implementation and plan to open source our implementation by the end of this year. We hope OpenMessaing will become better and better under CNCF. @vongosling @monadic

[vongosling]

@monadic I am really excited to hear from your feedback :-) Also thanks @Jason918

[ColinSullivan1]

I’ve taken a look through... What comes to mind with the OpenMessaging specification is JMS (Java Message Service). Some of us on the NATS team were involved in JMS in the early days, and we’re well versed in the benefits and drawbacks of an API specification over messaging.

Messaging has become somewhat commoditized, where standard patterns across various vendors and projects are supported, namely request/reply and publish/subscribe, and queue based semantics. The OpenMessaging specification certainly captures these patterns.

That being said, I think there are similar benefits and drawbacks to that of JMS. Benefits include avoiding vendor lock-in (as described) and providing a well defined singular API for developers to use. The drawbacks include managing the complexities involved in implementing these patterns in an API to be used over multiple messaging solutions, and generally, managing complexities found in solving non-trivial distributed problems. Implementers of a messaging API specification will invariably have slight differences in behavior left undefined, which is frustrating to users.

Concerns:

• The API has a similar complexity to JMS, possibly more so, and I can personally tell you it was difficult to get 100% compliance. JMS was java specific - multiple languages will multiply the level of complexity - you will have different APIs idiomatic to various languages, which could affect implementation.
• A common API will obscure features that are value propositions of different messaging systems, e.g. MQTT's last will and testament. There was an attempt in JMS to work around this with optional header extensions, but there’s no sure-fire way to address this problem.
• A specification to this level of detail would constrain implementations from other messaging solutions - e.g. gRPC, zeromq, NanoMsg, Redis Pub/Sub, etc.

I would suggest starting simple for CNCF. Break the specification apart into sections with a mandatory basic specification, and optional parts that are more complex. Starting with simple API that does a few things like connect, subscribe/receive, publish, and disconnect will get users on board and require less investment to become compliant, driving additional implementations. Optional, more complex portions of the specification can be met later. This will allow implementers to be compliant with portions of the specification, versus an all-or-nothing approach (which I think was one of the things that prevented more implementations of JMS). A payload/header specification isn’t necessary - other projects such as CloudEvents will define that.

I think it'd help to provide more detail about what design principles in the OpenMessaging project align with CNCF, and frame this in a cloud native perspective. For example, how multiple streaming consumers might be used with autoscaling. It would also be highly beneficial to get input from maintainers/developers of other messaging solutions.

Clearly a lot of work has gone into this, and I can appreciate the effort behind creating an API to cover multiple messaging solutions - this is not an easy task.

CC @monadic

[duhengforever]

@ColinSullivan1 @monadic Thanks for your review, as you pointed out, API is not the focus of our work, we are more concerned with the delivery of messages and their data exchange model, metadata，spec etc., just like cloudevents, openMetrics. but cloudevents more concerned about the event，and OpenMessaging is more concerned about messaging. As shown in our landscape, Streaming, Connector will be included in the future，which is something that cloudevents can't solve. But the existing system can't be ignored, there are many opportunities here, so we need a bridging technology to carry. Recently, we also found a fact that the above two specs have also started to ask the SDK in the community.

[sijie]

I would suggest starting simple for CNCF. Break the specification apart into sections with a mandatory basic specification, and optional parts that are more complex. Starting with simple API that does a few things like connect, subscribe/receive, publish, and disconnect will get users on board and require less investment to become compliant, driving additional implementations. Optional, more complex portions of the specification can be met later.

+1 I strongly believe open messaging should start with simple/basic api. so different messaging implementations can easily adopt the api.

For example, how multiple streaming consumers might be used with autoscaling. It would also be highly beneficial to get input from maintainers/developers of other messaging solutions.

in Apache Pulsar, we have a concept called shared subscription, which a group of consumers consume one (or a set of) topic(s) competingly. It is a stateless consuming mode. We are leveraging the autoscaling capability provided by k8s. The consuming mode has been using in pulsar io connector framework to autoscale pulsar io connectors as well. Hope this provide some insights from Apache Pulsar perspective, how a messaging system is related to cloud native infrastructure.

[duhengforever]

@ColinSullivan1 Yes, OpenMessaging only provides a few simple core APIs at the Interface level, rather than making this part very heavy. One is to ensure the adaptation of the existing system, and the second is to ensure the ease of use and acceptance of new users. In addition, we sincerely hope that NATS can join OpenMessaging and promote the development of the messaging field.

[vongosling]

I would suggest starting simple for CNCF. Break the specification apart into sections with a mandatory basic specification, and optional parts that are more complex. Starting with simple API that does a few things like connect, subscribe/receive, publish, and disconnect will get users on board and require less investment to become compliant, driving additional implementations. Optional, more complex portions of the specification can be met later. This will allow implementers to be compliant with portions of the specification, versus an all-or-nothing approach (which I think was one of the things that prevented more implementations of JMS). A payload/header specification isn’t necessary - other projects such as CloudEvents will define that.

Totally agree, actually, before we start the project, we dicussed with many users of messaging, and then scoped the required an optinal parts, thanks the community and @monadic suggestion, I am eagerly hope to work together with more messaging experts @ColinSullivan1 come together, make openmessaging grow and prosper under cncf vision of the pen source without vendor lock-in :-)

[duhengforever]

@ColinSullivan1 Although the focus of our work will be on the continuous improvement of the specification and the integration with the surrounding ecology, such as CloudEvents, OpenTracing, OpenMetrics etc.. Actually，as long as the definition of the data exchange specification is clear and concise enough, any implementation of this standard will be very simple, which is one of our principles.

But as I mentioned in the above reply, OpenMessaging still provides some simple interfaces, in the current version of OpenMessaging, the main work we did was to streamline the API and eliminate non-core redundant concepts, so we only defined the message create、send(Producer), receive(Consumer) interface and defined the unified access method to cloud vendors. On the one hand, it is to solve the problem of existing user migration, and on the other hand is to avoid the problems you mentioned, and in the next release version of OpenMessaging , we will continue to refine the Specification, improve the community ecology, and provide some minimal Connector and Streaming interfaces，and will strive to get more support from cloud vendors.

We firmly believe that this matter is meaningful, otherwise, there will be no standards such as JMS, MQTT, AMQP, CORBA, etc. but we also know that the difficulties of the matter, but I think if there are more messaging experts support, this thing, we can do it. Because in this day, unlike in the past, cloud computing is much more pervasive, and the messaging system has become the infrastructure, or even the next database. Hopefully the NATS experts can help us, to achieve this goal together.

[vongosling]

Let's stay in contact to push the forward.

[ColinSullivan1]

@vongosling , I'm happy to assist where I can.

[mattrpav]

Hello-

I'm Matt Pavlovich, I am currently working as CTO of HYTE Technologies. We offer a hybrid messaging platform. I've worked with messaging systems across all vendors and protocols for over 15 years and have founded and participated in open source projects for over 20.

I agree there is a need for a modern, cross-platform, cross-language messaging standard, and am happy to share and help out where I can. I'm probably behind most here in reviewing the current API draft and will make a point to put together some thoughts over the next couple weeks.

Thanks

[vongosling]

@mattrpav Thanks for your attention. Looking forward to your thoughts :-)

[amye]

Checking in here, are we still interested in contributing this to the CNCF?

[duhenglucky]

@amye thanks for your reminder, we are still very interested in entering CNCF. OpenMessaging has now received support from more than 30 cloud vendors and open-source vendors. Many cloud vendors including Alibaba Financial Cloud have begun to use OpenMessaging in their commercial messaging products. Active discussions are ongoing with several other cloud vendors to advance OpenMessaging to be a de-facto standard.

[brendandburns]

I think the right next step is a review with SIG-Runtime

[raravena80]

@brendandburns we (SIG-Runtime) briefly discussed this and believe that it's in SIG-Network's scope. (gRPC, NATS, CNI, etc). Thx.

[amye]

Marking this as closed with the new sandbox inclusion process.