search

apache / eventmesh

EventMesh is a new generation serverless event middleware for building distributed event-driven applications.
https://eventmesh.apache.org/
Apache License 2.0
1.62k stars 642 forks source link

[Doc] EventMesh application scenario case collection #5022

Open xwm1992 opened 5 months ago

xwm1992 commented 5 months ago

Search before asking

Documentation Related

This issue is used to record and summarize the application scenarios of EventMesh. Issues can be displayed according to the following parts(including but not limited to):

Are you willing to submit PR?

Code of Conduct

xwm1992 commented 5 months ago
KrispauI commented 2 months ago

  Data Synchronization

  Large enterprises typically have central data management platforms that connect multiple business systems. These business systems may perform asynchronous data changes, additions, and deletions, making data synchronization necessary. Traditional methods of data synchronization often rely on message queues or active polling, both of which have limitations: message queues struggle with separating data between different business units, leading to data management challenges; polling impacts the performance of the central data management platform and increases system coupling, complicating system upgrades and maintenance.

  Therefore, EventMesh is introduced to handle user data synchronization based on the Event-Driven Architecture (EDA). The data management center only needs to publish data change information to different event sources. Each business system can then configure event targets and subscribe to the relevant event sources to achieve data synchronization. During the subscription process, business systems can bind transformation rules (for data splitting and desensitization) and filtering rules to decouple the system architecture.

  In the data synchronization scenario, Geely Auto implemented an event integration platform based on EventMesh. For example, in Geely's user center system, most internal business systems cache user data locally after integrating with the user center. As these business systems frequently modify user data asynchronously, there is a strong demand for data synchronization.

  Geely Auto's event integration platform, built on EventMesh, enables the user center to publish new and changed user data as "events" within an event-driven architecture. Business systems subscribe to event sources to retrieve necessary user data updates. For instance, the finance system may only be interested in payment information updates, while the customer service system cares about contact information changes.

  To address the unique data needs of different business systems, the platform utilizes EventMesh's filter/transformer capabilities. Event sources are configured with transformation and filtering rules, enabling business systems to subscribe and process user data splitting and desensitization. This approach decouples direct interactions between the business systems and the user center while reducing the performance load on the user center.

  To ensure high performance and availability, Geely Auto's event integration platform is deployed across multiple regions and clusters. Different business systems can access and synchronize data across these clusters, minimizing data synchronization delays and maintaining system stability even under high concurrency conditions.

KrispauI commented 2 months ago

  The SaaS composite application integration standardization

  A SaaS composite application consists of multiple PBCs (Package Business Capabilities). A PBC can be defined as a SaaS application module with clearly defined business capabilities. Each module is business-driven and capable of independently fulfilling business requirements without external dependencies. Enterprise solutions are typically composed of multiple SaaS application modules, integrating various complex functionalities to present a unified and comprehensive user experience.

  The diagram below illustrates a single PBC, a combination of PBCs, and multiple composite PBCs:

  From the above architecture, it is clear that each SaaS application module (PBC) has a low degree of coupling. Modifying or adjusting a specific module does not affect the operation of other existing modules.This facilitates agile development and efficient iterative updates. However, one challenge of SaaS composite applications is the lack of standardized integration between different applications, as the absence of a unified communication protocol can hinder the implementation of this architecture.

  This issue can be solved by EventMesh. EventMesh integrates TCP and HTTP protocols, and supports bi-directional asynchronous communication between Client and Server through gRPC (Google’s open-source high-performance RPC framework based on HTTP/2) with SDKs available for multiple languages such as Java, Python, Go. Users do not need to worry about which communication protocol is used in different scenarios when using the SDK; the event-driven asynchronous communication can be achieved through the SDK APIs integrated by EventMesh, enabling the seamless event flow between different SaaS application modules.

  Regarding the specific implementation of the scenario, EventMesh officially introduced the gRPC framework starting from version v1.4.0. gRPC defines API interface data models using Protobuf. In the gRPC Protobuf event model, each event is represented by the SimpleMessage data model, with the event content stored in the content field.

  The gRPC event scenarios supported by EventMesh include: event sending and batch event sending, event broadcasting, event request and response, event subscription, and event pushing (for more details, see: eventmesh-client.proto).

  1. The event publishing service provides the following interfaces:

service PublisherService {

   rpc publish(SimpleMessage) returns (Response);

   rpc requestReply(SimpleMessage) returns (SimpleMessage);

   rpc batchPublish(BatchMessage) returns (Response);
}

  Events are presented in the SimpleMessage data model. Event publishing supports three modes: synchronous publishing, asynchronous publishing, and batch publishing.

  Synchronous publishing means the event producer sends the event to EventMesh, waits for the event to be successfully delivered to the event consumer, and receives a response from the event consumer. The end-to-end event publishing process is only considered complete once this is done.

  Asynchronous publishing means the event producer sends the event to EventMesh without waiting for the event to be successfully delivered to the event consumer.

  Batch publishing refers to asynchronously sending a batch of events to EventMesh.

  2. The event subscription service provides the following interfaces:

service ConsumerService {

   rpc subscribe(Subscription) returns (Response);

   rpc subscribeStream(stream Subscription) returns (stream SimpleMessage);

   rpc unsubscribe(Subscription) returns (Response);
}

  Event subscription supports two modes: cluster and broadcast. In cluster mode, only one instance within the event consumer cluster will consume the event. In broadcast mode, every instance in the cluster will consume the event.

  These subscription modes are defined in the subscription data model. Additionally, the subscription service provides two subscription interfaces: the subscribe API and the subscribeStream API. The subscribe API pushes events to consumers via a URL, also known as a webhook. This scenario is suitable for cloud-native microservices, custom applications, and functions. The subscribeStream API pushes events to consumers using gRPC bidirectional streaming, allowing the event consumer to return an acknowledgment (Ack) to the event producer. This supports the producer's need for synchronous event publishing using RequestReply.

  3. Event subscription service provides the following interfaces:

  To improve the performance of event production and consumption, the EventMesh server (EventMesh Runtime) defines thread pools within the gRPC service. Moreover, independent parameters are configured based on different performance requirements for event production and consumption. These parameters can be found in the EventMesh configuration file (eventmesh.properties).

  For example, the following are the number of threads for event production, subscription, and pushing:

eventMesh.server.sendmsg.threads.num=50
eventMesh.server.clientmanage.threads.num=30
eventMesh.server.pushmsg.threads.num=50

  When the gRPC service starts, it begins listening for client requests. Once a new request arrives, it is dispatched to the thread pool of the corresponding service, and the appropriate processor (Processor) handles it. This prevents blocking the handling of subsequent requests, thereby improving concurrency.

public void publish(SimpleMessage request, StreamObserver<Response> responseObserver){
    cmdLogger.info("cmd={}|{}|client2eventMesh|from={}|to={}", "AsyncPublish",
        EventMeshConstants.PROTOCOL_GRPC, request.getHeader().getIp(),
        eventMeshGrpcServer.getEventMeshGrpcConfiguration().eventMeshIp);

    EventEmitter<Response> emitter = new EventEmitter<>(responseObserver);

    threadPoolExecutor.submit(() -> {
        SendAsyncMessageProcessor sendAsyncMessageProcessor = new SendAsyncMessageProcessor(eventMeshGrpcServer);
        try {
            sendAsyncMessageProcessor.process(request, emitter);
        } catch (Exception e) {
            logger.error("Error code {}, error message {}", StatusCode.EVENTMESH_SEND_ASYNC_MSG_ERR.getRetCode(),
                StatusCode.EVENTMESH_SEND_ASYNC_MSG_ERR.getErrMsg(), e);
            ServiceUtils.sendRespAndDone(StatusCode.EVENTMESH_SEND_ASYNC_MSG_ERR, e.getMessage(), emitter);
        }
    });
}

  For instance, the above code is the implementation of the event publishing service. It uses a threadPoolExecutor to send events to the thread pool for downstream SendAsyncMessageProcessor handling.

KrispauI commented 2 months ago

  Serverless Event Bus for Cloud Services

  Cloud service providers can build a Serverless Event Bus using EventMesh to facilitate and manage events across various sources, including traditional applications, cloud-native services, and SaaS partner applications. In this scenario, the Serverless Event Bus powered by EventMesh helps developers create loosely coupled and distributed event-driven serverless microservice applications without the need to manage underlying infrastructure.

  This Serverless Event Bus supports asynchronous event communication across different cloud environments. It simplifies protocol management by routing events between services, making it suitable for multi-cloud, hybrid cloud architectures, and third-party SaaS systems. Additionally, the bus can be deployed in distributed environments, ensuring reliable event delivery even under heavy load, thanks to EventMesh’s cross-region failover and load-balancing capabilities.

  Cloud service provider Huawei Cloud offers a Serverless Event Bus service called EventGrid, which is based on EventMesh. EventGrid uses EventMesh as its runtime engine and supports the integration of Huawei Cloud services, custom applications, and SaaS applications in a standardized, centralized manner. By utilizing the standardized CloudEvents protocol, it flexibly routes events between applications, building a loosely coupled, distributed event-driven architecture. This not only creates more application scenarios for Huawei Cloud but also further enriches the developer ecosystem.

  As a crucial part of the event-driven architecture in a Serverless environment, it provides elastic, asynchronous, and decentralized event governance services. Key functions include event aggregation, schema validation, filtering, routing, transformation, and pushing. Additionally, it offers enhanced features like fault-tolerant retries, dead-letter storage, event query tracking, and event workflows.

  The diagram above represents the overall architecture of Huawei Cloud EventGrid. EventGrid connects to various cloud services as event sources, including Distributed Message Service, Object Storage Service (OBS), and Distributed Cache Service. The various events generated by these event sources are pushed to the event channels of the EventGrid Event Bus (Bus Runtime). The Event Bus configures event channels for EventGrid users on a per-tenant basis, allowing multiple event channels under a single tenant to carry events from different sources.

  EventGrid, as the standard event hub for Huawei Cloud, enables interconnection between various cloud services. Cloud services, acting as event sources or event targets, are deployed on Huawei Cloud's Serverless application platform. Applications push real-time business events to the event grid, where the event grid filters, routes, and transforms the events to trigger cloud services subscribed to those events. EventGrid currently supports over 100+ built-in Huawei Cloud event sources and allows for custom and partner event extensions. Additionally, it is supported by a vast array of official data sources, covering databases, messaging, serverless computing, big data, DevOps platforms, IoT, and more, with custom event integration capabilities.

KrispauI commented 2 months ago

  Serverless Solution for Connected Vehicle Services

  In the connected vehicle scenario, the huge number of vehicles and the variety of models from different automakers generate massive amounts of data. A single vehicle can have hundreds of sensors continuously producing data, and during peak hours (such as morning and evening rush hours), the volume of connected vehicle data can surge dramatically. Additionally, the automotive industry lacks standardized software development kits for connected vehicles to reduce development costs and shorten time-to-market. This creates a broad demand for Serverless solutions tailored to connected vehicle services.

  The Serverless solution for connected vehicle services can use EventMesh as an event bus and event center. The vast amounts of data from vehicle sensors, GPS devices, and cloud platforms can be ingested via API gateways or IoT devices. Once processed by the event-driven architecture, the solution provides real-time responses and scales smoothly to handle increased traffic during peak times(such as morning and evening rush hours).

  The Serverless solution for connected vehicle services, powered by EventMesh, offers dynamic scaling and high flexibility in an event-driven architecture. It meets the demand for high concurrency and rapid elasticity.It can scale dynamically during traffic surges to handle highly concurrent data. EventMesh also provides flexible routing capabilities, allowing events to be routed to different systems and services based on customizable configuration rules.

  The diagram above illustrates the Serverless solution for connected vehicle services. First, data can be cleaned, filtered using the filter and transformer capabilities of EventMesh,then data is processed via function computation.After processing, the data is stored in distributed object storage, and events are generated. EventMesh routes these events to the appropriate services based on predefined configuration rules. These services include peak-shaving and valley-filling consumption services, AI-based intelligent recommendation services, and multiple departmental business services. This architecture allows each service to respond only to relevant events without coupling to the data source directly.

  In addition, this solution offers features such as automatic scaling, fault tolerance with retry mechanisms, and cross-regional support, ensuring reliable event delivery even under high-load conditions. With EventMesh, external Webhooks can also be integrated, allowing events to be pushed to other systems or third-party services, facilitating future service expansion.