Messaging patterns - Githubissues

archenroot commented 6 years ago

Hi guys,

I like a lot this initiative, but I am missing something common in advanced messaging schemas, some common meta headers and messaging patterns. Sorry if I just overlooked it while browsing in javadocs.

There exists already quite robust definitions of messaging patterns, especially for event-driven system or asynchronous communication there are some common message headers to be used like message Id, correlation Id, global transaction id, etc. I saw this framework only introduces generic key/value store like system and user headers within message.

Question: So this framework is not going to introduce some base messaging schema? It more focuses on high level fundamentals and leave the user to do what he wants? I think there could be added in time some kind of proposal for messaging patterns as low level implementation, what do you think?

We for example go to adopt IEC 61968-100 standard which target all possible situations on low level. Or OpenMessaging is more about foundation to be used to implement something like IEC messaging standard?

Thank you for answers.

Ladislav

vongosling commented 6 years ago

Question: So this framework is not going to introduce some base messaging schema? How do you think about messaging schema? DId you notice our proposal for OpenMessaging https://github.com/openmessaging/specification.

As you mentioned low level and high-level problem. OpenMessaging has many APIs other than other already existing protocol. IMO, we could use OpenMessaging as our messaging programming model and use the MQTT or AMQP as our inner low-level wire protocol.

As for correlation Id, global transaction id and other optional heads, we just learn more about HTTP2 protocol redefinition, we hope to remove some unnecessary or less-useful feature at the first. If you have any suggestion about here, please let us know :-)

archenroot commented 6 years ago

@vongosling - I actually so far like OpenMessaging rise, but must study it in more detail and understand what it actually targets, I will be posting some of my observations later.

In general there were multiple stages which I went trough with Oracle/IBM or Apache Camel/ServiceMix based data integrations (async usually over JMS) in the past for example, while I left this domain and focus on microservices usually based on Kafka messaging:

Define messaging patterns, and even if you focus on so called "Message Contstruct" domain only which doesn't cover the integration flow itself, this is something which is here for ages and is really robust and works. I am just asking myself what do you bring so new and revolutionary when comparing to http://www.enterpriseintegrationpatterns.com/patterns/messaging/ which is the wheel I use for ages. But not only it defined objects, but wide usage scenarios of how messaging can be used/implemented.
For each pattern I am interested in, I see in your case clearly Request with async/dealyed Response or Event messages, etc. there could be more. But for each I usually define some kind of template which already includes metadata each message must transport, these are my metadata(correlationid, etc), if the standard for messaging doesn't define something like correlation id or global transaction id, than it doesn't define messaging for me, I have found in specification only Key/Value store, so you leave this area completely open (Correct me if I overlooked something and you define REAL FIELDS). While I like it to have it open, I think you/we should introduce also real instance of these message metadata as kind of example or generic instance.
Another step in evolution was so far usually building on these templates a concrete domain focused business message registry, These were already real business constructs to be only reused when you needed specific task to do: CreateInvoiceMessage CancelInvoiceMessage UpdateInvoiceMessage etc...
And last task after you are able to establish this model fully adopted in your specific area is actually to provide full implementation of the flow, so when you go, you just fill your message and it is processed without any need of integration, this has been materialized as Oracle AIA. It had its problems especially it was deadly easy to adopt, but required about 3x more time if you had to extend/change some schemas for any kind of reasons

So I still need understand what from these evolutionary steps is OpenMessaging bringing into place, and also if you bring full cover or you just focus on async as you mentioned AMQP and MQTT, we still have here REST api over HTTP while I hope it will get abandoned during my life within data integration as I prefer only async systems (and I know that http2 does a lot on async field as well, but second argument to not use it is simply the latency overhead to establish HTTP session...), but HTTP or notHTTP is different discussion.

Best regards,

Ladislav

vongosling commented 6 years ago

@archenroot I really understand your question about OpenMesssaging. IMO, OpenMessaging is not entirely an API Specs for your mentioned Construct Domain, we also cover the router, streaming computation, connector, benchmark, metrics and so on, It is even like an Ecosystem around the messaging. Until now, we are trying our best to polish the core of the OpenMessaging, we hope it lightweight, scalable and full-featured(but non-optional) in the messaging field. Base on it, we could further to integrate with other existing wire-level protocols(AMQP, MQTT, WebSocket, HTTP2), of Course, including serverless. As you mentioned the message pattern, I also use it in the past ten years, It is really a good practice in the Enterprise Integration with Messaging solution. Also, if you have any opinion about here, we could discuss more deeply :-)

zhouxinyu commented 6 years ago

Hi @archenroot The correlation id seems to make sense, but I couldn't image the specific scenario, could you please introduce the case or scenario you encountered in a real project?

vongosling commented 6 years ago

@archenroot Recently, some guys get to me to hope to let gRPC have a messaging async capability. IMO, it is a good start to re-add the correlation id, thanks for your suggestion.

archenroot commented 6 years ago

@vongosling - thank you for your answer, I missed it back in time, so response with delay. Otherwise this is really great topic and I thought that Open Messaging is more than just message patterns, thanks for confirmation.

Here is sample message header I used in SOA, but also in emerging microservices architectures. The message headers should be mostly generic and optional and they can mixture both technical (prefered from my perspective) but with business (optional) metadata. Sample of header (it is not full scenario), while it is in XSD form there is no problem to convert it into Json of course on the fly:

<?xml version="1.0" encoding="UTF-8"?>
<xs:schema 
    targetNamespace="http://integration.coreso.eu/IntegrationObjectLibrary/Core/Common/v1"
    xmlns="http://integration.coreso.eu/IntegrationObjectLibrary/Core/Common/v1"
    xmlns:custom="http://integration.coreso.eu/IntegrationObjectLibrary/Core/Custom/v1"
    xmlns:xs="http://www.w3.org/2001/XMLSchema"
    xmlns:vc="http://www.w3.org/2007/XMLSchema-versioning" vc:minVersion="1.0" vc:maxVersion="1.1"
    elementFormDefault="qualified"
    attributeFormDefault="unqualified">
    <xs:include schemaLocation="Components.xsd"/>
    <xs:import namespace="http://integration.coreso.eu/IntegrationObjectLibrary/Core/Custom/v1" schemaLocation="../../Custom/v1/CustomMeta.xsd"/>

    <xs:annotation>
        <xs:documentation xml:lang="en">
            Version 1.0.0
            This schema defines the standard metadata that should accompany messages flowing through 
            the integration layer. It defines a standard canonical message  header and Error types.

            Backwards-compatible modifications should increment the minor version number.
            Backwards-incompatible modifications must increment the major version number and namespace.
        </xs:documentation>
    </xs:annotation>

    <xs:complexType name="CommonMessageHeaderType">
        <xs:annotation>
            <xs:documentation xml:lang="en">
                Default integration message header object.
            </xs:documentation>
        </xs:annotation>
        <xs:sequence>
            <xs:element ref="MessageIdentification"/>
            <xs:element ref="Sender"/>
            <xs:element ref="Target"/>
            <xs:element ref="Sequencing" minOccurs="0"/>
            <xs:element ref="Properties" minOccurs="0"/>
            <xs:element ref="DomainValueMap" minOccurs="0"/>
            <xs:element ref="Messaging" minOccurs="0"/>
            <xs:element ref="Extension" minOccurs="0"/>
           </xs:sequence>
        <xs:attribute name="version" use="optional">
            <xs:annotation>
                <xs:documentation xml:lang="en">
                    Version number of the message header. Must match  the version number 
                    defined in the namespace.
                </xs:documentation>
            </xs:annotation>
        </xs:attribute>
    </xs:complexType>
    <xs:element name="CommonMessageHeader" type="CommonMessageHeaderType"/>

    <xs:complexType name="PropertyType">
        <xs:annotation>
            <xs:documentation xml:lang="en">
                A standard key-value pair object.
            </xs:documentation>
        </xs:annotation>
        <xs:simpleContent>
            <xs:extension base="xs:string">
                <xs:attribute name="name" type="StringType" use="required"/>
                <xs:attribute name="dataType" type="StringType"/>
            </xs:extension>
        </xs:simpleContent>
    </xs:complexType>
    <xs:element name="Property" type="PropertyType"/>

    <xs:complexType name="PropertiesType">
        <xs:annotation>
            <xs:documentation xml:lang="en">
                Collection of key-value objects.
            </xs:documentation>
        </xs:annotation>
        <xs:sequence>
            <xs:element ref="Property" minOccurs="0" maxOccurs="unbounded"/>
        </xs:sequence>
    </xs:complexType>

    <xs:element name="Properties" type="PropertiesType">
        <xs:annotation>
            <xs:documentation xml:lang="en">
                A structure for providing optional extra metadata about the message itself.
            </xs:documentation>
        </xs:annotation>
    </xs:element>

    <!---->
    <xs:complexType name="ObjectCrossReferenceType">
        <xs:sequence>
            <xs:element ref="SenderObjectIdentification" minOccurs="0"/>
            <xs:element name="BusinessObjectDocumentId" type="IdentifierType" minOccurs="0"/>
        </xs:sequence>
    </xs:complexType>
    <xs:element name="ObjectCrossReference" type="ObjectCrossReferenceType"/>
    <xs:element name="SenderObjectIdentification" type="IdentificationType"/>
    <xs:complexType name="ApplicationType">
        <xs:sequence>
            <xs:element name="Id" type="IdentifierType" minOccurs="0"/>
            <xs:element name="Version" type="NumericType" minOccurs="0"/>
        </xs:sequence>
    </xs:complexType>

    <xs:element name="Application" type="ApplicationType"/>

    <xs:complexType name="SenderType">
        <xs:sequence>
            <xs:element name="SystemId" type="IdentifierType" minOccurs="0">
                <xs:annotation>
                    <xs:documentation xml:lang="en">
                        It containst sender system's unique identification code. It is mandatory field.
                    </xs:documentation>
                </xs:annotation>
            </xs:element>
            <xs:element name="Description" type="TextType" minOccurs="0" maxOccurs="unbounded">
                <xs:annotation>
                    <xs:documentation xml:lang="en">
                        Description about the sender system.
                    </xs:documentation>
                </xs:annotation>
            </xs:element>
            <xs:element name="BusinessUnit" type="StringType">
                <xs:annotation>
                    <xs:documentation>
                        Company business unit.
                    </xs:documentation>
                </xs:annotation>
            </xs:element>
            <xs:element name="Domain" type="StringType" minOccurs="0">
                <xs:annotation>
                    <xs:documentation>
                        Logical or business domain of the source application.
                    </xs:documentation>
                </xs:annotation>
            </xs:element>
            <xs:element name="Subdomain" type="StringType" minOccurs="0">
                <xs:annotation>
                    <xs:documentation>
                        Subdomain if applicable.
                    </xs:documentation>
                </xs:annotation>
            </xs:element>
            <xs:element name="IpAddress" type="UriType" minOccurs="0">
                <xs:annotation>
                    <xs:documentation xml:lang="en">
                        This element contains the IP address of the sender system.
                    </xs:documentation>
                </xs:annotation>
            </xs:element>
            <xs:element name="SenderMessageID" type="IdentifierType" minOccurs="0">
                <xs:annotation>
                    <xs:documentation xml:lang="en">
                        This ID can be used to identify the unique message sent by sender system.
                    </xs:documentation>
                </xs:annotation>
            </xs:element>
            <xs:element name="TransactionCode" type="CodeType" minOccurs="0">
                <xs:annotation>
                    <xs:documentation xml:lang="en">
                        This is task identification code. This element should be used to generate message by the sender system.
                    </xs:documentation>
                </xs:annotation>
            </xs:element>
            <xs:element name="CallingServiceName" type="NameType" minOccurs="0" maxOccurs="unbounded">
                <xs:annotation>
                    <xs:documentation xml:lang="en">
                        Name of the application business component service.
                    </xs:documentation>
                </xs:annotation>
            </xs:element>
            <xs:element ref="Application" minOccurs="0">
                <xs:annotation>
                    <xs:documentation xml:lang="en">
                        This element should have information about the originating application.
                    </xs:documentation>
                </xs:annotation>
            </xs:element>
            <xs:element name="ContactName" type="NameType" minOccurs="0" maxOccurs="unbounded">
                <xs:annotation>
                    <xs:documentation xml:lang="en">
                        Sender application's contact name.
                    </xs:documentation>
                </xs:annotation>
            </xs:element>
            <xs:element name="ContactEmail" type="UriType" minOccurs="0">
                <xs:annotation>
                    <xs:documentation xml:lang="en">
                        Sender application's contact email ID.
                    </xs:documentation>
                </xs:annotation>
            </xs:element>
            <xs:element name="ContactPhoneNumber" type="StringType" minOccurs="0">
                <xs:annotation>
                    <xs:documentation xml:lang="en">
                        Sender application's contact phone number.
                    </xs:documentation>
                </xs:annotation>
            </xs:element>
            <xs:element ref="Properties" minOccurs="0" maxOccurs="unbounded">
                <xs:annotation>
                    <xs:documentation xml:lang="en">
                        This element contains more detail about the BMD (Business Message Document)
                    </xs:documentation>
                </xs:annotation>
            </xs:element>
            <xs:element ref="ObjectCrossReference" minOccurs="0" maxOccurs="unbounded">
                <xs:annotation>
                    <xs:documentation xml:lang="en">
                        This element contains an identifier of the sender object and corresponding cross-reference identifier.
                        This element can be used for bulk processing messaging.
                    </xs:documentation>
                </xs:annotation>
            </xs:element>
            <xs:element name="SourceMessageSize">
                <xs:annotation>
                    <xs:documentation xml:lang="en">
                        Size of source message.                    
                    </xs:documentation>
                </xs:annotation>
                <xs:complexType>
                    <xs:simpleContent>
                        <xs:extension base="xs:string">
                            <xs:attribute name="units" type="xs:string" use="optional"
                                default="Bytes"/>
                        </xs:extension>
                    </xs:simpleContent>
                </xs:complexType>
            </xs:element>
            <xs:element name="SourceDataSize">
                <xs:annotation>
                    <xs:documentation xml:lang="en">
                        Size of source data object(s).
                    </xs:documentation>
                </xs:annotation>
                <xs:complexType>
                    <xs:simpleContent>
                        <xs:extension base="xs:string">
                            <xs:attribute name="units" type="xs:string" use="optional"
                                default="Bytes"/>
                        </xs:extension>
                    </xs:simpleContent>
                </xs:complexType>
            </xs:element>
            <xs:element name="Custom" type="custom:CustomSenderType" minOccurs="0"/>
        </xs:sequence>
    </xs:complexType>
    <xs:element name="Sender" type="SenderType">
        <xs:annotation>
            <xs:documentation>
                Mandatory structure for providing information about the originator of the message.
            </xs:documentation>
        </xs:annotation>
    </xs:element>

    <xs:complexType name="TargetType">
        <xs:sequence>
            <xs:element name="Id" type="IdentifierType" minOccurs="0"/>
            <xs:element name="OverrideRoutingIndicator" type="IndicatorType" minOccurs="0"/>
            <xs:element name="PipelineName" type="NameType" minOccurs="0"/>
            <xs:element name="ApplicationTypeCode" type="CodeType" minOccurs="0"/>
            <xs:element name="EndPointUri" type="UriType" minOccurs="0"/>
            <xs:element name="Custom" type="custom:CustomTargetType" minOccurs="0"/>
        </xs:sequence>
    </xs:complexType>
    <xs:element name="Target" type="TargetType"/>

    <xs:complexType name="ExtensionType">
        <xs:sequence>
            <xs:any namespace="##any" processContents="skip" >
                <xs:annotation>
                    <xs:documentation>
                        Optional structure allowing for further instructions to be added in the header.
                        This is edge scenario, usually key-value store should be first instance when extending header.
                    </xs:documentation>
                </xs:annotation>
            </xs:any>
        </xs:sequence>
    </xs:complexType>
    <xs:element name="Extension" type="ExtensionType">
        <xs:annotation>
            <xs:documentation>
                Structure for providing any extension beyond capabilities of key-value store, e.g. Properties element.
            </xs:documentation>
        </xs:annotation>
    </xs:element>

    <xs:complexType name="SequencingType">
        <xs:sequence>
            <xs:element name="RunID" type="xs:string" minOccurs="0">
                <xs:annotation>
                    <xs:documentation>
                        Identifies the batch or run that this message forms a part of.
                    </xs:documentation>
                </xs:annotation>
            </xs:element>
            <xs:element name="MessageNumber" type="xs:int" minOccurs="0">
                <xs:annotation>
                    <xs:documentation>
                        Uniquely identifies and orders this message in a sequence. Message numbers
                        should be generated by the calling application as consecutive ascending integers 
                        starting at 1.
                    </xs:documentation>
                </xs:annotation>
            </xs:element>
            <xs:element name="TotalCount" type="xs:int" minOccurs="0">
                <xs:annotation>
                    <xs:documentation>If the sequence is not initialised by other
                        means, this element should contain the total count of messages in
                        the sequence.
                    </xs:documentation>
                </xs:annotation>
            </xs:element>
        </xs:sequence>
    </xs:complexType>
    <xs:element name="Sequencing" type="SequencingType">
        <xs:annotation>
            <xs:documentation xml:lang="en">
                A structure for providing information pertaining to whether this message forms
                part of a sequence.
            </xs:documentation>
        </xs:annotation>
    </xs:element>

    <xs:complexType name="MessagingType">
        <xs:sequence>
            <xs:element name="Name" type="xs:string" minOccurs="0">
                <xs:annotation>
                    <xs:documentation>The name of the root element or message being
                        carried in the payload.
                    </xs:documentation>
                </xs:annotation>
            </xs:element>
            <xs:element name="Version" type="xs:string" minOccurs="0">
                <xs:annotation>
                    <xs:documentation>
                        Version number (major.minor) of the root element in the payload. 
                        Must match the version given in the namespace of the payload element.
                    </xs:documentation>
                </xs:annotation>
            </xs:element>
            <xs:element name="Timestamp" type="xs:dateTime">
                <xs:annotation>
                    <xs:documentation>Date/time that the message was originally created
                        by the originating application.
                    </xs:documentation>
                </xs:annotation>
            </xs:element>
            <xs:element name="ExpiryTime" type="xs:dateTime" minOccurs="0">
                <xs:annotation>
                    <xs:documentation>Date and time after which the message should be
                        considered as expired and should no longer be processed. If
                        present, should be treated by the receiving application as a
                        mandatory instruction.
                    </xs:documentation>
                </xs:annotation>
            </xs:element>
            <xs:element name="Priority" minOccurs="0" type="PriorityType">
                <xs:annotation>
                    <xs:documentation>
                        Enumeration. Equivalent to JMS Priority property. Designates the importance 
                        of a message. Value is in the range 0-9 where 0 is the lowest priority and 9
                        is the highest, with the range 0-4 as normal priority. Interpreted by the 
                        receiving application as guidance only, and not as a mandatory instruction.
                    </xs:documentation>
                </xs:annotation>

            </xs:element>
        </xs:sequence>
    </xs:complexType>
    <xs:element name="Messaging" type="MessagingType">
        <xs:annotation>
            <xs:documentation>
                Structure for providing information about the message contained in the payload.
            </xs:documentation>
        </xs:annotation>
    </xs:element>

  <!-- to be reviewed as this is for tracking purposes and I have those information in Sender already -->
    <xs:complexType name="IntermediateMessageHopType">
        <xs:annotation>
            <xs:documentation>This component allows for capturing of messaging properties in a multi-hopping messaging.
                The information can be used to support for example, guaranteed delivery and auto recovery/resubmit of Errored messages
            </xs:documentation>
        </xs:annotation>
        <xs:sequence>
            <xs:element name="SenderResourceTypeCode" type="CodeType" minOccurs="0"/>
            <xs:element name="SenderResourceID" type="IdentifierType" minOccurs="0"/>
            <xs:element name="SenderMessageID" type="IdentifierType" minOccurs="0"/>
            <!-- <xs:element name="Custom" type="custom:CustomIntermediateMessageHopType" minOccurs="0"/>  -->
        </xs:sequence>
    </xs:complexType>
    <xs:element name="IntermediateMessageHop" type="IntermediateMessageHopType">
        <xs:annotation>
            <xs:documentation>This component allows for capturing of messaging properties in a multi-hopping messaging.
                The information can be used to support for example, guaranteed delivery and auto recovery/resubmit of Errored messages
            </xs:documentation>
        </xs:annotation>
    </xs:element>

    <xs:complexType name="DataObjectType">
        <xs:annotation>
            <xs:documentation>
                Data object type. Here is defined the data object itself which the message transports.
            </xs:documentation>
        </xs:annotation>
        <xs:attribute name="name" type="xs:string" use="optional"/>
        <xs:attribute name="createdOn" type="xs:dateTime" use="optional"/>
        <xs:attribute name="systemOrigin" type="xs:string" use="optional"/>
        <xs:attribute name="externalReference" type="xs:string" use="optional">
            <xs:annotation>
                <xs:documentation>
                    Could be web page of documentation, external object identification, etc.
                </xs:documentation>
            </xs:annotation>
        </xs:attribute>
        <xs:attribute name="schemaUrl" type="StringType" use="optional">
            <xs:annotation>
                <xs:documentation>
                    Downloadable XSD/JSON/Avro schema http/https endpoint.
                </xs:documentation>
            </xs:annotation>
        </xs:attribute>
        <xs:attribute name="businessOwnerName" type="xs:string" use="optional"/>
        <xs:attribute name="businessOwnerEmail" type="EmailAddressType" use="optional"/>

    </xs:complexType>
    <xs:element name="DataObject" type="DataObjectType"/>

    <xs:complexType name="DataObjectsType">
        <xs:annotation>
            <xs:documentation>
                Collection wrapper used in most cases to transport single data object, but is open to support multi object configuration.
            </xs:documentation>
        </xs:annotation>
        <xs:sequence>
            <xs:element ref="DataObject" minOccurs="0" maxOccurs="unbounded"/>
        </xs:sequence>
    </xs:complexType>

    <xs:element name="DataObjects" type="DataObjectsType"/>
    <xs:complexType name="SenderReferenceType">
        <xs:sequence>
            <xs:element name="Id" type="IdentifierType" minOccurs="0"/>
            <xs:element name="SenderMessageID" type="IdentifierType" minOccurs="0"/>
            <xs:element name="TransactionCode" type="CodeType" minOccurs="0"/>
            <xs:element ref="ObjectCrossReference" minOccurs="0"/>
            <xs:element ref="Application" minOccurs="0"/>
        </xs:sequence>
    </xs:complexType>
    <xs:element name="SenderReference" type="SenderReferenceType"/>
    <!--
    <xs:complexType name="BusinessScopeReferenceType">
        <xs:sequence>
            <xs:element name="ID" type="IdentifierType" minOccurs="0"/>
            <xs:element name="InstanceID" type="IdentifierType" minOccurs="0"/>
            <xs:element name="EnterpriseServiceName" type="NameType" minOccurs="0" maxOccurs="unbounded"/>
            <xs:element name="EnterpriseServiceOperationName" type="NameType" minOccurs="0" maxOccurs="unbounded"/>
        </xs:sequence>
    </xs:complexType>
    <xs:element name="BusinessScopeReference" type="BusinessScopeReferenceType"/>

    <xs:complexType name="BusinessObjectDocumentReferenceType">
        <xs:sequence>
            <xs:element name="EBMID" type="IdentifierType" minOccurs="0"/>
            <xs:element name="EBMName" type="NameType" minOccurs="0" maxOccurs="unbounded"/>
            <xs:element name="EBOName" type="NameType" minOccurs="0" maxOccurs="unbounded"/>
            <xs:element name="VerbCode" type="CodeType" minOccurs="0"/>
            <xs:element ref="BusinessScopeReference" minOccurs="0"/>
            <xs:element ref="SenderReference" minOccurs="0"/>
        </xs:sequence>
    </xs:complexType>
    <xs:element name="BusinessObjectDocumentReference" type="BusinessObjectDocumentReferenceType"/>
    -->
    <!-- 
    ACKNOWLEDGE MESSAGE

    -->
    <xs:simpleType name="EventClass">
        <xs:annotation>
            <xs:documentation>Purpose - Provide a high level categorisation of events Detail - Message events pertain to the structure and delivery of aseXML messages. Application events are generated by the applications communicating via aseXML. Processing events reflect environmental issues associated with running applications.</xs:documentation>
        </xs:annotation>
        <xs:restriction base="xs:string">
            <xs:enumeration value="Message"/>
            <xs:enumeration value="Application"/>
            <xs:enumeration value="Processing"/>
        </xs:restriction>
    </xs:simpleType>
    <xs:simpleType name="EventSeverity">
        <xs:annotation>
            <xs:documentation>Purpose - Provide an indication as to the action required as a result of receiving an event Detail - The severity dictates the further processing possible after receiving the event.</xs:documentation>
        </xs:annotation>
        <xs:restriction base="xs:string">
            <xs:enumeration value="Information"/>
            <xs:enumeration value="Warning"/>
            <xs:enumeration value="Error"/>
            <xs:enumeration value="Fatal"/>
        </xs:restriction>
    </xs:simpleType>
    <xs:complexType name="EventCode">
        <xs:annotation>
            <xs:documentation>Purpose - Provide a numeric indentification of the event that has occurred Detail - The aseXML guidelines define a set of common codes, as well as ranges for application specfiic errors. Where the receiving system does not know the textual definition of the code, the description attribute should be provided with the code, as per the aseXML guidelines.</xs:documentation>
        </xs:annotation>
        <xs:simpleContent>
            <xs:extension base="EventCodeBase">
                <xs:attribute name="description" type="xs:string" use="optional"/>
            </xs:extension>
        </xs:simpleContent>
    </xs:complexType>

    <xs:simpleType name="EventKeyInfo">
        <xs:annotation>
            <xs:documentation>Purpose - Provide information to allow identification of the data that triggered the event Detail - Where the combination of class and code are insufficient to completely describe an event, KeyInfo may be used to provide further detail as to the information needed to locate the source of the event within the original transaction.</xs:documentation>
        </xs:annotation>
        <xs:restriction base="xs:string">
            <xs:maxLength value="80"/>
        </xs:restriction>
    </xs:simpleType>

    <xs:simpleType name="EventContext">
        <xs:annotation>
            <xs:documentation>Purpose - Identify the specific data/condition that caused the event Detail - The portion of the input to which the event applies should be provided.</xs:documentation>
        </xs:annotation>
        <xs:restriction base="xs:string">
            <xs:maxLength value="80"/>
        </xs:restriction>
    </xs:simpleType>
    <xs:simpleType name="EventCodeBase">
        <xs:annotation>
            <xs:documentation>Purpose - Helper type in defining EventCode</xs:documentation>
        </xs:annotation>
        <xs:restriction base="xs:nonNegativeInteger"/>
    </xs:simpleType>
    <xs:simpleType name="ReleaseIdentifier">
        <xs:annotation>
            <xs:documentation>Purpose - Identify a release of aseXML Details - Release identifiers are used extensively in aseXML to enable versioning of transactions.</xs:documentation>
        </xs:annotation>
        <xs:restriction base="xs:string">
            <xs:pattern value="r[0-9]*|r[0-9]*_[a-z][0-9]*"/>
        </xs:restriction>
    </xs:simpleType>
    <xs:complexType name="EventSupportedVersions">
        <xs:annotation>
            <xs:documentation>Purpose - Provide the set of versions supported for a particular transaction Detail - Where an unsupported version of a supported transaction is received, the receiver should indicate those versions that are supported, allowing the receiver to choose a compatible transaction version.</xs:documentation>
        </xs:annotation>
        <xs:sequence>
            <xs:element name="Version" type="ReleaseIdentifier" maxOccurs="unbounded"/>
        </xs:sequence>
    </xs:complexType>

    <xs:complexType name="Event">
        <xs:annotation>
            <xs:documentation>Purpose - Common format used to report events Detail - aseXML encourages the use of events in transactions to report status information. They are also used in message and transaction acknowledgements. Where possible, an element name of "Event" should be used.</xs:documentation>
        </xs:annotation>
        <xs:sequence>
            <xs:element name="Code" type="EventCode"/>
            <xs:element name="KeyInfo" type="EventKeyInfo" minOccurs="0"/>
            <xs:element name="Context" type="EventContext" minOccurs="0"/>
            <xs:element name="Explanation" type="xs:string" minOccurs="0"/>
            <xs:element name="SupportedVersions" type="EventSupportedVersions" minOccurs="0"/>
        </xs:sequence>
        <xs:attribute name="class" type="EventClass" use="optional" default="Application"/>
        <xs:attribute name="severity" type="EventSeverity" use="optional" default="Fatal"/>
    </xs:complexType>

    <xs:element name="Event" type="Event"/>

    <xs:complexType name="Acknowledgements">
        <xs:annotation>
            <xs:documentation>Purpose - The payload used to carry acknowledgments Detail - The aseXML acknowledgement model calls for each message and transaction to be acknowledged, subject to certain rules. For further information, refer to the aseXMl guidelines.</xs:documentation>
        </xs:annotation>
        <xs:sequence>
            <xs:element name="MessageAcknowledgement" type="MessageAcknowledgement" minOccurs="0" maxOccurs="unbounded"/>
            <xs:element name="TransactionAcknowledgement" type="TransactionAcknowledgement" minOccurs="0" maxOccurs="unbounded"/>
        </xs:sequence>
    </xs:complexType>

    <xs:complexType name="MessageAcknowledgement">
        <xs:annotation>
            <xs:documentation>Purpose - Acknowledge a received message Detail - For every message, a message acknowledgment is generated (with the exception of messages containing message acknowledgements). Any errors are reported via Event elements. If a message is rejected, no receipt identifier need be provided with the acknowledgement.</xs:documentation>
        </xs:annotation>
        <xs:sequence>
            <xs:element name="Event" type="Event" minOccurs="0" maxOccurs="unbounded"/>
        </xs:sequence>
        <xs:attribute name="initiatingMessageID" type="UniqueIdentifier" use="required"/>
        <xs:attribute name="receiptID" type="ReceiptIdentifier" use="optional"/>
        <xs:attribute name="receiptDate" type="xs:dateTime" use="required"/>
        <xs:attribute name="status" type="MessageStatus" use="required"/>
        <xs:attribute name="duplicate" type="IndicatorType" default="false"/>
    </xs:complexType>

    <xs:element name="MessageAcknowledgement" type="MessageAcknowledgement" />

    <xs:simpleType name="MessageStatus">
        <xs:annotation>
            <xs:documentation>Purpose - Indicate the acceptance or rejection of the message.</xs:documentation>
        </xs:annotation>
        <xs:restriction base="xs:string">
            <xs:enumeration value="Accept"/>
            <xs:enumeration value="Reject"/>
        </xs:restriction>
    </xs:simpleType>

    <xs:simpleType name="UniqueIdentifier">
        <xs:annotation>
            <xs:documentation>Purpose - Uniquely identify the associated data element or attribute. Detail - Note that unique identifiers do not have to be globally unique, only unique to a particular sender. However, the length has been chosen such that UUIDs can be used as identifiers if considered appropriate, guaranteeing global uniqueness.</xs:documentation>
        </xs:annotation>
        <xs:restriction base="xs:string">
            <xs:minLength value="1"/>
            <xs:maxLength value="36"/>
        </xs:restriction>
    </xs:simpleType>

    <xs:simpleType name="ReceiptIdentifier">
        <xs:annotation>
            <xs:documentation>Purpose - Provide a unique identifier to acknowledge a message or transaction. Detail - A receipt identifier is generated by the receiver of an aseXML message or transaction. Its purpose is to provide the sender with a string that can be quoted when inquiring with regard to the progress of processing for a message or transaction. As implied by the name, it is the sender's guarantee that the reciever will process the message or transaction. Note that receipt identifiers do not have to be globally unique, only unique to a particular receiver. However, the length has been chosen such that UUIDs can be used as receipt identifiers if considered appropriate, guaranteeing global uniqueness.</xs:documentation>
        </xs:annotation>
        <xs:restriction base="UniqueIdentifier"/>
    </xs:simpleType>

    <xs:simpleType name="TransactionIdentifier">
        <xs:annotation>
            <xs:documentation>Purpose - Uniquely identify every transaction generated by the transaction sender Detail - Note that transaction identifiers do not have to be globally unique, only unique to a particular sender. However, the length has been chosen such that UUIDs can be used as transaction identifiers if considered appropriate, guaranteeing global uniqueness. A transaction acknowledgement identifies which transaction it is acknowledging by providing the transaction identifier as an attribute. In addition to carrying its own unique identifier, a transaction generated in response to the initial transaction of a transaction exchange also carries the transaction identifier of the initial transaction. This allows the sender to determine the context in which the response transaction should be interpreted.</xs:documentation>
        </xs:annotation>
        <xs:restriction base="UniqueIdentifier"/>
    </xs:simpleType>

    <xs:complexType name="TransactionAcknowledgement">
        <xs:annotation>
            <xs:documentation>Purpose - Acknowledge a received transaction Detail - For every transaction, a transaction acknowledgment is generated. Any errors are reported via Event elements. If a transaction is rejected, no receipt identifier need be provided with the acknowledgement.</xs:documentation>
        </xs:annotation>
        <xs:sequence>
            <xs:element name="Event" type="Event" minOccurs="0" maxOccurs="unbounded"/>
        </xs:sequence>
        <xs:attribute name="initiatingTransactionID" type="TransactionIdentifier" use="required"/>
        <xs:attribute name="receiptID" type="ReceiptIdentifier" use="optional"/>
        <xs:attribute name="receiptDate" type="xs:dateTime" use="required"/>
        <xs:attribute name="status" type="TransactionStatus" use="required"/>
        <xs:attribute name="duplicate" type="IndicatorType" default="false"/>
        <xs:attribute name="acceptedCount" type="xs:nonNegativeInteger" use="optional"/>
    </xs:complexType>

    <xs:simpleType name="TransactionStatus">
        <xs:annotation>
            <xs:documentation>Purpose - Indicate the acceptance, partial acceptance or rejection of the transaction.</xs:documentation>
        </xs:annotation>
        <xs:restriction base="xs:string">
            <xs:enumeration value="Accept"/>
            <xs:enumeration value="Partial"/>
            <xs:enumeration value="Reject"/>
        </xs:restriction>
    </xs:simpleType>

</xs:schema>

It is really just a sample where this schema is followed by other schemas where one can define and I used, it is quite big... so I discuss here just the header as message metadata, in this particular scenario I showed my correlationId is hidden under Sender object as SenderMessageID. I am not saying my schema is holly grail :-)), and my whole setup is quite heavy, so it could be done in more lightweight way, but I think we might come up with something like generic messaging headers. We could come up with some proposal as PR and maybe ask community if they agree on specific parameters before added in. I believe genericity is the judgement here.

@zhouxinyu - correlationId you can see as global transaction ID, when you have sequence of messages within a flow like:

request-reply
sequence of events in event driven architectures (fully async or combination of sync and async), you can use correlation ID as master key (grouper) of all messages related to each other within flow

Also note that for example for reading operations I think using message headers can be optional, but is fully required for write operations as you can use those metadata for audit purposes like:

who requested object creation/update/deletion
when they requested it
show me full flow of messages for specific object creation/update/deletion

This is especially very handy when you integrate 2 or more foreign systems owned by different parties. When you have issue with some message flow, you just provide to the other party your reuqestId, correlation Id, but is optional. I define in my schema so called object DomainMap, this is nothing else than multidimensional table and Properties which is nothing more than key/value store. These are optional and actually you can really put there anything which will help both sides to smoothly integrate.

Thx for any comments.

vongosling commented 6 years ago

I have looked through your XSD quickly, some event sourcing showing ...Hah, recently, we add messaging schema in OpenMessaging, would you like to help us to review the initial schema(Is it suitable for your usecase), we will appreciate your sincere advice.

“I believe genericity is the judgement here.” Yes, we have the same belief.

archenroot commented 6 years ago

I will take a look, good to hear, could you share direct link of the schema you talk about, just I look at right place.

On Sun, Jun 24, 2018, 1:39 PM von gosling notifications@github.com wrote:

I have looked through your XSD quickly, some event sourcing showing ...Hah, recently, we add messaging schema in OpenMessaging. Now, you could help us to review the initial schema, we will appreciate your sincere advice.

“I believe genericity is the judgement here.” Yes, we have the same belief.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/openmessaging/openmessaging-java/issues/19#issuecomment-399750103, or mute the thread https://github.com/notifications/unsubscribe-auth/AAhyKHdMvjsbBypn6MSOblEz8z1kYUqtks5t_3p8gaJpZM4SZB77 .

archenroot commented 6 years ago

@zhouxinyu - it is very simple use cases for correlationId.

Use case 1 - 2 application first responsible for requesting payments, second for validation of payments first send request to second one about to validate payment construct domain object in either sync or async mode, what it gets is Acknowledgement, not real response (as this pattern is Delayed response). Second services/system when validates the payment it will send again back to system 1 validation response (positive/negative/error)

The flow in case of sync flow will be as following: Request - App1 -> request correlationId=9, messageId=1 -> App2 Acknowledgement(but it could be also negative acknowledgement -Nack or Rejection if you like) - App2 -> ack correlationId=9, messageId=345 -> App1 Delayed Response - App2 -> response correlationId=9, messageId=35467498 ->App2

So this example demonstrates correlation Id, in your Audit Log (Audit Pattern) which is implemented in Event sourcing architectures via Event Store directly and does not require any additional components shows 3 messages bounded by correlationId, it tells you they are all part of same flow/process instance.

openmessaging / openmessaging-java

Messaging patterns #19