Message Within Message

[adkapur]

We have a requirement in which a wrapper message is composed of a number of sub messages or templates with each of these representing specific sections of data, for example:

|--------------message-----------------------------------------------| |---common---||---specific message---||---metadata--|

Common template represents data which is common across all messages, specific message refers to NewOrderSingle, OrderCancelReplaceRequest etc and metadata refers to internal system data

This way if changes are made to a specific template such as metadata then the impact could be isolated to consumers of that section only

Currently each of these sub messages or templates is required to be present in its own schema with a separate schema for the wrapper message but instead of having to maintain multiple schemas is it possible to include the wrapper template and all of these sub templates in a single schema?

[donmendelson]

Note: this enhancement changes wire format and therefore is a breaking change with version 1.0.

[donmendelson]

Related to issue #47 - both discuss composability of messages from common elements, using different terminology.

[donmendelson]

This way if changes are made to a specific template such as metadata then the impact could be isolated to consumers of that section only

I'm in favor of improving message composibility, but I don't see how this proposal isolates changes. In SBE, versioning is atomic at the level of a message schema, not individual message templates. A change to either the container message or any submessage requires an updated schema anyway.

If a change to a submessage is a breaking change under the rules of schema extension, it would require a new template. In that case, the entire submessage would be unreadable to a client not on the current version.

[adkapur]

In SBE, versioning is atomic at the level of a message schema, not individual message templates. A change to either the container message or any submessage requires an updated schema anyway.

Yes correct but the outer message should rarely change for this to work and submessage changes will be of interest only to its interested consumers. Consumers not interested in that particular submessage can disregard it anyway

If a change to a submessage is a breaking change under the rules of schema extension, it would require a new template. In that case, the entire submessage would be unreadable to a client not on the current version.

Yes that is true only for a consumer interested in that particular submessage but if a consumer is not then it should be transparent to them

This proposal also included fields in the outer message which will contain the offset and template ID of each submessage allowing consumers to skip the submessage that they are not interested in and not having to decode that aside from reading the bytes

We are anyway thinking to drop this proposal since it requires double encoding in favor of the framing header wrapping around multiple templates and containing the total message length of the logical message (which could encompass multiple templates) and extending the SBE message header to include total message size

[RFrenkel]

yeay!