Re-reuse of repeating group entries

[zamhassam]

We would like to share our repeating group definitions across multiple messages. We have tried modelling this as having repeating groups with each having one entry and that entry being a composite, however composites are much more restrictive than repeating groups.

The motivation is we have some shared entities which are present in repeating groups in multiple messages, but currently each one is scoped by its own enclosing message and the same codec is generated multiple times. As a consequence, our application has several areas of duplicated code to deal with several identical codecs.

Related to #47 and #13.

Sample schema to demostrate the idea using a new type called component which should provide the same flexibility as a repeating group (fixed length fields, nested groups, variable length data) to define the structure of a group entry:

<sbe:messageSchema xmlns:sbe="http://fixprotocol.io/2016/sbe" byteOrder="littleEndian" description="" id="1" package="" semanticVersion="0.0.1" version="1">
    <types>
        <component id="16" name="marketDataEntry">
            <field name="price" id="17" type="decimalFloat" description=""/>
            <field name="quantity" id="19" type="decimalFloat" description=""/>
            <field name="quantityType" id="20" type="quantityType" description=""/>
            <field name="side" id="21" type="side" description=""/>
            <group>
            </group>
            <data name="referenceId" id="22" type="varStringEncoding" description=""/>
        </component>
    </types>
    <sbe:message id="1" name="MarketDataIncremental" description="">
        <group id="24" name="MarketDataEntries" dimensionType="groupSizeEncoding" component="marketDataEntry"/>
    </sbe:message>
    <sbe:message id="2" name="MarketDataSnapshot" description="">
        <group id="24" name="MarketDataEntries" dimensionType="groupSizeEncoding" component="marketDataEntry"/>
    </sbe:message>
</sbe:messageSchema>

[kleihan]

Summary of discussion of May 8 in HPWG conference call:

• A grouping mechanism is useful for re-use of identical parts and would be an optional feature, i.e. it could also only be used internally and flattened when sent to the counterparty.
• Names for components should only be arbitrary when used for a non-FIX application layer. FIX has a grouping concept in the repository (FIX Orchestra) that includes semantics, i.e. repeating groups are a 1:n design compared to 1:1 for a non-repeating group. SBE schemas should be able to be automatically validated against FIX Orchestra for FIX compliance. Requires to define the related application layer in the schema.
• FIX components, e.g. "Instrument" always refer to all of its fields and nested components. Actual implementations will always only use a subset of possible fields and nested components. Hence a naming convention may have to include two parts: 1) the FIX name 2) an abitrary name for the subset (aka scenario in FIX Orchestra), e.g. "Instrument"+"Inbound" and "Instrument"+"Outbound", whereby the outbound scenario contains additional instrument fields that are not needed/available inbound.
• Next step is to come up with a syntax proposal and extension mechanism if needed.

[donmendelson]

Let's assume that the boundaries of a component block such as Instrument are respected. Then, a component is the same thing as a repeating group, except that the numInGroup is always 1, or perhaps zero if the component is optional. Like a repeating group, it would support extension and could include nested groups and var length fields. To support extension, you must send blockLength on the wire, so we might as well keep the dimension header the same format as for a repeating group, except that the numInGroup element is constrained by maxValue, an attribute that already exists.

<composite name="componentSizeEncoding">
    <type name="blockLength" primitiveType="uint16"/>
    <type name="numInGroup" primitiveType="uint16" maxValue="1"/>
    <type name="numGroups" primitiveType="uint16"/>
    <type name="numVarDataFields" primitiveType="uint16"/>
</composite>

[mjpt777]

This could be difficult to implement efficiently. An alternative which would be to allow a repeating group to contain another message. Allowing a message to contain a repeating group of messages is more like other type systems and composable. I'd be tempted to not add this to the specification without prototyping to get a better understanding of the consequences.

[juddgaddie]

After a discussion with @zamhassam we agree with Martin's assessment and are happy for this issue to be closed until we are able to get an indication of its implementation viability, if we are able to prototype a viable solution we will submit a new issue with the details.