Interface hierarchy for generated types

[benalexau]

SBE Tool currently generates MessageEncoder and MessageDecoder interfaces in the sbe:messageSchema-defined package. The tool also ensures that flyweights generated for each sbe:message implement either MessageEncoder or MessageDecoder.

I am currently developing a library for storing SBE data in the LMDB embedded key-value store (via LMDBJNI), but have encountered several difficulties in evolving this into a general-purpose library of value to others:

• The generation of interfaces in end user projects prevents them from being referenced by a general-purpose library. One alternative is to provide a code generator for end user projects, although this is a lot of extra work, not to mention inconvenient for end users to configure and use. Alternately I could model all interfaces using Object, but this has adverse impacts around self-documentation, type safety and reflection overhead.
• Flyweights generated for sbe:composite types do not declare any interface. This prevents them from being referenced in interfaces. There are uses for composites as first-class citizens, for example as keys in an ordered map (particularly given the lack of versioning that a composite implies and its non-reliance on a messageHeader that would impact key ordering via its version field).
• There is no way of referencing a message or a composite as a top-level type that is independent of its encoder or decoder. This prevents symmetric interfaces that regularly need to reference both the encoder and decoder flyweights for a given type (as shown in my PoC below).

To explore what might be feasible, I have produce a proof of concept project. It shows:

• A potential Agrona interface hierarchy that would provide a foundation for third party libraries that need to expressively reference SBE generated types and flyweights
• A type-safe DAO interface and implementation that illustrates use of the proposed Agrona hierarchy
• The SBE generated “car” types, with manual changes to reflect the proposed Agrona hierarchy (note such changes are minimal: (a) the current MessageEncoder and MessageDecoder are no longer generated, (b) the currently-generated flyweights would instead refer to a single Agrona interface (see example), and (c) there is a need to generate a new zero method Type class for each sbe:composite and sbe:message)

I am not advocating for this exact solution, but rather seeking discussion on whether pull requests that modify SBE Tool and Agrona to achieve something along these lines would be reasonable. Of course I’m happy to prepare PRs, but preferred some initial discussion first.

As an aside, I’ll be happy to release the resulting LMDB solution if we can resolve the third-party referencing challenges mentioned above. I expect it will be useful for SBE users seeking fast, embedded, key-ordered storage for their SBE types. The introduction of interfaces such as these would also enable others to build similar SBE-aware libraries.

[mjpt777]

Would a simple solution be to optionally generate the schema specific interface?

[benalexau]

@mjpt777, do you mean SBE Tool having an option to generate the org.agrona.sbe package and associated interfaces in the user's sbe.output.dir? It would work, but it feels unusual to generate static .java files versus just putting them in a versioned JAR dependency. Or do you mean something else?

[mjpt777]

It could be as simple as an option to not generate any interfaces?

[benalexau]

Adding an sbe.generate.interfaces (?) option to SBE Tool has some advantages:

• It's simple
• It can default to false, meaning no change from today aside from removing the currently-generated MessageEncoder and MessageDecoder (and I'd imagine anyone using those two interfaces would benefit from the new interface model anyway)
• It means only modifying SBE, not Agrona
• It gives time for the interfaces to evolve, stabilise and demonstrate utility without creating any release coupling between SBE and Agrona

If you are happy with this, I can produce a PR. Or if you have other comments on the interface hierarchy etc, please feel free.

[mjpt777]

Yes. I think it is the best way to begin with. We can then have time to discuss what makes the most sense longer term.

BTW I think it is great that SBE gets used with LMDB :-)

[benalexau]

Issue links:

• 160 contained original request for interfaces

• 334 contained initial interface implementation

• 350 is the PR for this functionality

[mjpt777]

Adding the empty Structure interfaces adds a lot of complexity to this and the JavaDoc does not help explain why. Could you either update the JavaDoc to include an example of why these are need to consider dropping this interfaces and just using the Flyweights/Encoders/Decoders interfaces?

[benalexau]

I'm happy to provide a PR with more JavaDocs, but if you are finding Structure adding a lot complexity it's perhaps appropriate to consider whether it's worth keeping at all.

My Java annotation processor generates (at compile time) concrete DAO implementations that persist SBE types to LMDB. Here is an example:

import baseline.CarStructure;
import baseline.EngineStructure;

@GenerateDao(maxMessageLength = 2048, key = EngineStructure.class, value = CarStructure.class)
public interface CarDao {
}

As shown, @GenerateDao relies on Structure so it can express the key and value types, with the former constrained to a CompositeStructure and the latter to a MessageStructure (we don't want the SBE message header for a key, as it would impact LMDB key sort order).

In reality I am having to use string concatenation to arrive at the encoder and decoder types anyway within the annotation processor:

    String key = info.keyStruct.getQualifiedName().toString();
    String val = info.valStruct.getQualifiedName().toString();
    assert key.endsWith(STRUCTURE);
    assert val.endsWith(STRUCTURE);
    key = key.substring(0, key.length() - STRUCTURE.length());
    val = val.substring(0, val.length() - STRUCTURE.length());
    .....
    map.put("keyStruct", info.keyStruct.getQualifiedName().toString());
    map.put("keyEncoder", key + ENCODER);
    map.put("keyDecoder", key + DECODER);
    map.put("valStruct", info.valStruct.getQualifiedName().toString());
    map.put("valEncoder", val + ENCODER);
    map.put("valDecoder", val + DECODER);
    map.put("interface", info.iface.getQualifiedName().toString());

As such, it's not an issue in my use case to abolish Structure and change the annotation to require the key to be a CompositeDecoderFlyweight and the value a MessageDecoderFlyweight. It's a small hack in that we are referring to the decoder when we really want to refer to the type, but it's not major issue if you feel Structure adds complexity to the interface hierarchy or output package directory.

If you like I can submit a PR with JavaDocs for Structure interfaces, or a PR which removes Structure, or you are welcome to remove it if you prefer. I don't mind; just let me know.

[mjpt777]

Thanks for the clarification. This helps a lot.

I think it would be best to drop the Structure interfaces and keep the Flyweights. This will reduce the complexity and avoid the need for overlapping interfaces.

As a separate point have you considering using the id field as your key to flyweight lookup in the map? It would be more efficient and you could use primitive maps.

If you could do a PR to remove that Structure interfaces then that would be much appreciated. I can then move the remaining interfaces to Agrona to simplify further.

[benalexau]

I'll submit a PR removing it early next week (weekend here now).

Thanks for the map ID suggestion. I usually use primitives everywhere, but in this case the map is an input to the Mustache template library which goes off and writes the resulting Java source code. The map keys are needed in the template file, such as private {{keyEncoder}} keyEncoder;. I started writing the source code out directly within the annotation processor, but moving it to a template made it much easier to incrementally refactor the generated code.

[mjpt777]

New interfaces are now in Agrona.

[benalexau]

Thanks!