Xtype / Dynamic Type Examples?

[willstott101]

I'm trying to use DDS in a commercial robotics framework and think I have a handle on the features I need. My limited fiddling with Fast-DDS has left me very unimpressed, but they did have enough information to get me started. CycloneDDS feels more inviting code-wise but I'm really struggling to understand what is/isn't possible/easy right now and what's missing, due to the lack of examples.

In an attempt to help others with my research I'd rather contribute small examples to this repo than make my own toys that I throw away. There are a few examples I'd like to see in C before even thinking about trying to contribute them to cyclonedds-cxx or even cyclonedds-rs.

• listtopics but with an IDL printout of any introspectable types
• A simple publisher which publishes introspectable type information

I have a few questions:

• If I understand correctly, the cyclonedds-python project has implemented more of the XTypes dynamic type stuff than cyclonedds proper?
• Is there anything actually missing from cyclonedds which would prevent the above suggested examples being written?

[thijsmie]

Hi @willstott101,

Indeed dynamism with types is possible with XTypes and as of today a dynamic subscription is even possible with C, as you can turn TypeInfo back into a sertype and create a Topic using that. That indeed could be turned into an example, though it is of limited use because introspection at runtime is limited in C so doing anything useful with a dynamic type is very difficult.

On the python side things are different. There are API functions to resolve datatypes and they can actually be useful at runtime. In fact, there is a command line tool installed by the python backend that allows cyclonedds typeof <topic name> and cyclonedds subscribe<topic name> that rely on the dynamism of XTypes to function.

I would be happy to work with you to write some nice annotates examples and then add them to the repository. Is that something you would be interested on collaborating on?

[willstott101]

Yes I would be very interested in getting some help to put together some examples, thankyou.

I will explore those python tools tomorrow, in the meantime I wonder if you could expand on what you mean by runtime introspection of the type in C being "limited". Do you mean, it would take a large volume of code to use it effectively, or there's type information that the C library just can't parse and make available to a user (as a set of structs).

Do you think for instance, dynamic subscription -> JSON would require reading the DDS XTypes spec and adding to cyclonedds core, or it's just limited cause the example would be a lot of tedious C code?

[eboasson]

Hi @willstott101, the "problem" with doing it in C is simply the tediousness of it. Probably the easiest way to do is to walk over the (deserialised) type object while computing offsets in the data and converting the fields.

You'll probably run into some things that aren't exported in the stable API because we have really been planning on using Python for doing such things. For example: dds_get_typeobj returns what is today documented as an opaque pointer, making it mostly useless without relying on a few implementation details that are probably pretty stable by now. The implementation detail you need to know is that really returns a pointer to the type object in the IDL-to-C mapping of the DDS::XTypes::TypeObject type defined in the XTypes specification and that the C type definition is in some implemention header file that does get installed along with the rest. So you could take the IDL from the spec, fix the bugs, run it through IDLC and rely only on the returned type object being a instance of this type ... or you could just include that implementation header file and get going.

What I think you are looking to do is basically the following (no attempt made at checking it, treat this as buggy pseudo code):

#include "dds/dds.h"
#include "dds/ddsi/ddsi_typelib.h"
#include "dds/ddsi/ddsi_xt_impl.h"

static void print_sample_as_json (const void *sample, const DDS_XTypes_TypeObject *typeobj) {
  // the tedious bit of implementing this is left as an exercise for the reader :)
}

dds_entity_t dcpspublication_reader = dds_create_reader (..., DDS_BUILTIN_TOPIC_DCPSPUBLICATION, ...)
void *epraw = NULL
... dds_take (dcpspublication_reader, &epraw, 1, ...) ...
dds_builtintopic_endpoint_t *ep = epraw
dds_typeinfo_t *typeinfo;
dds_typeid_t *typeid;
// ddsi_typeinfo_complete_typeid or the equivalent appears to be missing in the API ...
if (dds_builtintopic_get_endpoint_type_info (ep->qos, &typeinfo) &&
    (typeid = ddsi_typeinfo_complete_typeid (typeinfo)) != 0)
{
  dds_entity_t topic = dds_find_topic (..., ep->topic_name, typeinfo, ...)
  dds_entity_t rd = create_reader (..., topic, ep->qos /* or null, or whatever QoS you want */, ...)
  dds_typeobj_t *typeobj;
  dds_get_typeobj(..., typeid, ..., &typeobj)
  void *sample = NULL;
  while (1) {
    if ((n = dds_take (reader, &sample, 1, ...)) < 0) abort ();
    print_sample_as_json (sample, &typeobj->x);
    dds_return_loan (reader, &sample, n);
  } 
}

As you can see there is not a lot of non-public API stuff needed, and we'll be happy to extend the latter so that this is not needed at all. Indeed, the missing bit I ran into in the above should be added and something like the above should then be added to the tests as well, because right now the focus on Python for this type pattern means the C tests currently only test the components, but not the entire chain. That's not good!

I have no doubt that you'd run into some problems. Just ask, we are happy to help you deal with them and/or extend or fix the core to make life better.

P.S. If all you need is a textual representation of the data, then dds_takecdr + ddsi_serdata_print might be enough already. That prints something like a C initialiser, no field names, but enough to know whether you're on the right track.

[willstott101]

Thanks! I'll see what I can get to with a little fiddling. It's exactly the tedious bit I want to write :) The serialization is just a simple task to prove I can walk the type and decode via it.

What I actually want to do is be able to have components of my framework publish structs, some field's types of which are well-known, so that generic subscriber code can operate on the subset of the data they understand. Without resorting to a single well-known type with map<string, string> and a bunch of optional fields or something in it.

The ideal would be to be able to make use of this in Rust, but it's looking like my initial implementation will end up being Python for expediency. However I would still like to contribute to enabling more dynamic type usage in rust - and where the C core can be extended to help that effort will surely help all future language bindings (particularly those with dynamic types / JITs themselves).

[thijsmie]

While I applaud your willingness to write this tedious code and would be happy to have it as an example: isn't using an @appendable or @mutable type with the known subset good enough for your purpose? Then you could subscribe with the normal approach without the dynamic fiddling: the dynamism of datatype compatibility being already handled by XTypes.

[eboasson]

@willstott101 I wanted to know whether I had given you correct advice ... it turns I made a few mistakes, but the upside is that this: https://github.com/eclipse-cyclonedds/cyclonedds/pull/1344 probably can probably save you some time 😀

[willstott101]

Quite possibly @thijsmie ... perhaps using @mutable and @hashid("...") cause ideally I'd able to treat the well-known types a little like mixins. According to the RTI docs the default for hashed IDs is to use just the field name which seems a little brittle to me. A hash of the Type Definition along with the field name would suit me well... however that wouldn't work well for nested partial types. Still struggling to understand the basics to be honest. Perhaps I'd be better off trying to help with a XTypes Dynamic Types section in https://cyclonedds.io/docs/cyclonedds/0.9.1/

[thijsmie]

Note that for assignibility checking the type definition is considered so a matching hash is not enough (although I do agree with you there are some weaknesses in the design spec, but that is something we cannot help). You can have nested @mutable and @appendable things and things should keep working (we run fuzzing tests in the python backend to verify this).

If indeed you want to scan for "known types" within a complex structure you'll have to go the dynamic route as you originally proposed.

Any contribution to the docs is very welcome! I started to write a little bit about XType features but it is nowhere near complete: https://cyclonedds.io/content/blog/xtypes-features.html

[eboasson]

Hi @willstott101, after having spent yesterday afternoon with a "proper" keyboard — no plastic anywhere in sight, very touch sensitive and a much better layout than QWERTY, Dvorak, &c., and that has stood the test of time 😁 — I had a few hours to catch up on. #1344 just got upgraded to "ready for review" after fixing the most egregious problems, including a fix for the handling of dependent types.

If you hadn't played around with it yet, then I'm happy I sorted this out before you ran into problems. If you did, then it is worth checking out the improvements.

[thijsmie]

New example is merged! 😄. Thanks @willstott101 for raising the question.

[willstott101]

Thanks a lot to both of you. I haven't had a chance to dig into the example you made but I would like to when I explore a cpp/c version after the Python POC I am working on. Hopefully the example proves useful to other people too.

[eboasson]

@willstott101 doing this dynamic type stuff is definitely non-trivial and so it really needed an example. It seems like this kind of example only ever gets written in response to a question, so thank you, too!

(And feel free to ask other questions as well, who knows what other examples they will result in 😀.)