robshakir
commented
5 years ago Hi Ganesh,
Thanks for the question.
The common thread to link different representations of YANG models together is the schema and datatree paths that are used for a particular element. We annotate both the Go files, and the generated protobufs with the paths within the schema, and runtime libraries can then subsequently determine what the data tree paths are by appending the keys for elements to this. So, generically, I think the answer is that we should be able to take <path, value> tuples and deserialise these into any form of generated entity with reflects a YANG schema.
gNMI's serialisation defines a common type structure that is used for exchange between systems, so the use of the <path, value> tuples using gNMI's TypedValue message would then mean that we have a way to unify types across the different generated code -- such that we can optimise for the best type for each different code artifact's users.
Let's say that we made this <path, value> tuple a gnmi_proto.Update - which can store both the path (as gnmi_proto.Path) and value (as suggested above as gnmi_proto.TypedValue). This has another advantage insofar that it is useful library code to generate gnmi_proto.Notification from generated structures for telemetry use-cases, and deserialising gnmi_proto.Update into a data tree is required for the gNMI Set implementations.
This approach now allows you to arbitrarily map between any two representations of a YANG schema that has had code generated for it. It also maintains the advantage that there is no need for consistency between the different code generation libraries -- each language can choose the most idiomatic approach for itself.
For Go, ygot already implements much of the required functionality here for the ygot.GoStruct interface:
ygot.TogNMINotificationsconverts aygot.GoStructto[]*gnmipb.Notification.ytypes.SetNodecan set field in aygot.GoStructbased on agnmipb.Pathandgnmipb.TypedValue.
For protobuf, we also designed and started implementation on similar infrastructure -- it needs mapping between a particular protobuf message or field and the corresponding schemapath. The design document for this is accessible here (Google account required). The initial protoc-gen-go plugin to implement the output of the annotations can be found in the protoyangplugin directory in ygot. Similar plugins within the protoc framework are required per-language (similarly to the code generation initially required, and the runtime libraries).
Please let me know if this isn't clear -- hopefully the design doc answers most of the initial queries.
r.
gsindigi
With the current set of tools/support (YGOT & GoYang), there are 3 different ways to generate the Go Structs for a given YANG schema.
1 YGOT-Generator
This generates the related Go artifacts directly and can readily be used for gNMI related support. Also, its possible to un-/marshal from/to RFC7951 compliant JSON. All such functions are included either in the
ygotor the generated structs.2 Combination of proto_generator & protoc
This is a two step process, YANG schema is converted to an intermediate
protofile withproto_generatorand converted to Go struct usingprotoc's compiler pluginprotoc-gen-go.3 Combination of goyang & protoc
This too is a two step process, YANG schema is converted to an intermediate
protofile withgoyangand converted to Go struct usingprotoc's compiler pluginprotoc-gen-go.All above 3 ways are great and work as expected in isolation. However, when these are used in combination e.g., 1 & 2, 1 & 3 or 2 & 3, there is no easy way to convert from one form to other due to different reasons though they are all referring to same YANG schema.
Listed below are some of such, which I came across.
difference in name generation e.g., use '_', '-' in paths, field names etc.
This is translated as
difference in enum handling & encoding values e.g.,
difference in data-types used e.g.,
The possibility of inter-conversion is very much required in a polyglot technologies/development scenarios. Imagine, the device software is in C++ (for legacy reasons) which supports all OpenConfig modules information e.g.,
stateinformation and intended to provide gNMI support by leveraging YGOT support available. One easy option to begin with would be choosing to combine above conversion mechanisms 1 & 2 or 1 & 3, so that C++ can make use protoc generatec C++ Class artefacts and Go implementation can depend onygot.GoStructalone (to leverage YGOT support for gNMI to large extent). Now, with differences enlisted above, such a design choice may not work easily and would need identification & intermittent conversions of such differences (as and when they are uncovered?) at multiple places. This forces not to use Protobuf's efficient on-the-wire encoding. Instead, if JSON is used to convert from one form to other, then it too will run into some non-matching fields issues and thus either errors out or skips.I understand, these tools have evovled independently and at different phases. And it may not be right to expect a convergence.
I am not sure, how easy or difficult a task it would be to come up with a conversion mechanism for Go-Struct (option 2/3 above) suitable for
ygot.GoStructs.Do you see any easy way to converge generated Go structs across different forms and have them ready as
ygot.GoStructs for subsequent gNMI related handling? And there will be other design alternatives to overcome such.Appreciate any inputs/clarifications on this front. Thanks.