openconfig / gnmic

gNMIc is a gNMI CLI client and collector
https://gnmic.openconfig.net
Apache License 2.0
170 stars 55 forks source link

Add gNMI Extension field parsing support #509

Open Ichabond opened 4 weeks ago

Ichabond commented 4 weeks ago

gNMI allows for the use of an extension field in each top-level message of the gNMI RPCs: https://github.com/openconfig/reference/blob/master/rpc/gnmi/gnmi-extensions.md Given this is an arbitrary Protobyte payload, the default gNMI protobufs can't decode the payload contained within the field. This PR adds the necessary configuration options to load in an arbitrary Protobuf file per extension identifier, with a message-name to lookup the message type.

In this change:

  1. A new function DecodeExtension is added. This function uses protoreflect to dynamically marshal arbitrary protoBytes into JSON. The loaded JSON is then put back in the Extension message as bytes (mainting type)
  2. The Target type has an ExtensionProtoMap added, allowing for the lookup of Extension IDs to loaded-in protobufs
  3. The required changes to the TargetConfig type to support loading in the new configuration
  4. Modified collector.go to output the gNMI message after inlining the decoded protoBytes
  5. Loading in the protobufs was added to app/target.go: parseExtensionProtos. This uses the Parser provided by protoreflect/desc/protoparse
  6. Added functionality to event.go to insert the K/Vs provided by the Extension as Tags. Given we come from JSON, all numbers are float64, so the only 2 types supported currently are string and float64
  7. Minor helper function to turn the arbitrary JSON into an arbitrary map.

This has been tested with a device emiting an extension field:

[gnmic] target "edge01_test01": gNMI Subscribe Response: &{
SubscriptionName:port_stats
SubscriptionConfig:{"name":"port_stats","paths":["/interfaces/interface/"],"mode":"STREAM","stream-mode":"SAMPLE","encoding":"JSON","sample-interval":15000000000,"heartbeat-interval":15000000000,"outputs":["prom-scrape-output"]}
Response:
update:{timestamp:1723653363502452302  prefix:{elem:{name:"interfaces"}  elem:{name:"interface"  key:{key:"name"  value:"et-1/0/3"}}}
update:{path:{elem:{name:"state"}  elem:{name:"hardware-port"}}  val:{json_val:"\"FPC1:PIC0:PORT3\""}}
update:{path:{elem:{name:"state"}  elem:{name:"transceiver"}}  val:{json_val:"\"FPC1:PIC0:PORT3:Xcvr0\""}}}
extension:{registered_ext:{id:1
    msg:"{\"systemId\":\"edge01_test01\",\"componentId\":65535,\"sensorName\":\"sensor_1005_2_1\",\"subscribedPath\":\"/interfaces/interface/\",\"streamedPath\":\"/interfaces/interface/\",\"component\":\"chassisd\",\"sequenceNumber\":\"770002\",\"payloadGetTimestamp\":\"1723653363502\",\"streamCreationTimestamp\":\"1723653361858\",\"exportTimestamp\":\"1723653363504\",\"streamId\":\"PERIODIC\"}"}}}

Which is then properly rendered to a Prometheus metric:

gnmi_interfaces_interface_state_hardware_port{component="chassisd",componentId="65535",hardware_port="FPC1:PIC0:PORT3",interface_name="et-1/0/3",metric_source="edge01_test01",subscription_name="port_stats",systemId="edge01_test01"} 1

Note that some label-drop rules have been added to remove the spurious labels to avoid a cardinality explosion.

google-cla[bot] commented 4 weeks ago

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

Ichabond commented 4 weeks ago

This is a proposed solution for #503

Ichabond commented 4 weeks ago

Updated the commit to fix the type assertion issue as well as the variable assignment issue.

Ichabond commented 4 weeks ago

Anything else needed here @karimra ?

karimra commented 4 weeks ago

While this works for that junos extension, I'm not sure it will for any other extension. The data in an extension is not always meant to be added as tags, we should find a way to pass it on to the output and use processors to select some keys to set as tags. The other thing I'm not entirely convinced by is that these proto files are set by target. If you have 100s of targets of the same type (same extensions), we will have to parse the proto files 100s of times. This is fine for now, but it should be improved/fixed in a future PR.