Add protobuf text and binary formatted example data

[nuclearpidgeon]

As part of some work on the Fluentbit project (fluent/fluent-bit#8964, fluent/fluent-bit#8734), I needed some sample OpenTelemetry data in protobuf/gRPC format for adding some tests.

I translated the sample JSON files by hand to the Protobuf text format and then compiled these text representations into binary files as well using protoc --encode. They've worked for me so far in some cURL testing of the Fluentbit opentelemetry input plugin.

For now I have done the metrics and trace samples, will try do logs next to leave a complete package before closing the PR. (Probably need to update the examples README.MD too.)

[linux-foundation-easycla[bot]]

• :x: - login: @nuclearpidgeon / name: Stewart Webb . The commit (f6bb64114ae482bf52712683d740bc70baadb6e3) is not authorized under a signed CLA. Please click here to be authorized. For further assistance with EasyCLA, please submit a support request ticket.

[yurishkuro]

I suggest keeping only one representation in source control. If we already have json, the other formats can be trivially generated on the fly.

[nuclearpidgeon]

It was pretty non-trivial for me to get these samples together for use with cURL to test the protobuf-encoded side of the Fluentbit Otel input plugin. (The plug-in is still missing code paths for handling JSON input for metrics and traces)

If you know a better/easier way to convert the JSON to a compiled protobuf message (without having to write any custom program or script that pulls in protobuf libraries) then I guess I can just update the README here with that. I tried using grpcurl but didn't seem to be able to get anywhere successful with it.

[yurishkuro]

without having to write any custom program or script that pulls in protobuf libraries

what's the problem with protobuf libraries? This repo already generates all kinds of classes as part of the CI, and it should be fairly trivial to write a small Go program to do the conversion using those classes (or by importing pdata from otel/collector).

It was pretty non-trivial for me to get these samples together

even more reason to have them auto-generated. What if the input files need to be changed, how would the next person go about generating the binary files?

[nuclearpidgeon]

I just didn't really want to have to write (and publish/maintain?) a whole random one-off Go program (and work out its file I/O APIs again, and the protobuf serialisation APIs) just to generate translated sample data into the protobuf format for testing purposes. If it really is trivial to translate the data from JSON to protobuf then I'm surprised that I couldn't seem to find a CLI tool that would do it in the same one-liner manner as the protoc usage I've got as comments at the top of the protobuf text format files here.

In the context of fluentbit as well, the tests are run in CI so if I wanted any other dependencies (Go, scripting language with protobuf libraries) it would have to be added as a dependency to the docker build image which seems excessive just for one small part of the Fluentbit codebase. (even including grpcurl as a dependency might have to be justified to the fluentbit project).

Perhaps I need to move over to the protoc repo repo and submit a PR for JSON input and output for the --decode and --encode flags...

[nuclearpidgeon]

OK so I spent some more time fiddling with grpcurl and was finally able to get it to send the JSON into Fluentbit as gRPC. From the root of the opentelemetry-proto repo, the required command is the following:

cat examples/metrics.json | grpcurl -proto opentelemetry/proto/collector/metrics/v1/metrics_service.proto -d @ -v -plaintext localhost:4318 opentelemetry.proto.collector.metrics.v1.MetricsService.Export

(I think part of the issue may have been that grpcurl defaults to using TLS, and also perhaps me messing up the last argument (has to be the RPC method name not the message type).) I can perhaps submit a PR adding this to the README.md in the examples folder

The only thing this doesn't still cover is the case of deliverying protobuf data over HTTP with Content-Type: application/x-protobuf, which was the initial use case I wanted the protobuf binary data for.

If the README.md were to recommend a way to test this particular delivery method in a generated manner based off the JSON... what approach would you suggest?

Actually now that I think of it, maybe using the otel collector itself as the translator and network payload deliverer might be the easiest approach...

[joaopgrassi]

Actually now that I think of it, maybe using the otel collector itself as the translator and network payload deliverer might be the easiest approach...

When I initially added the JSON examples here, that was what I was doing. I wanted to test a OTLP http/proto receiver but crafting the protobuf message was "complicated" - I either had to use a app, install the SDK and export or I install the proto libraries and manually craft the OTLP request, which either is some amount of work.

With the JSON, you can simply send it to the collector via curl and let the collector translate it for you. For what I was using it was great because I could iterate fast, try many combinations of OTLP data.