Support for Protocol Buffers

[weiyen-rokt]

While it looks like k6 has support for load-testing gRPC services, it is less clear to me whether k6 supports load-testing with just protocol buffers, minus gRPC.

Feature Description

Support for serializing and deserializing protocol-buffer data

[na--]

Can you elaborate a bit more about what exactly your use case is? You want to serialize protobuf data with k6, but send it as the body of a normal http/https call instead of via gRPC?

[weiyen-rokt]

That's right. We've got a regular http endpoint that can send and receive protocol-buffer payloads that we wanted to load test.

[kieoo]

+1

[na--]

Please do not leave "+1" comments, just add a :+1: to the original post. That way we won't have spam in the issues and they can be sorted by the number of :+1: they have.

[tom-miseur]

See here for an example using pure JS. It has not been tested against an actual deployment yet.

[tom-miseur]

I've since tested this against a simple Go server. Have a look at it here: https://github.com/tom-miseur/test-k6-protobuf.js

[shekharshukla]

Hi @tom-miseur Thanks for posting the solution. I am able to mimic a server as you have explained and sending request using protobuf js. However it's not working on my actual server. In my actual server I am able to send protobuf request using Go code by generating the data and then using proto.Marshal for marshalling and sending it using Go http client and it works. But it's not working with K6 when I am trying to do the way you suggested. The difference i see in encoded data is Go generates uint8 array of type [8 10 15] while javascript generates buffer of type {"0": 7, "1":8} something like that

[mstoykov]

@shekharshukla it will be beneficial if we have a proto file that reproduces this problem and preferably a js script as well.

You can for example make a PR ot the repo linked above with that do showcase how it isn't working

[jonaskello]

I would also like to have possibility to encode/decode protobuf. I'm using NATS for communication and there is an extension for that however NATS is payload agnostic and our specific service use protobuf. I think this would be useful for any protocol even for http if you use protobuf over http. Perhaps the grpc plugin could expose encode/decode of protobuf as separate calls as it seems to be already present there.

[na--]

Yeah, k6 probably imports all of the dependencies that would be required for this already, due to the k6/net/grpc module. So that is unlikely to be an obstacle for exposing protobuf encoding/decoding JS APIs :tada:

The biggest problem would be to design these JS APIs so they are nice to use. We can hopefully avoid most of the potential footguns if we base the APIs on some already existing JS library though :thinking: I found these two with a few minutes of searching, though there are probably others:

• https://github.com/protobufjs/protobuf.js/ (docs at https://protobufjs.github.io/protobuf.js/)
• https://github.com/protocolbuffers/protobuf-javascript

I don't have experience with either, but if someone wants to do some research and port the best existing JS protobuf API to k6 (i.e. goja), the k6 team will likely try to help you out a bit and get it merged as a k6/experimental API. And, if not, :+1: the issue, so we can prioritize it higher on our own side.

[danielnaumau]

@na-- Do you have any ideas when this feature can be implemented?

These libs which you mentioned are not very compatible with k6. So it would be quite nice if k6 supported Protocol Buffers.

[na--]

Sorry, I can't give you an estimate for when this will be available. I'll discuss it with the rest of the team, since it seems like it should be relatively easy to make at least an experimental/ API for this soon, but also have quite a few other high-priority things we are busy with. The best bet for getting this sooner is still for someone to make a PR.

[danielnaumau]

I created a xk6 plugin to work with protobuf https://github.com/traveltime-dev/xk6-protobuf. It decodes proto bytes to json string and vice versa. So maybe someone will need it.

[thenakulchawla]

@danielnaumau does it also support the protoset binary format?

[danielnaumau]

@thenakulchawla unfortunately no. When I was implementing this plugin, I didn't really need this format.