Implement the ProtoBuf format

[zah]

The ProtoBuf format is heavily used in LibP2P. We have implemented a set of low-level routines in https://github.com/status-im/nim-libp2p/blob/master/libp2p/protobuf/minprotobuf.nim. The goal of this bounty is to integrate them nicely with the high-level API of nim-serialization.

[jyapayne]

I'm interested in taking this one. Can you give a bit more details on the requirements?

As for my Nim ability, I've been programming in it on and off since 2015 and have recently been adding Category support to the Nim forum, which you can view on my fork and the main nimforum repo for an idea of my skill level.

[jangko]

Great, this is what you can do in no particular order:

• Take a look at json-serialization for inspiration.
• The serializer will produce the protobuf binary wire format and doesn't care about protobuf schema language.
• Start pushing commits into this repo.
• Setup the CI - and use Nim compiler from nimbus-build-system.
• Publish it as nimble package -- give it a name: protobuf-serialization for consistency.
• It should at least pass generic test suite.
• You may need to implement some private api and spec compliant test suite.
• A decent documentation would be great.
• The order of the serialized fields or key-value are simpler to implement, you can use default Nim behaviour. But when decoding them, the deserializer need to support any order permitted by the spec.
• All basic Nim datatypes should be supported. Some common data structure such as Table may need to be included.
• If you encounter problems with ref object, it is a known problem with nim-serialization or a Nim limitation.
• add some tests involving libp2p datatypes will be nice.

[jyapayne]

Great, that all sounds good and like a healthy chunk of work. I'll have lots of questions about some of the above tasks if you choose me to work on this, which I will ask on your Discord server once this task starts on March 10th.

Looking forward to seeing what the bounty estimate is and collaborating with you guys!

[zah]

Thank you, @jangko, the provided list of steps is a great start.

@jyapayne, before beginning work on this, you must understand the context where the newly developed library will be used. Currently, in nim-libp2p, we are relying on a low-level set of ProtoBuf routines defined in minprotobuf.nim. I would start by adapting this code to use nim-faststreams. This should be a straight-forward refactoring that replaces the buffer-indexing code with calls to stream.append. Some very similar refactoring has been carried out in our snappy package as an example.

With the help of the new low-level routines, you can create reader and writer modules by following the example in nim-json-serialization.

Finally, you'll hunt down all the usages of minprotobuf in nim-libp2p and you'll replace them with the new high-level code.

Please note that the users of this library will need to provide only Nim type declarations that match the desired ProtoBuf schema (it wouldn't be necessary to write .proto files). Consuming .proto files is outside the scope of this bounty, but if you wish to deliver it you'll find some existing packages that can parse .proto files with compile-time Nim code or through protoc. Since both packages transform the ProtoBuf definitions into regular Nim type declarations, it should be possible to reuse their code in nim-protobuf-serialization.

Please don't hesitate to get in touch on Discord to get additional guidance. The goal of this bounty is to cover our ProtoBuf-related needs in nim-libp2p and to lay out the foundations for creating a completely comprehensive package down the road. We'll be happy to fund a week of development towards this goal (3 days for creating nim-protobuf-serialization and 2 days for the integration in nim-libp2p, 1200 USD in total).

[jyapayne]

@zah Thank you. The information you laid out is well organized and what needs to be done looks straightforward. I will definitely follow up on Discord should I encounter any confusion or simply have questions for you.

Just for clarification of the process (in my own words), as @jangko mentioned, I just need to be filing PRs to the nim-protobuf-serialization repo in order to get started, correct? If that's the case, I will get ready to file PRs on that repo starting March 10th.

Great, thank you for the estimate on one week of funding. One point to add is that I will be doing this work in the evenings (in MST) and will only be able to spend 2-4 hours per night on this, plus some time on the weekend. Which means it should take me around 2 weeks to do this one week of work, if that's alright with you.

[jangko]

I just need to be filing PRs to the nim-protobuf-serialization repo in order to get started, correct?

correct

[zah]

Great, thank you for the estimate on one week of funding. One point to add is that I will be doing this work in the evenings (in MST) and will only be able to spend 2-4 hours per night on this, plus some time on the weekend. Which means it should take me around 2 weeks to do this one week of work, if that's alright with you.

This is certainly alright. Also, if at any point you start feeling that the initial estimation was not accurate, don't hesitate to bring this up. We all know that estimating software development upfront is notoriously hard :)

[jyapayne]

Sure thing! I'll definitely let you know ;)

[jyapayne]

@jangko or @zah please add the first commit to https://github.com/status-im/nim-protobuf-serialization because I cannot fork an empty repo.

[zah]

@jyapayne, done!

[jangko]

protobuf-serialization already implemented, closing.