questions + related packages

[timotheecour]

see also: https://github.com/PMunch/protobuf-nim/issues/1 in which I list other related nim packages

just wondering whether it would make sense to consolidate efforts with @PMunch or whether these have different goals/designs.

[PMunch]

I would think the goal is pretty much the same. My implementation is double pure though as it doesn't rely on the protoc compiler to create any code for it. Whether that's a good or a bad thing is up to the user. So yeah, pretty much the same goal I'd say, but a bit different design.

[oskaritimperi]

Dang, I didn't know there was another protobuf library for Nim. Could've done more googling! I only looked at nimble packages index and didn't find it from there.

[PMunch]

To be fair my library only became usable in the past couple of days. Before that it's mostly been parsing the protobuf grammar and reasoning about how the interface should be designed. So it's not that strange you hadn't heard about it until now 😃 And since I wrote a new parser library for it as well, which still isn't on Nimble, I haven't put my protobuf library up either. But this will probably happen in the next couple of days.

[oskaritimperi]

As I mentioned in https://github.com/nim-lang/packages/pull/694, these packages now have a bit different way of working. @PMunch's library is pure Nim, no need for an external .proto -> .nim compiler. My library is now completely depending on protoc doing the .proto -> .nim translation.

I think it's okay to have two completely different packages for different situations. Though we could think of making the user-facing APIs similar so that users don't have too much trouble if they want to switch from one library to the other.

[timotheecour]

Though we could think of making the user-facing APIs similar so that users don't have too much trouble if they want to switch from one library to the other.

definitely good, wherever it makes sense (eg doesn't deteriorate API)

Honest question, in which situation would one prefer to use protoc approach as opposed to @PMunch 's compile time direct approach? These are dimensions I can think of:

• compile time speed: IIRC direct approach would require redoing the compile time parsing (parseProto) every time a module that uses that proto is modified, even if the proto is not modified; whereas protoc approach would generate the nim files just once; so potentially that could be large speedups for large proto definitions? Could that be measured on large synthetic protos?
• runtime performance: any benefit here?
• anything else?

[oskaritimperi]

The only things that come to my mind are

• the first bullet of your list
• if someone has a workflow where she is already generating code for other languages with protoc (not a big deal I think)
• if someone wants to generate the code once, commit it to version control and make sure that it stays as is always or if there is some hand-tuning that needs to be made

I don't think there's any runtime performance issues if the libraries are otherwise comparable.

The largest protobuf I know of is the conformance suite. That generates a lot of code. We could maybe time that. It includes the conformance suite messages themselves and IIRC all of the well known types also. In total it's around 11k generated lines, if using my library.

[PMunch]

protoc is definitely faster than my library, by large margin. That's mainly caused by the parser though, and if I either optimise that, or switch it out entirely there would be less of a gap.

I'd say that runtime speeds would probably be about the same. It of course depends on how the two libraries implement various things, but there shouldn't be any inherent flaw in either approach. The only thing I can think of is that since @oswjk seems to now switch 100% to using protoc for the generation it would mean that he's at the mercy of that compiler to generate optimised code. Theoretically it could be possible for my implementation to use some Nim-specific optimisations to make it faster, but I haven't done any thus far.

The difference is mostly in workflow. With my library it's a simple Nimble install, and run. With this library you would need to install protoc for your OS, set it up with this extra plugin, and then compile and run. Of course if you have a project that is merely using protobuf it wouldn't be an issue to just check in the generated code file (a bit messy, I tend to not wont auto-generated stuff in my repos), but to make any changes you would have to do the aforementioned steps.

[oskaritimperi]

I'm closing this issue. Some things that have happened meanwhile:

• nimpb can be used with simple nimble install, as I've bundled protoc compiler for different platforms in https://github.com/oswjk/nimpb_protoc
• almost all of conformance suite passes for protobuf3 (json input/output, protobuf input/output and mixes of those)
• almost all WKTs are at least somewhat implemented
• service generation hook allows a third party library to generate service code (an example: https://github.com/oswjk/nimtwirp)
• a nimpb_build command line tool for compiling .proto -> .nim, that can use an existing protoc or the protoc from nimpb_protoc package

I think I've gotten the flow to a quite nice shape. It's true that it's still not as convenient as generating the code at compile time, but I'm mostly happy with it, which is the main thing ;-)