feature: self contained, auto installing protoc and programatic usage from `build.zig`, enable zig package manager

[menduz]

This PR introduces a bash-less, cross platform self-contained installation of protoc for each OS and architecture.

How to use

1. Add protobuf to your build.zig.zon.

   .{
       .name = "my_project",
       .version = "0.0.1",
       .dependencies = .{
           .protobuf = .{
               .url = "https://github.com/Arwalk/zig-protobuf/archive/<some-commit-sha>.tar.gz",
               .hash = "00000000000000000000000000000000000000000000000000000000000000000000",
               // leave the hash as zeroes, the build system will tell you which hash to put here based on your commit
           },
       },
   }

2. Use the protobuf module

   pub fn build(b: *std.build.Builder) !void {
       // first create a build for the dependency
       const protobuf_dep = b.dependency("protobuf", .{
           .target = target,
           .optimize = optimize,
       });
   
       // and lastly use the dependency as a module
       exe.addModule("protobuf", protobuf_dep.module("protobuf"));
   }

Generating .zig files out of .proto definitions

You can do this programatically as a compilation step for your application. The following snippet shows how to create a zig build gen-proto command for your project.

const protobuf = @import("protobuf");

pub fn build(b: *std.build.Builder) !void {
    // first create a build for the dependency
    const protobuf_dep = b.dependency("protobuf", .{
        .target = target,
        .optimize = optimize,
    });

    ...

    // create the "zig build gen-proto" step
    const gen_proto = b.step("gen-proto", "generates zig files from protocol buffer definitions");

    // create the compilation step to build the protocol buffers
    const protoc_step = protobuf.RunProtocStep.create(b, protobuf_dep.builder, .{
        .destination_directory = .{
            // out directory for the generated zig files
            .path = "src/proto",
        },
        .source_files = &.{
            "protocol/all.proto",
        },
        .include_directories = &.{},
    });

    // and make the gen-proto step depend on it.
    gen_proto.dependOn(&protoc_step.step);

    // alternatively, you can make your entire executable or test depend on the generation of the sources
    // test.step.dependOn(&protoc_step.step);
    // exe.step.dependOn(&protoc_step.step);
}

[menduz]

Think it would be possible to avoid running protoc every build?

something around the lines of

if file not generated:
  calc hash of .proto file
  generate file from .proto
  save hash somewhere
else:
  if hash of .proto is still the same:
    do nothing
  else:
     regenerate, save new hash

i think blake3 is available in std for hashing?

forcing generation would be made by deleting the generated file, or maybe having a way to pass a --force flag in the RunProtocStep ?

I thought about it, but since .proto files may include files from different locations, it may be very complex to traverse all dependencies to check for changes. That's the reason why the example of the readme proposes a "generation step" that can be optionally added as a dependency to your project, and not vice-versa

[HurricanKai]

This is great! Personally I'd probably prefer a config where this is output into some temp directory, but I think this should be possible also?

I'm not a fan of checking generated code into the repo. Some extra logic to like checking hashes should be easy to implement in build.zig. I'd probably make a config where all the generated files are output somewhere temporary and made available as a separate module. That way protobuf might not even need to be a direct dependency, and only the generated module is referenced.

[menduz]

This is great! Personally I'd probably prefer a config where this is output into some temp directory, but I think this should be possible also?

I'm not a fan of checking generated code into the repo. Some extra logic to like checking hashes should be easy to implement in build.zig. I'd probably make a config where all the generated files are output somewhere temporary and made available as a separate module. That way protobuf might not even need to be a direct dependency, and only the generated module is referenced.

That's for sure a desired behavior, you can do it via setting the out dir to somewhere inside zig-cache and then creating a module pointing that folder. But the same complications appear in knowing which file is the entry point. Protobuf allows you to set multiple input and multiple output files, and modules require only one entry point. Moreover, modules hide every file that is not the entry point, making the N+1 output files not accessible. Making assumptions here may be risky, I think that the safest option here is to offer an output path option and then allowing each developer to create their own modules if needed.

This repository checks out the generated code to track changes in the code generation only.

Needless to say, in my own projects the code is not tracked in git and I do use a single module as you suggest, because I have only one entry point with several import public in the .proto, but that's not everyone's scenario

[Arwalk]

Think it would be possible to avoid running protoc every build? something around the lines of

if file not generated:
  calc hash of .proto file
  generate file from .proto
  save hash somewhere
else:
  if hash of .proto is still the same:
    do nothing
  else:
     regenerate, save new hash

i think blake3 is available in std for hashing? forcing generation would be made by deleting the generated file, or maybe having a way to pass a --force flag in the RunProtocStep ?

I thought about it, but since .proto files may include files from different locations, it may be very complex to traverse all dependencies to check for changes. That's the reason why the example of the readme proposes a "generation step" that can be optionally added as a dependency to your project, and not vice-versa

Fair enough. Can we at least make the protoc command printing from line 216 to line 220 optional as some kind of verbose option in the protoc gen target? It feels mostly useful for debugging if something went wrong on the generation side.

[Arwalk]

This is great! Personally I'd probably prefer a config where this is output into some temp directory, but I think this should be possible also?

I'm not a fan of checking generated code into the repo. Some extra logic to like checking hashes should be easy to implement in build.zig. I'd probably make a config where all the generated files are output somewhere temporary and made available as a separate module. That way protobuf might not even need to be a direct dependency, and only the generated module is referenced.

Another solution instead of using a temporary directory in zig-cache would be to rely on gitignore configuration to avoid tracking generated files. Put your generated files in the directory you want, don't track files in it, and you're done.

As pointed by @menduz it's not that easy to manage dependency in the .proto files. But if you feel like tackling le subject, we'd love your contributions! 💪🏻

Thank you for your feedback and interest.

[Arwalk]

@menduz merged and added https://github.com/Arwalk/zig-protobuf/commit/ea5b5885930059c0661d076ea70b1761fe0eee8f to make protoc commands silent by default.