Discussing the inclusion of a helper binary in ocaml-protoc for automated rule generation

[mbarbin]

I'm quoting an exchange we had with Simon in #231:

@c-cube:

maybe` ocaml-protoc should also ship with a small binary (ocaml-protoc-genrules?) that takes a list of .proto files as arguments as well as arguments for ocaml-protoc itself, and spits out a dune.inc file that contains the rules for these particular modules? Like:

ocaml-protoc-genrules
  foo.proto bar.proto --pp --binary -I dir

produces a file with:

; dune.inc

(rule
  (targets foo.ml foo.mli)
  (deps (:file foo.proto))
  (action ocaml-protoc %{file} --pp --binary --ml_dir=.))

; … same for bar

What do you think?

Here is my response:

I support this idea. In fact, I've implemented a similar concept in a project of mine, calling it "protoc-helper". At the time, I didn't consider proposing it for upstream inclusion, but having something like that in the distribution makes sense to me.

My implementation cannot be directly reusable due to its dependency footprint, but I'm including it here as a data point.

let main =
  Command.basic
    ~summary:"generate dune stanza for all *.proto files present in the directory"
    (let%map_open.Command exclude =
       flag
         "--exclude"
         (optional_with_default [] (Arg_type.comma_separated string))
         ~doc:"FILE[,..]* file to exclude"
     in
     fun () ->
       Eio_main.run
       @@ fun env ->
       let files =
         Auto_format.find_files_in_cwd_by_extensions
           ~cwd:(Eio.Stdenv.fs env)
           ~extensions:[ ".proto" ]
         |> List.filter ~f:(fun file -> not (List.mem exclude file ~equal:String.equal))
       in
       let generate_rule ~file =
         let open Sexp in
         let list s = List s
         and atom s = Atom s in
         let basename = Fpath.rem_ext (Fpath.v file) in
         let include_services =
           String.is_suffix (Fpath.to_string basename) ~suffix:"_service"
         in
         list
           [ atom "rule"
           ; list
               [ atom "targets"
               ; atom (Fpath.add_ext "ml" basename |> Fpath.to_string)
               ; atom (Fpath.add_ext "mli" basename |> Fpath.to_string)
               ]
           ; list [ atom "deps"; list [ atom "glob_files"; atom "*.proto" ] ]
           ; list
               [ atom "action"
               ; list
                   [ atom "run"
                   ; atom "ocaml-protoc"
                   ; atom file
                   ; atom "--binary"
                   ; (if include_services then atom "--services" else atom "--yojson")
                   ; atom "--ml_out"
                   ; atom "."
                   ]
               ]
           ]
       in
       Eio_writer.with_flow (Eio.Stdenv.stdout env)
       @@ fun stdout ->
       Eio_writer.printf
         stdout
         "; dune file generated by '%s' -- do not edit.\n"
         (Sys.get_argv ()
          |> Array.to_list
          |> List.mapi ~f:(fun i s -> if i = 0 then Fpath.v s |> Fpath.basename else s)
          |> String.concat ~sep:" ");
       List.iter files ~f:(fun file -> Eio_writer.print_s stdout (generate_rule ~file)))
;;

Here's how it can be used in a dune file:

(rule
 (deps
  (glob_files *.proto))
 (action
  (with-stdout-to
   dune.proto.inc.gen
   (pipe-stdout
    (bash "%{bin:protoc-helper} gen-dune")
    (run dune format-dune-file)))))

Note: I used dune format-dune-file to avoid worrying about formatting during generation. This requires the pipe-stdout construct, introduced in dune.2.7, so may need to update your dune-project config to use it.

I suspect that the dependency computation in my implementation might be flawed. I was concerned about whether the compiler needed to revisit a file, say dep.proto, while compiling another file, a.proto, that depends on dep.proto. As a result, I made all invocations depend on all proto files systematically. This might be an overkill and potentially unnecessary.

[c-cube]

Indeed I was more thinking of something without dependencies (besides the ocaml-protoc compiler-libs perhaps). :-).

I suspect it'd be possible to use the ocaml-protoc compiler-libs to compute the actual dependencies based on import statements. But it's starting to be some work.

[Lupus]

Why not just glob proto files in dune recursively (typically they are organized as a hierarchy corresponding to package names etc), and produce one output by ocaml-protoc as the result, with all the modules scoped by corresponding packages/filenames?