Add support for automatic protoc

[jannickj]

Would be really nice if you could just parse in -P file.proto and it would just generate the fdset on the fly if it doesn't exist.

[sevagh]

I wonder if I could do that programatically or I'd need pq to shell out to protoc.

[farzadmf]

+1 if we can call pq directly on a .proto file :slightly_smiling_face:

[sevagh]

I tried exploring this ages ago: https://github.com/stepancheg/rust-protobuf/issues/210#issuecomment-284266769

[jannickj]

@sevagh and what was the conclusion? :)

[sevagh]

Note, rust-protobuf does not have own .proto parser. So you need to obtain a list of FileDescriptorProto somehow, e. g. by invoking protoc --descriptor_set_out which outputs FileDescriptorSet message.

Looks like it could be possible now: https://github.com/stepancheg/rust-protobuf/blob/442fc304c1a7511292cf6336db35e70bd6f959be/protoc-rust/src/lib.rs

[sevagh]

I generally have a pretty low desire to write Rust code these days. As always, PRs are welcome.

[sevagh]

I just learned that https://github.com/yogeshsr/kafka-protobuf-console-consumer supports proto files directly, so maybe this feature request is more important than I thought.

[sevagh]

That's not so bad:

sevagh:pq $ cat tests/samples/dog | ./target/debug/pq --protofile ./utils/dog.proto --msgtype com.example.dog.Dog
{
  "breed": "gsd",
  "age": 3,
  "temperament": "excited"
}

I'm always using the path std::env::temp_dir().join("tmp-pq.fdset") to store the recompiled proto. I use protoc to compile it, borrowing ideas from prost-build: https://github.com/danburkert/prost/blob/master/prost-build/src/lib.rs

I'm mostly concerned about OSX or Windows support, since I can only test my code on Linux.

[sevagh]

Implementation here: https://github.com/sevagh/pq/compare/feat/protofile Features:

• Set PROTOC to use your own binary, otherwise look for protoc in PATH
• Set PROTOC_INCLUDES to set protoc include paths
• Invocation is protoc --include_imports --include_source_info -o /tmp/tmp-pq.fset /path/to/proto -I="$PROTOC_INCLUDE"

[sevagh]

1.4.2-rc0: https://github.com/sevagh/pq/releases/tag/1.4.2-rc0 Check it out, let me know if it works for you:

sevagh:pq $ ./pq --version
pq 1.4.2-rc0
sevagh:pq $
sevagh:pq $ ./pq --protofile ./tests/protos/dog.proto --msgtype com.example.dog.Dog <./tests/samples/dog
{
  "breed": "gsd",
  "age": 3,
  "temperament": "excited"
}

[farzadmf]

Sorry if this is a stupid question, what's the tests/samples/dog we're passing here? Is it something that needs to be created from the proto file? If that's the case, how can I do that?

[sevagh]

That's an actual protobuf encoded message with a dog inside it, to test the decoding. You should use pq the way you normally do (with --protofile), the above is just me using the test sample files to prove that it works.

What I put above is equivalent to this, using cat and pipe instead:

sevagh:pq $ cat -A tests/samples/dog
$
^Cgsd^P^C^Z^Gexcited
sevagh:pq $ cat tests/samples/dog | ./pq --protofile ./tests/protos/dog.proto --msgtype com.example.dog.Dog

[farzadmf]

Ah OK, and sorry again if it's another stupid question, but how would you create that file "statically"? I would think of writing a program to generate the binary and write it to a file, but is there another way to do that?

[sevagh]

In this repo I created them with the Python script in the util dir. It's a bit ugly but:

    obj = dog_pb2.Dog(
        age=random.choice(range(0, 20)),
        breed=["rottweiler", "gsd", "poodle"][random.choice(range(0, 3))],
        temperament=["chill", "aggressive", "excited"][random.choice(range(0, 3))],
    )

    obj = obj.SerializeToString()
    sys.stdout.buffer.write(obj)

[farzadmf]

OK, cool, that makes sense :slightly_smiling_face:

[sevagh]

I added this feature back then, documented in the README: https://github.com/sevagh/pq#usage