Plugin should not compile all dependencies

[aselus-hub]

Right now I have a lot of library protos and grpc protos living in one place, the library protos are cross-used by multiple rpc protos, but they seem to not follow the output standard that protoc has for other languages.

Example:

root
├── util
│   ├── resource.proto
│   └── someotherutil.proto
├── boxes
│   ├── box_schematic_schemas.proto
│   └── boxes.proto
├── some_rpc
│   └── some.rpc.proto
├── box_rpc
│   └── box.rpc.proto
├── quadcopters_rpc
│   └── quadcopters.rpc.proto

the example above is made up on the fly, but this is similar to what I have today in my environment to work with (the server impls are written in golang using grpc) Now I want to make python libraries for the above, but when i call protoc betterproto generation on "bpx.rpc.proto" two problems happen: 1) betterproto doesn't generate paths, it sticks everything in root (other protoc plugins use . as reletive to the file) 2) all the utils are re-generated for each proto as opposed to use the ones that had been generated prior (my guess is because of 1

This is the real issue here is that we have a LOT of proto files and sometimes only generate some (e.g.: I've updated quadcopters, just want to re-generate that, not all the dependancies. Is this a missing features or am i missing something in betterproto that allows for doing standard generation w/o generating dependancies like in the other language examples?

[boukeversteegh]

I'm not sure I can answer all your questions, but I'll respond where I can:

1. betterproto doesn't generate paths, it sticks everything in root (other protoc plugins use . as reletive to the file)

since 2.0.0b1, betterproto will generate the target files according to package-structure of the proto files, not according to file-structure. Did you use this version? If your proto files do not have packages, you could generate them into a subfolder, or would this not work for your case?

If you really need betterproto to generate files according to the file-structure, I think we're going to need an option to switch between the two strategies. For that I suggest to make a proposal in a separate issue.

2. all the utils are re-generated for each proto as opposed to use the ones that had been generated prior (my guess is because of 1

Betterproto receives from protoc a list of all .proto files that were either passed into it explicitly via the commandline, or which are dependencies. At the moment we do not have any mechanism that distinguishes the dependencies from the explicit arguments. If this is actually possible, we could implement an option to toggle generating dependencies.

Do you know how other plugins do this?

[aselus-hub]

I'm using the version that pip3 downloaded as default(1.2.5) so that would be behind 2.x, is 2.x in pip somewhere? I tried pre and still got 1.2.5

All the other plugins i've dealt with only generate the protos passed to them. Some plugins handle import directories differently (for example go has ways to specify the import path explicitly via the "go_package" option, but by default uses relative path to the generation.)

Meaning if I say do the following:

cd protos
protoc -I. --gogoslick_out=plugins=grpc:. defs/boxes/box.rpc.proto

i will then have a file in protos/defs/boxes/box.rpc.pb.go, but it won't generate the util.pb.go file or any of the others (all the google generators, including python act this way by default). I would then need to generate those proto files themselves

An example for how this works differently ins oem plugins is the go_package options, if it is set then the package will get placed and included from that GOPATH relative directory[i can't find a python equivalent to this sadly, since we use FQ gopaths for where the proto files are], otherwise using the default as stated above

The file based format of 1.2.5 actually works well for me, i just honestly want to avoid re-generating the same file lots and lots of times(like util library)

[boukeversteegh]

Thank you for clarifying 😄

I'm using the version that pip3 downloaded as default(1.2.5) so that would be behind 2.x, is 2.x in pip somewhere? I tried pre and still got 1.2.5

v2 is published right now as a pre-release (see https://pypi.org/project/betterproto/2.0.0b1/). On my system it installs when I run pip install --pre betterproto. That doesn't work? You could also try the command mentioned on the website pip install betterproto==2.0.0b1

All the other plugins i've dealt with only generate the protos passed to them.

I see.. then we need to figure out how they do that. There are probably some flags passed to the plugin input, or perhaps the exact commandline parameters can be read.

An example for how this works differently ins oem plugins is the go_package options, if it is set then the package will get placed and included from that GOPATH relative directory.

I'm not sure I fully understand, as I'm hardly familiar with Golang. Does this refer to the generated import statements in the target .go files, that packages would be prepended to it?

The file based format of 1.2.5 actually works well for me, i just honestly want to avoid re-generating the same file lots and lots of times(like util library)

I understand the situation, it seems like an important option.

If you don't mind, I'll repurpose this issue specifically for excluding dependencies in the output.

To figure out:

• Identifying which parts of our analysis and compilation we can skip for dependencies. Further complicated by the fact that analysis and compilation are not cleanly separated in the code.
• Unclear how much of a performance gain we'd get from skipping compilation only for dependencies.

Question

• To be clear, is your main concern the duration of running the plugin? If so, how long does it take to run the plugin, in your specific case? Or is it an issue that the lib files are overwritten?
• Could you have a try to see if you could add a conditional around this statement
  https://github.com/danielgtaylor/python-betterproto/blob/8bcb67b66fdf22b1597c2db0f2180edd2ba045fb/betterproto/plugin.py#L352 to see if that would speed up things? For example, you could skip the file if the package starts with 'lib' or contains 'utils', or something like that, just as a proof of concept.

[boukeversteegh]

Also, feel free to join us on slack and ask for help 😃

[boukeversteegh]

Yes, please follow the link at https://github.com/danielgtaylor/python-betterproto#community 😄

[alexifm]

Hey, just wanted to chime in on this:

Betterproto receives from protoc a list of all .proto files that were either passed into it explicitly via the commandline, or which are dependencies. At the moment we do not have any mechanism that distinguishes the dependencies from the explicit arguments. If this is actually possible, we could implement an option to toggle generating dependencies.

In the generate_code function https://github.com/danielgtaylor/python-betterproto/blob/b5dcac125024c4579efc9096200a371f67c283f1/src/betterproto/plugin/parser.py#L68-L70

request.file_to_generate is the list of files that are explicitly passed to protoc: https://github.com/protocolbuffers/protobuf/blob/e667bf6eaaa2fb1ba2987c6538df81f88500d030/src/google/protobuf/compiler/plugin.proto#L68-L73

Another conditional could be added here to just build if the files are in the list request.file_to_generate: https://github.com/danielgtaylor/python-betterproto/blob/b5dcac125024c4579efc9096200a371f67c283f1/src/betterproto/plugin/parser.py#L85-L94