Request for comments: Subprocess plugin API

[alexeagle]

I originally raised this on https://github.com/google/cargo-raze/issues/195 as a suggestion for how Rust generates BUILD files.

@achew22 paired with me to start on a prototype for a Gazelle plugin for a fake language called "rosetta". This plugin is configured with a binary to run, which is executed as a subprocess. We mirrored the gazelle plugin API to protobuf so that we can communicate with this subprocess over stdio. Here was our WIP

The advantage of doing this is that plugins can be written in any language. This isn't just out of vanity of course, there are APIs that a BUILD file generator may need to use. For example, to generate TypeScript projects, we need to use the typescript API which is only available in a nodejs runtime. POC in a gazelle-like nodejs tool

@jayconrod this would make development of gazelle a little trickier. You either need to generate the Go API from the protobuf one, or keep the two of them in sync. However I think it would open up the plugin ecosystem to a wider audience. For another example, for rules_python we are thinking of a basic generator written in Python.

[achew22]

I've poked at this a little more and gotten things to a place where there is a simple channel in the gazelle side of things so you can just select on the read channel and write out into a buffered channel quite simply. This was in effort to validate that communicating in this style is possible, which I believe I have validated, but leaves open the question of "is this the best way to address this?"

The parts I would ask you to focus on would be the listener/sender loops and my example for sending/receiving from the subprocess. Thanks so much!

[jayconrod]

Tentatively, I think this would be great to have.

If this were implemented, would one or both of you be open to maintaining it long term? I'm unfortunately struggling to find time to maintain rules_go and Gazelle, so I'm not able to handle any scope expansion myself.

Some early thoughts:

• Protobuf over stdin/stdout pipes seems like a decent way to do this. JSON would be another option. For the Gazelle side of the pipe, I'd say the protobuf code should be pre-generated so folks can use Gazelle without needing to build protoc.
• For external dependencies, it would be difficult to make this work inside a generic gazelle_repository rule. You'd probably need a specialized repository rule for each language.
• It may not always be possible to serialize rules. Private attributes can have any value and might not be marshalable. That's probably fine 99% of the time though.
• At some point, Gazelle may parallelize work across directories, so the protocol should be able to handle responses coming back in a different order than requests. That probably just means including a serial number with each message.

[asuffield]

Similar theme: I'd like to have a plugin that does this for java, and this is only practical when run on a JVM, because of the need to execute annotation processors.

In that case, it is inevitable that some developer configuration would need to be passed in, for at least some subtrees: the determination of what is a java compilation unit is something developers actively control, either via pom.xml or source layout. This also means that not every subdirectory will be a package. That has interesting implications for how the API would work.

[alexeagle]

I would hope that affordances for user customization continue to be plumbed through the directives-in-BUILD-comments scheme that gazelle uses already.

[UebelAndre]

Hey, any updates here?

[sluongng]

There seems to be a new bounty set on this issue. I would appreciate if whoever is aiming for the bounty discuss their design here first. It sounds like the previous discussion was leaning toward something similar to https://bazel.build/remote/persistent#implementation

[alexeagle]

I'm hoping to convince @pcj who had some WIP on this at one point as well. Happy to discuss design more - my choices here were based on @achew22 suggestion.

[pcj]

I may work on this. I can prepare a more detailed design document if we decide to move forward. At a high level:

• New extension in language/plugin that delegates the core gazelle extension lifecycle methods to a subprocess plugin (similar to the rosetta prototype by @achew22)
• Plugin API defined by a plugin.proto whose message request / response types resemble the serializable parts of the arguments / return values of the existing extension API. PluginRequest and PluginResponse would be the high-level wrapper types that contain specific method arguments / return types.
• Plugin extension would offer a flag like --gazelle_plugin=NAME=PATH_TO_BINARY that registers a plugin executable under a language name.
• Plugin extension would be responsible to launching the binary at startup
• Communication between the plugin extension (client) and a plugin implementation (server) over JSON-or-raw-proto serialized stdin/stdout (alt JSON-RPC vs gRPC vs hashicorp/go-plugin).