implement bazel as build system

[tgummerer]

Use bazel as build system. Right now it's working side by side with the old Makefiles, and we're adding an extra CI job to run the builds. Eventually this can completely replace the Makefile based builds.

(I'll comment inline on some of the changes required for this)

[tgummerer]

As someone who hacks on multiple Pulumi repos frequently, my biggest concern is that this could make that more difficult if I currently use Go Workspaces. It seems like we'd need to keep the makefile and have two build systems to make that work.

I'm really curious what your workflow for this is! Do you have a go.work file that lives outside of the repos and ties them all together in some way? Do you run the make commands to do some setup inside the repo and then run go command outside, tied together by the Workspaces?

One of the major advantages of Bazel (not necessarily in this repo, but once we have more repos set up with it) is that it supports multiple languages. E.g. currently pulumi-dotnet uses a completely different build system. pulumi/pulumi uses makefiles that run various tools for each language, and the targets are not very fine grained, which makes it tricky to work with, and the onboarding experience honestly quite bad (In my 6 months working on it I've never been able to just run make in the repo and have it build everything).

[AaronFriel]

Yeah, I use a go.work file in a parent directory to all of my GitHub packages - though I will sometimes need to set GOWORK=off when working outside of those. The current contents are:

go 1.21.0

use (
        ./pulumi-java/pkg
        ./pulumi/pkg
        ./pulumi/sdk
        ./pulumi/tests
)

And as a result, everything I expect works in my dev tools, including (importantly) running go-to-definition, tests, process debugging. I can make install in the Pulumi repo, make provider in a provider repo, and so on.

[tgummerer]

And as a result, everything I expect works in my dev tools, including (importantly) running go-to-definition, tests, process debugging.

This will continue working with or without the makefiles. There's no changes in the Go tooling here, and I expect us to be able to continue running tests outside of the build system similar to how that's currently possible. In fact Gazelle expects the repo to follow the Go conventions to work, so there's no issue here.

The only difference would be here:

I can make install in the Pulumi repo, make provider in a provider repo, and so on.

make install would either be replaced by a script, or calling bazel directly, or we could even have make install just call bazel for backwards compatibility. So really the only difference would be that you need to have bazel installed. The intention is the have everything else in the dev workflow you're describing still working except without make, but with bazel instead.

[AaronFriel]

That's a relief and I appreciate your humoring my questions! Just want to make sure I understand the limitations, if any.

Re: make install changing, it sounds like, however, for end to end testing - building Pulumi YAML and Pulumi CLI using my workspace source - this might not work as expected? Or I would need to learn the "bazel way" to link together multiple modules from source?

[tgummerer]

That's a relief and I appreciate your humoring my questions! Just want to make sure I understand the limitations, if any.

For sure, I definitely don't want to break anyones workflow, or at the very least make sure there's an easy transition.

testing - building Pulumi YAML and Pulumi CLI using my workspace source - this might not work as expected? Or I would need to learn the "bazel way" to link together multiple modules from source?

Ah interesting, I hadn't considered building it that way tbh. I don't think this would work out of the box as it does right now, but before we consider removing the make targets in pulumi/pulumi we should make sure that this works as expected. This might mean writing a little bit of starlark code to make it easy for users, but it should definitely be doable to have something that either just works, or is very easy and well documented.

[AaronFriel]

Yeah, it's very common - because of Pulumi's distributed system nature - that we need to make cross-cutting changes to the Go Pulumi/Pulumi SDK module that affect both the engine and a plugin.