Adding please to a large repo - how to migrate to please and best practices for upkeep

[canuck3141]

Hi,

Please is awesome, thanks for sharing it!

If I'm coding in a repo with a single large go package containing hundreds of sub packages and 100 KLOC or more, any specific advice how to introduce please there?

For example:

1. That repo uses a go.mod to track hundreds of external third party go modules added via go get. Are there tools to migrate these to the please go_repo format from go.mod?
2. If I stick with just go.mod and don't use go_repo what would I lose from please' functionality/perf? I can't walk away from go.mod since I rely on other go tooling/functionality (that uses go.mod) that please doesn't support.
3. If I have other developers modifying go.mod directly, how to keep please' go_repo up to date with that?
4. Do I need to create a BUILD file in each subpackage dir?
5. What about needing to add go_test entries? Are these necessary for please and what would I miss out if I don't generate them?

Part of my concern is that if developers add new tests, subpackages, etc what might break with please if those developers don't also add go_repo, go_test and am I adding overhead if I require developers to add this config?

Thanks!

[Tatskaari]

Yup! You should mostly be able to run puku sync -w to generate all the third party stuff, and then run puku fmt ... to generate all the first party build files. We haven't built automated workflows to manage drift between the go.mod and third_party/go/BUILD yet but it's on the cards.

To get this to work nicely I recommend adding the following config options:

[Plugin "go"]
ModFile = //:modfile ; a filegroup for your go.mod file. This allows Please to resolve third party deps the same way as go_repo
TestRootCompat = true ; makes go_test run the test with the same working directory as `go test`

4. Do I need to create a BUILD file in each subpackage dir?

Yes. Puku can do this for you though.

5. What about needing to add go_test entries? Are these necessary for please and what would I miss out if I don't generate them?

Please tests provide a lot of control that go test can't. Internally, we run full e2e tests against our gRPC microservices, spinning up kafka, postgres, etc. all inside a go_test(), but this only works on linux where tests are run in a sandbox. You can see a slightly less advanced version of this for puku where we run e2e tests here: https://github.com/please-build/puku/tree/master/e2e

This doesn't use sandboxes but still uses please to manage all the test deps. You can see what's going on there with plz test --shell //e2e/tests/codegen:codegen_test, and have a look at how this is set up here: https://github.com/please-build/puku/blob/master/e2e/build_defs/e2e.build_defs

[jbcpollak]

Hi, I read the Puku documentation and also looked at the implementation itself, but I'm still having trouble getting this working. I think the problem I am running into is that we don't have a top-level go.mod, and the examples seem to assume there is one.

At the top level of my repo I currently have a BUILD file:

subinclude("///go//build_defs:go")

filegroup(
    name = "first_binary",
    srcs = ["first_binary"],
    visibility = ["PUBLIC"],
)

and the .plzconfig has:

[Plugin "go"]
Target = //plugins:go
; a filegroup for your go.mod file. This allows Please to resolve third party deps the same way as go_repo
ModFile = //:gomod
; makes go_test run the test with the same working directory as `go test`
TestRootCompat = true

running sync gives me:

$ plz puku sync
00:02:15 CRITICAL exit status 1
00:02:15.559   ERROR: //:gomod failed:
Parsed build file BUILD but it doesn't contain target gomod
Build stopped after 20ms. 1 target failed:
    //:gomod
Parsed build file BUILD but it doesn't contain target gomod

I don't really understand the syntax for ModFile. What should I do if I have two projects in the mono repo, with different dependencies? For example, first_binary and second_binary? Each of those would have their own go.mod, and there wouldn't necessarily be one in the top level directory right?

[Tatskaari]

You can remove that config entirely and it will try and resolve dependencies itself. The go_repo rules really do work better with a go.mod file, though, as without it we can't resolve the build list (list of resolved modules) consistently. Internally, we have a dummy one under //third_party/go that we use to resolve dependencies, but technically you can go without. You may just have to pass in modules via requirements = ["github.com/example/module"], to help resolve imports to the correct module path.

I've made a small update to the docs to make it clearer that that step is optional. Let me know if that makes more sense.