search

facebook / buck2

Build system, successor to Buck
https://buck2.build/
Apache License 2.0
3.39k stars 201 forks source link

How to build go? #455

Open testinfected opened 8 months ago

testinfected commented 8 months ago

Hi, I'm trying to use the example go toolchain, but it seems out of date.

Apparently it does not include some require values for GoToolChainInfo, namely compiler_flags_shared, compiler_flags_static, linker_flags_shared, linker_flags_static as well as prebuilt_stdlib and prebuilt_stdlib_shared.

I tried filling out those values, but don't know what to provide for prebuilt_stdlib. With an empty map I get the following error:

Error running analysis forroot//... (prelude//platforms:default#524f8da68ea2a374)`

Caused by: Traceback (most recent call last): File , in

Is there an example of a working Go Toolchain somewhere?

Any help would be appreciated.

Thanks

testinfected commented 7 months ago

Should I understand nobody is using Buck2 to build go?

ndmitchell commented 7 months ago

We are using Buck2 internally to build Go. But we don't have any open source samples. That's kind of the weird middle ground - you can probably get it working, but it will require a bit of work. I'll see if anyone internally can be persuaded to shed some light (I know the build system, but not much about how Go works with it).

podtserkovskiy commented 7 months ago

Hey @testinfected,

I can share some context on buck2&Go integration, we are using it internally for limited purposes.

This is in active development at the moment. We will make some breaking changes time to time to make it better. So, at this moment we can't give any promise of API compatibility.

However I can answer some of your questions.

compiler_flags_shared, compiler_flags_static, linker_flags_shared, linker_flags_static

Just leave them empty for the beginning.

don't know what to provide for prebuilt_stdlib

You need to pre-build it manually using following command

export GODEBUG="installgoroot=all"
shared_flags="-gcflags -shared -asmflags -shared"
go install -pkgdir "$out_dir/prebuilt_std/static" std
go install -pkgdir "$out_dir/prebuilt_std/shared" $shared_flags std

Then wrap into into filegroups

    filegroup(
        name = "stdlib_static",
        srcs = native.glob(["prebuilt_std/static/**/*"]),
    )

    filegroup(
        name = "stdlib_shared",
        srcs = native.glob(["prebuilt_std/shared/**/*"]),
    )

And then supply it into GoToolChainInfo. I suppose you are using go_toolchain for it 😀

sluongng commented 7 months ago

That’s close to what we do in Bazel’s rules_go https://github.com/bazelbuild/rules_go/blob/2f29bd3956a37156f0dee39b761109a490f0a47b/go/tools/builders/stdlib.go#L115

Note that would would probably want to filter out go install generated buildid as it could make your build not reproducible.

testinfected commented 7 months ago

Thanks, will try that. I'm very new to Buck, so fingers crossed

dvtkrlbs commented 4 months ago

@podtserkovskiy hey what should be the param for gen_stdlib_importcfg should be. I am trying to make https://github.com/sluongng/buck2-go work with the current version of buck2. and currently getting an error like this

shell output ```sh Running action: anon//:http_archive@f05825da4029d31f (http_archive) (build), local executor: env -- 'TMPDIR=/Users/dvtkrlbs/DEV/safarifruits/geomys/buck-out/v2/tmp-anon/anon/6c0c0ee7f897af38f05825da4029d31f/__http_archive__/http_archive' 'BUCK_SCRATCH_PATH=buck-out/v2/tmp-anon/anon/6c0c0ee7f897af38f05825da4029d31f/__http_archive__/http_archive' 'BUCK2_DAEMON_UUID=407d0906-13c3-4401-b87a-c2c64e02a357' 'BUCK_BUILD_ID=55f10eed-5e07-408a-934c-520e98d71512' /bin/sh buck-out/v2/gen-anon/anon/6c0c0ee7f897af38f05825da4029d31f/__http_archive__/unpack.sh Running action: prelude//go/tools:stdlib (prelude//platforms:default#6c0c0ee7f897af38) (go_build_stdlib go_build_stdlib) (build), local executor: env -- 'TMPDIR=/Users/dvtkrlbs/DEV/safarifruits/geomys/buck-out/v2/tmp/prelude/6c0c0ee7f897af38/go/tools/__stdlib__/go_build_stdlib/go_build_stdlib' 'BUCK_SCRATCH_PATH=buck-out/v2/tmp/prelude/6c0c0ee7f897af38/go/tools/__stdlib__/go_build_stdlib/go_build_stdlib' 'BUCK2_DAEMON_UUID=407d0906-13c3-4401-b87a-c2c64e02a357' 'BUCK_BUILD_ID=55f10eed-5e07-408a-934c-520e98d71512' env 'GOEXPERIMENT=nocoverageredesign' 'GOARCH=arm64' 'GOOS=darwin' 'GOROOT=buck-out/v2/gen-anon/anon/6c0c0ee7f897af38f05825da4029d31f/__http_archive__/http_archive' 'CGO_ENABLED=1' 'GODEBUG=installgoroot=all' 'CGO_ENABLED=0' '' buck-out/v2/gen-anon/anon/6c0c0ee7f897af38f05825da4029d31f/__http_archive__/http_archive/bin/go install -pkgdir buck-out/v2/gen/prelude/6c0c0ee7f897af38/go/tools/__stdlib__/stdlib_pkgdir '-gcflags=-buildid=' std Running action: root//:example (prelude//platforms:default#6c0c0ee7f897af38) (go_filter_srcs) (build), local executor: env -- 'TMPDIR=/Users/dvtkrlbs/DEV/safarifruits/geomys/buck-out/v2/tmp/root/6c0c0ee7f897af38/__example__/go_filter_srcs' 'BUCK_SCRATCH_PATH=buck-out/v2/tmp/root/6c0c0ee7f897af38/__example__/go_filter_srcs' 'BUCK2_DAEMON_UUID=407d0906-13c3-4401-b87a-c2c64e02a357' 'BUCK_BUILD_ID=55f10eed-5e07-408a-934c-520e98d71512' env 'GOEXPERIMENT=nocoverageredesign' 'GOARCH=arm64' 'GOOS=darwin' 'GOROOT=buck-out/v2/gen-anon/anon/6c0c0ee7f897af38f05825da4029d31f/__http_archive__/http_archive' 'CGO_ENABLED=1' /usr/bin/env 'PYTHONPATH=buck-out/v2/gen/prelude/6c0c0ee7f897af38/go/tools/__filter_srcs__/__filter_srcs__' python3 buck-out/v2/gen/prelude/6c0c0ee7f897af38/go/tools/__filter_srcs__/filter_srcs.py '--go=buck-out/v2/gen-anon/anon/6c0c0ee7f897af38f05825da4029d31f/__http_archive__/http_archive/bin/go' '--tags=' '--output=buck-out/v2/gen/root/6c0c0ee7f897af38/__example__/__filtered_srcs__.txt' buck-out/v2/gen/root/6c0c0ee7f897af38/__example__/__srcs__ Action failed: prelude//go/tools:stdlib (go_build_stdlib go_build_stdlib) Local command returned non-zero exit code 127 Reproduce locally: `env -- 'BUCK_SCRATCH_PATH=buck-out/v2/tmp/prelude/6c0c0ee7f897af38/go/tools/__stdlib__/go_build_stdlib/go_build_stdlib' env 'GOEXPERIMENT=nocoverageredesign' 'GOARCH=arm64' 'GOOS=darwin' 'GOROOT=buck-out/v2/gen-anon/anon/6c0c0ee7f897af38f05825da4029d31f/__http_archive__/http_archive' 'CGO_ENABLED=1' 'GODEBUG=installgoroot=all' 'CGO_ENABLED=0' '' buck-out/v2/gen-anon/anon/6c0c0ee7f897af38f05825da4029d31f/__http_archive__/http_archive/bin/go install -pkgdir buck-out/v2/gen/prelude/6c0c0ee7f897af38/go/tools/__stdlib__/stdlib_pkgdir '-gcflags=-buildid=' std` stdout: stderr: env: : No such file or directory Build ID: 55f10eed-5e07-408a-934c-520e98d71512 ```

https://github.com/dvtkrlbs/buck2-prelude here is my fork of the prelude

dvtkrlbs commented 4 months ago

nvm i figured it out. apparently some of the tools like go_wrapper and gen_stdlib_import_cfg were not included in the attrs declaration. defining them made the issue go away. Will open a PR about this. another question I have is if I open a PR with remote go toolchain downloading implemented would this be approved if not I will just contribute the simple fixes I did

podtserkovskiy commented 4 months ago

I'm sorry for the late reply. Thanks for dong this work.

I removed need to pre-compile stdlibs, so now it should be easier. I've just fixed some inconsistencies we had in GoToolchainInfo attributes and now #577 is ready for merge and will be merged shortly.

podtserkovskiy commented 4 months ago

if I open a PR with remote go toolchain downloading implemented would this be approved

Let's double check with @ndmitchell and @JakobDegen, but personally I'm happy to have it supported in buck

JakobDegen commented 4 months ago

I don't think we'd just approve it, I honestly think that more generally useful toolchain rules across languages is one of the highest value contributions buck2 could see right now. So yes, please do!

podtserkovskiy commented 4 months ago

BTW, I'm working on making system_go_toolchain work with go binary from PATH. Also, I'm adding some examples which will be compiled on CI and prevent breaking the system toolchain in the future.

I've just noticed we have some examples of downloading Go toolchain in no_prelude directory. You can use it together with system_go_toolchain for the inspiration 😀

podtserkovskiy commented 4 months ago

@dvtkrlbs @testinfected I've merged #588. So, system_go_toolchain works on Linux and MacOS, has an example of usage in examples/with_prelude/go/hello and those examples being validated on CI.

testinfected commented 3 months ago

Great news @podtserkovskiy ! Kudos

dankox commented 3 weeks ago

This issue is a bit old and the system_go_toolchain was published, but I'm curious how this can be used to do cross compilation of the Go binary.

I was trying to create toolchain similar to system_go_toolchain where I hardcoded env_go_os and env_go_arch, but I still need to provide stdlib which is compiled for that platform (it's using the default one which appears to be compiled always for the current platform, even though I thought it would pick it from toolchain definition) and I'm not sure how.

podtserkovskiy commented 2 weeks ago

This issue is a bit old and the system_go_toolchain was published, but I'm curious how this can be used to do cross compilation of the Go binary.

I was trying to create toolchain similar to system_go_toolchain where I hardcoded env_go_os and env_go_arch, but I still need to provide stdlib which is compiled for that platform (it's using the default one which appears to be compiled always for the current platform, even though I thought it would pick it from toolchain definition) and I'm not sure how.

@dankox setting env_go_os and env_go_arch should be enough for cross-compilation, there's no need to supply stdlib anymore, we build it on-demand. If you still see it in GoToolchainInfo please upgrade buck2/buck2-prelude.

If something doesn't work I'm here and happy to help 😀

dankox commented 15 hours ago

@dankox setting env_go_os and env_go_arch should be enough for cross-compilation, there's no need to supply stdlib anymore, we build it on-demand. If you still see it in GoToolchainInfo please upgrade buck2/buck2-prelude.

If something doesn't work I'm here and happy to help 😀

@podtserkovskiy Hey, thanks for coming back to me for this issue:) I'm still having a problem and I have the latest prelude. I could configure the env_go_os and env_go_arch but the stdlib is causing troubles anyway.

I have pushed my setup here (it's really simple one file project where I was trying it first): https://github.com/dankox/colorx/tree/buck2-test

I'm not sure if I'm using it correctly, but it seems that the build of the main.go works fine, until it tries to import fmt package which was build for another platform. I thought it might be caused by me building first the binary for my platform and later trying to cross-compile, but even if I remove everything and go for other platform, I will get this error:

Connected to new buck2 daemon.
Action failed: root//:colorx_linux (go_compile main)
Local command returned non-zero exit code 1
Reproduce locally: `env -- 'BUCK_SCRATCH_PATH=buck-out/v2/tmp/root/ca7581441d28fa24/go_compile/main' 'GOARCH=amd64' 'GOE ...<omitted>... g -o buck-out/v2/gen/root/3a2dd161323a3ea9/__colorx_linux__/__action__4__/go_compile_out.a ./main.go (run `buck2 log what-failed` to get the full command)`
stdout:
./main.go:4:2: could not import fmt (object is [go object darwin amd64 go1.22.2 GOAMD64=v1 X:regabiwrappers,regabiargs,allocheaders,exectracer2
] expected [go object linux amd64 go1.22.2 GOAMD64=v1 X:regabiwrappers,regabiargs,allocheaders,exectracer2
])
stderr:
Build ID: 899d8947-949a-4091-a477-bb48bb370f2f
Jobs completed: 75. Time elapsed: 20.8s.
Cache hits: 0%. Commands: 6 (cached: 0, remote: 0, local: 6)
BUILD FAILED