support aarch64 for go-bootstrap/go

[darmac]

I found that go-bootstrap/go package cannot support aarch64 now.

If we want to support aarch64 , the flow is like that:

1) Download the 1'st arm64 version as below: https://dl.google.com/go/go1.8.7.linux-arm64.tar.gz

2) unpack go1.8.7.linux-arm64.tar.gz

3) export GO_BOOTSTRAP=/xxx/go

4) After finishing above steps, the latest go package can be download and compile.

But the question is that: I think it's totally deferent way of x86 bootstrap compare for the above steaps. So I'm not sure how to modify the go-bootstrap package.py.

Is there any suggestion for developing this patch?

[adamjstewart]

@hartzell

[hartzell]

Yeah, the Go compiler is written in Go, so you need a bootstrap compiler to build it. The last Go compiler written in C was the 1.4 release, and the Go team is committed to keeping it alive for people who insist on bootstrapping from a C environment.

But aarch64 support didn't arrive until 1.5 (according to the intro), and @darmac's comment suggests that one needs 1.8.7.

The docs list 4 options for getting a bootstrap compiler:

1. Bootstrap toolchain from binary release
2. Bootstrap toolchain from source
3. Bootstrap toolchain from cross-compiled source
4. Bootstrap toolchain using gccgo

As mentioned above, 2 is out. I can't see anything useful to be done with 3 either.

That leaves 1 and 4 and leads to a Philosophical Question about how Spack ought to do things: is it worth the trouble to build an aarm64 gcc with go support so that we continue to always build things from source or should we grab one of the Go team's builds and use it the same way we use the system C compiler? How important is the build-all-the-things-from-source ethos?

I'm probably not the best arbitrar for that question, I don't always seem to look at things the same way as other Spack users. Hopefully others will chime in. @adamjstewart -- thoughts on who to tag?

I don't have access to an aarch64 system. @darmac -- can you play around with using Spack to build gcc@<VERSION_TBD> languages=go and report back about how long it takes and how tricksy it is? I'm unsure about the version, the bootstrap docs seem to suggest something in the 5 series will work, but...

If we go with 1, then some package needs to fetch the tarball, unpack it some place, and setup any additional necessary environment variables to use it. This could be conditional logic in the go-bootstrap package or a separate go-binary-bootstrap. If we go with the latter, the go package would need to depend_on the correct thing as the situation requires.

If we go with 4, then it could either be hidden in the go-bootstrap package or the go package could be be adjusted to conditionally depend_on gcc@VERSION languages=go, where VERSION is tbd.

Either way, perhaps all of the go architectures should switch to using method 1 or 4. That'd be a big bunch of churn, but the consistency might be worth it. Again, feedback....

[adamjstewart]

I’m personally fine with grabbing a Go binary and using it to bootstrap a newer version. We use binaries for other packages like Java. And since were using it to bootstrap, the final product is still built from source and optimized.

[hartzell]

Yeah, it's nicely analogous to how the icedtea package depends on jdk; the jdk package just unpacks stuff, install_tree's it into place, and does the appropriate setupdependent* stuff.

So, that's option 1.

[hartzell]

Do we need other inputs? I've guessed wrong a few times lately....

What about reworking the dependency for all the platforms?

[adamjstewart]

@hartzell As far as I'm concerned, you're in charge of anything Go-related in Spack. If you know other Spack users who use Go, you can ping them, but I'm fine with letting you make the final call. I agree with doing things uniformly across all platforms.

[hartzell]

Ok, pinging a couple of other folks who've had their fingers in the package at one point or another and might have constructive input: @trws @paulbry @michaelkuhn @matthiasdiener

[hartzell]

@darmac -- What machines are you using? AWS Graviton/A1 instances or ???

[trws]

While I originally wrote it, I haven't touched this in some time. For whatever it's worth, the plan at the time was to support this kind of thing by at some point allowing gccgo to provide the dependence through a virtual dep or a compiler dep (if those ever materialized).

It would probably be most general to find a way to allow gccgo to provide this, but if we're stuck with a binary it's not the end of the world. I'd prefer it be scoped as narrowly as possible though, there are too many binary packages that blindly grab a single version regardless of actual architecture compatibility around.

[hartzell]

@trws -- Thanks for the feedback. I'm having parsing this sentence:

I'd prefer it be scoped as narrowly as possible though, there are too many binary packages that blindly grab a single version regardless of actual architecture compatibility around.

Which scoping? Are you suggesting that we only use binary bootstrapping for packages that we can't build from source, and go@1.4 for the rest?

One interesting way to figure out what binary package to use to bootstrap go@<VERSION> would be to just use the Go team provided relase of Go at <VERSION>. That should ensure that no features are in use that an older binary doesn't support.

[trws]

Yup, that's precisely what I meant. Use it for only the architectures or similar that are absolutely required, or at least only those where it's verified to work. There are too many ways for the binary packages to go wrong unless they're only enabled for exactly what they support, specific architecture, operating system, calling convention, perhaps not here but usually also libc version, etc.

[darmac]

@darmac -- What machines are you using? AWS Graviton/A1 instances or ???

=> I'm using TaiShan2280-V2 as test machines. And the CPU is based ARMv8 (Kunpeng920 and Kunpeng916).

@darmac -- can you play around with using Spack to build gcc@<VERSION_TBD> languages=go and report back about how long it takes and how tricksy it is?

=> I will tried it and feedback the result.

That leaves 1 and 4 and leads to a Philosophical Question about how Spack ought to do things

=> Just in my opinion, I think option 1 is much easier to transport the go-bootstrap and go packages to aarch64 platform. I seldom use the go language, so I'm not sure if option 4 is a better way, what I tried is compile latest go1.13.7 by using go1.8.x. And also, if we can run pass the GO Unit Test Cases, the compatibility risk will be reduced?

Anyway, thanks so much for your suggestion.

[darmac]

@hartzell

@darmac -- can you play around with using Spack to build gcc@ languages=go and report back about how long it takes and how tricksy it is?

=> I checked it, and can build it automatically. Then I tested a hello_world.go and run it well.

[darmac]

I have tried both way of Bootstrap toolchain manually, both of them can work well.

Method 1: Bootstrap toolchain using gccgo spack spack install -v gcc languages=go spack load gcc languages=go

Method 2: Bootstrap toolchain from binary release wget <go_src_tarball> export GOROOT=<your path> export PATH=$GOROOT/bin:$PATH

GO TOOL BUILD : After bootstrap is ready for using, we can start build go tools wget <go_src_tarball> cd go/src/ ./make.bash

I think these steps above is quite easy, and it should work in most of platforms. So, do you have any suggeustion for the next? Shall we refine the go-bootstrap package?

[darmac]

@hartzell @adamjstewart Is there any suggestion update?

[alecbcs]

Hey all, just ran into this attempting to compile Go on an Arm based Mac. If bootstrapping with gccgo is possible. Would it be feasible for the go-bootstrap package to just be a wrapper around installing gcc with the go language support enabled?

Are there any specific use cases where compiling GCC wouldn't be an option and we would have to fallback to a binary bootstrap?

[trws]

Not that I know of. That’s a good solution, but an expensive one if you don’t have gccgo available already. That’s why we started the way we did, even bootstrapping from an ancient one through c was a great deal faster, but that should be much more reliable if it actually covers all the versions and platforms.

Sent from my iPhone

On May 6, 2021, at 11:22 AM, Alec Scott @.***> wrote:



Hey all, just ran into this attempting to compile Go on an Arm based Mac. If bootstrapping with gccgo is possible. Would it be feasible for the go-bootstrap package to just be a wrapper around installing gcc with the go language support enabled?

Are there any specific use cases where compiling GCC wouldn't be an option and we would have to fallback to a binary bootstrap?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://urldefense.us/v3/__https://github.com/spack/spack/issues/14900*issuecomment-833754747__;Iw!!G2kpM7uM-TzIFchu!iKD5ehsMV0nmp_OLsrKGVAXbGp3yx1iK4vMKxPG1aS8aAT5_uymp_stiA5BWL4M3Wg$, or unsubscribehttps://urldefense.us/v3/__https://github.com/notifications/unsubscribe-auth/AAFBFNJTM77LRUJH2NE6ASLTMLM5VANCNFSM4KTRBDBQ__;!!G2kpM7uM-TzIFchu!iKD5ehsMV0nmp_OLsrKGVAXbGp3yx1iK4vMKxPG1aS8aAT5_uymp_stiA5A5TCp0KA$.

[rgmiller]

I'd like to point out that the currently, AArch64 users can't actually build Go with Spack. Pull #28850 changed go-bootstrap/package.py to conflict when the target is aarch64 and the platform is darwin and changed also go/package.py to remove the option to bootstrap using gccgo. Unfortunately, as discussed above, the go-bootstrap is too old to support any AArch64 hardware, so go-bootstrap doesn't work even if you're not using Apple hardware.

I've updated both package.py files locally so that I can build go using gccgo on our AArch64 cluster. Should I create a pull request for these changes?

Also note that there's a bug in some versions of gcc that causes the build of go to fail so you have to specify particular versions of gcc in the depends_on line in go/package.py. (See golang/go#47771.)

[trws]

If it works, yes you definitely should.