proposal: runtime: drop support for linux/armv5E and linux/armv6

[minux]

Since the introduction of Linux/ARM support, the minimum required hardware to run Go with Linux was ARMv5E, e.g.ARM926.

At that time, this choice makes sense because ARM9 systems are still around, and we even had an ARMv5E builder for some time.

However, ARMv5E lacks some important atomic instruction support. LDREX/STREX is introduced in ARMv6 (ARM11), and LDREXB/STREXB, LDREXH/STREXH, LDREXD/STREXD are introduced in ARMv6K. Without these instructions, we resort to emulating the required 64-bit atomic operation using 57 spinlocks selected by address (which uses mod and is a slow operation on ARM) [see https://tip.golang.org/src/runtime/internal/atomic/atomic_arm.go]

I don't know how many projects out there that still uses ARMv5E and Go, but I don't expect many. I propose to increase the minimum architectural requirement to ARMv6 (and possibly ARMv6K), so that at least we can use LDREX/STREX, which should help a lot with the new atomic-heavy runtime.

To summarize the benefits,

1. if we raise requirement to ARMv6, then we can intrinsicify 32-bit atomics.
2. if we raise requirement to ARMv6K, then in addition to 1, we no long can remove the emulated 64-bit atomics from runtime by using the native LDREXD/STREXD instructions, but also intrinsicify them in the compiler like other ports. We can also use YIELD instruction to address #16663.

If you still use Go on ARMv5E systems, please help by listing the processor model used. Thanks.

PS: other supported OSes do require ARMv6K as only Linux provides required 64-bit cas kernel helper for sync/atomic. Another way to go is to modify the runtime/internal/atomic package to:

1. use LDREXD/STREXD on non-linux,
2. use kernel cas64 on linux. But this solution will complicate maintaining the code and as we don't have a real ARMv5E builder, I think the support is certainly going to bitrot.

PS2: The core used by Raspberry Pi 1, ARM1176JZF-S, is ARMv6K, so this proposal won't affect Raspberry Pi 1, the most popular ARMv6 systems.

[janflyborg]

I don't think it's that big a deal because I doubt users of ARMv5 use the most recent version of gcc or linux kernel. But why do you insist on using the latest Go version? Are you really using the latest Go version for your ARMv5/v6 products?

We have a mechanism put into place where we can update part of the software running in the device without flashing a complete firmware. The kernel and compiler might be old, but the beauty of Go is that all the applications we install in the devices can be built with the same toolchain from exactly the same codebase regardless of CPU. Removing support for one of the CPU:s would force us to have a different branch for older products (when Go and its library moves forward), something we have managed to avoid up until now. This is of course not a complete show stopper, but it would probably complicate things for us.

As Go 1.8 will receive security updates until at least Feb 2018. Are you still planning to produce and maintain ARMv5/6 hardware in Feb 2018?

Most certainly supporting. Some of the products we support are several years old. New low cost products might also happen. It all depends on the cost/performance. We are currently using all kinds of ARM:s and also MIPS32.

[janflyborg]

Still I understand your reasons, and if it is hard to generate better code for newer CPU:s while still supporting ARM5, it probably has to go.

[jtolio]

Removing support for one of the CPU:s would force us to have a different branch for older products (when Go and its library moves forward), something we have managed to avoid up until now. This is of course not a complete show stopper, but it would probably complicate things for us.

Space Monkey is in the same boat.

[ThomasKurz]

My company has been manufacturing ARM926 based industrial control systems with a specialized Linux based C++ application ~ 1k/yr for > 10 yrs and in 2017 will move to a fresh armv7 based hardware. As our customer expects new applications to run on the old hardware, our migration plan of using Go heavily depends on Go providing armv5 support at least for next 4-5 years ... I can probably live with using Go 1.8 for following years after the release of Go 1.9, provided the libraries we use stay sufficiently backwards compatible, hoping this will not hit us too hard when Go 2.0 ever comes out.

But wouldn't it be a cleaner solution to somehow fix the GOARM flaw, so that also other arm variants (Cortex-M, ...) could be more specifically targeted?

[davecheney]

Thank you to everyone who commented. I understand your concerns but want to highlight one thing from earlier in the thread.

We need reliable ARMv5 builders. Without them, even if Go technically says it supports ARMv5, but is untested, what is the point?

Several people have offered to set up builders, so now is the time to step up and get those builders online.

Thanks

Dave

[renannprado]

Just leaving it here. One more person concerned about this: https://news.ycombinator.com/item?id=13369857

[davecheney]

@renannprado I don't think there is any confusion about the concern this proposal has raised.

I ran the ARMv5 builder for three years until it became impossible to find a piece of real ARMv5 hardware that was both reliable enough and had enough memory to run the build successfully.

The issue at hand, in my mind, is the lack of a builder for ARMv5. What I want people watching this thread to do is to focus on this part of the problem.

Without a working ARMv5 builder, in my mind, the ARMv5 port is already dead.

[jtolio]

Oh, sorry @davecheney , I didn't realize I had dropped the ball here. When @bradfitz had asked if we could provide builders on the mailing list, I assumed that was one factor into this decision. I have just been waiting to hear a go-ahead - seemed silly to set up a bunch of builders if ARMv5 support is dropped anyway. Further, based on the conversation on the mailing list (https://groups.google.com/forum/#!topic/golang-dev/CO15hgslZ0o), it sounded like the build system or tests might require some small changes to work on our specific ARMv5 boards, but maybe that was a misread.

If the Go team is actually just waiting for ARMv5 builders, is https://github.com/golang/go/wiki/DashboardBuilders still the right instructions to follow? If a Go Team member can get me a builder hash I'll set it up.

[OrangeTux]

@jtolds I'm not sure how my company could help. Our hardware doesn't have enough memory, but if you need help in a different way don't hesitate to ask.

[janflyborg]

Same thing here. Our ARM5 based cameras only have a small amount of memory (I think typically 256MB). But if there is anything else we can do (e.g buying better suited hardware - if it exists?), don't hesitate to ask us too and I will speak to my manager.

[bradfitz]

@jtolds, how many do you have available to dedicate to the cause? How long does all.bash take on one of them? Do you have good network connectivity for them? I'd prefer to shard the tests over multiple, but that requires the new-style builders and better connectivity.

[jtolio]

Realistically we have dozens of reliable builders but we could get more. Maybe low hundreds. They do have good network connectivity, but I'll get back to you shortly with all.bash timing.

[jtolio]

Alright. I don't have exact timing but it took around 53 minutes for the compilation phase of go 1.7.4 sources on armv5.

The full all.bash took:

real    152m46.856s
user    70m29.980s
sys     2m47.020s

Not all the tests passed! Here's the output of the test phase: https://gist.github.com/jtolds/9d1c0618b896b940cc86d597296329dc

[jtolio]

This was using a slow 5400 RPM drive - we can definitely speed this up by using an SSD. Notably, the compilation of bootstrap/compile/internal/gc took a very long time, maybe something like 10-20 minutes by itself? I assume the SSD would help swap performance the most.

Is this in the right order of magnitude? If so, I'll put an SSD in and rerun.

[bradfitz]

Well, normally decent machines can run all the tests in like 15 minutes. But our existing ARM builders can do ARM5 mode in like an hour. 152m seems, uh, much slower. But whatever.

Try setting GO_TEST_TIMEOUT_SCALE to, say, 3 or 5 to increase timeouts.

For the time failure: which distro/version are you running? Maybe your tzdata is stale. Or, what version of Go did you build at? Is that master?

SSD would help. tmpfs would help even more, but you probably don't have the RAM.

[ALTree]

The TestLoadFixed failure is expected and the math/big timeout not very surprising..

[bradfitz]

Why is TestLoadFixed expected? Wasn't it fixed?

As for math/big and others: I think we skip or massively shorten a few of the tests on our existing "arm5" (not really arm5) builders.

[cherrymui]

Notably, the compilation of bootstrap/compile/internal/gc took a very long time

Could you try bootstrapping from tip or 1.8 beta/rc? The SSA compiler could make it much faster (it made a huge difference on my Raspberry Pi).

[ALTree]

@bradfitz he's running go1.7.4, that's not fixed there.

[jtolio]

Sorry for the lack of communication today (busy day) but I just kicked off 1.8 RC1's all.bash with GO_TEST_TIMEOUT_SCALE=5. I'll report back in the morning when I'm awake and it's done.

[minux]

On Thu, Jan 12, 2017 at 1:55 PM, JT Olds notifications@github.com wrote:

Alright. I don't have exact timing for just the build step, but it took around 53 minutes for the compilation phase of go 1.7.4 sources on armv5.

The full all.bash took:

real 152m46.856s user 70m29.980s sys 2m47.020s

Not all the tests passed! Here's the output of the test phase: https://gist.github.com/jtolds/9d1c0618b896b940cc86d597296329dc

If it takes 152 minutes to run make.bash and the std tests, I expect the full all.bash will take twice that time.

There are a few things that could help though:

1. use zswap and then hard disk/SSD backed swap
2. use a smaller GOGC value, like 10. But please note that small GOGC value like 10 trade CPU for memory very aggressively. Usually, with small amount of zswap, GOGC=25 can pass all.bash on a machine with 256MB memory.
3. use tip to bootstrap, instead of Go 1.4. Go tip has much better GC and memory consumption is also lower.
4. patch cmd/dist so that it only runs one command at a time (change "var maxbg = 4" line in cmd/dist/util.go)

Please try these tips first before changing disks. I've been able to run all.bash with only 256MB memory and almost no swap usage with this setup.

[jtolio]

My run last night (using Go 1.4 to bootstrap) took 230m, but had some cgo dependency issues which should now be fixed on my end.

zswap got merged in 3.11 but unfortunately I'm on a 3.10 kernel. I could get a newer kernel, but I'll attempt without that change for now.

Running with GO_TEST_TIMEOUT_SCALE=5 GOGC=20 GOROOT_BOOTSTRAP=/root/gobuild/go-tip and after having changed cmd/dist. My schedule is very swamped again today so I probably won't be able to report back until late tonight. I'll let you know! Thanks for all the help!

[jtolio]

ALL TESTS PASSED

---
Installed Go for linux/arm in /root/gobuild/go-tip
Installed commands in /root/gobuild/go-tip/bin
*** You need to add /root/gobuild/go-tip/bin to your PATH.

real    324m21.381s
user    258m58.760s
sys     8m40.640s
root@spacemonkey:~/gobuild/go-tip/src# env|grep GO
GO_TEST_TIMEOUT_SCALE=5
GOROOT_BOOTSTRAP=/root/gobuild/go-tip-built/
GOGC=20

[jtolio]

So, unless this is still just way off, I think I'll attempt to get a zswap-supporting kernel, some SSDs, and if parallelism helps, a stack of 20 of these or so. Sound alright?

[jtolio]

Haven't done zswap yet but with an SSD for swap:

ALL TESTS PASSED

real    298m44.313s
user    263m36.300s
sys     8m4.120s

Going to try and back off on GOGC see if it goes faster. maxbg doesn't matter since there's only one core anyways.

[jtolio]

Using default GOGC is much better:

ALL TESTS PASSED

real    167m18.671s
user    142m44.710s
sys     7m48.490s

Okay, what do I need to do to hook these up?

[OrangeTux]

@jtolds Thanks for putting effort into this.

[benshi001]

As this proposal is accepted, the runtime library will support from ARMv6. But how should the generated assembly of the compiler be? It should support only ARMv6 instructions? Or if some ARMv7 instructions (such as the RBIT instruction which can benefit the CTZ intermediate operator) can be used if GOARM=7 is specified?

I suggest use the GOARM env-var and 6 is its default value. And ARMv7 users can benefit by specifying it to 7.

[3mam]

The core used by Raspberry Pi 1, ARM1176JZF-S, is ARMv6K, so this proposal won't affect Raspberry Pi 1, the most popular ARMv6 systems.

When I first heard about abandoning ARMv6 thought that I lose the ability to use Go on the Raspberry Pi Zero. But now I can sleep peacefully.

[jtolio]

As this proposal is accepted, the runtime library will support from ARMv6.

I hope we can reopen that! I'm ready to start some ARMv5 builders. Just need the creds.

[janflyborg]

Without a working ARMv5 builder, in my mind, the ARMv5 port is already dead.

I'm just curious. So now that @jtolds has done the excellent job of providing us with ARMv5 builders, does this change anything?

[minux]

For creds, please email bradfitz (at the obvious Go domain) to request a linux/arm builder key. Thanks.

[bradfitz]

So, what's the story here? Are we not dropping this for Go 1.9 or not?

It sounds like we can get real builders, which moves my position away from killing arm5 back towards neutral or maybe even keeping it. What do others feel? Sounds like 3 or 4 companies are interested it in it remaining. I don't know how much work it is to keep it in place.

/cc @minux @cherrymui @rsc @randall77 @mdempsky @josharian

[josharian]

My 2c, which should not be given much weight, is: Given the responses in this issue, we should keep ARMv5, iff the real builders prove to be reliable (good uptime, reasonably latency, etc).

[ThomasKurz]

Please keep in mind that despite ARMv5 being already old, it is not yet dead!

It once came as ARM926ES-J, XScale, Feroceon, ... and is still actively being sold by Atmel, TI, NXP, Digi and more. Well-known devices include the above mentioned SAM9xxx from Atmel and former Freescale's i.MX23..28, all very common in the field of industrial automation with large numbers, think e.g. of Lego EV3 which has Ti AM1808 ARM9. Many less well-known chip manufacturers like e.g. Netsilicon/Digi are still producing their own ARM9 derivatives (with Linux support) for niche markets like telecom in remarkable numbers, without making fuzz.

What many of these platforms today have in common, is the scenario i already mentioned above: Software development for the successors of these hardware veterans starts now, and we would love to use Go (instead of C or C++) for it.

Now usually in these (often industrial) applications we have 10000's of ARMv5 devices in the field, which the customer expects us to maintain for the next 5 or so years, so if we cannot run at least a limited version of the new application on the old hardware, Go is a no-go and we are forced to stick with C++ to make the customer happy. Thus we are already really happy about Go 1.8 still supporting ARMv5, opening at least the option of sticking at 1.8 for this hardware version, although I'm not sure if then it is worth the risk.

Unfortunately all our ARMv5 systems here have only ≤ 64MB of RAM, but given @jtolds ' builder fulfills the expectations, please consider it useful to make ARM code generation generally flexible enough to make it simple to keep ARMv5 as lowest-end family member.

[trukumaran]

I support @ThomasKurz as my company also uses the same specs . and we would love to use Go for our applications. Hope you will consider our side.

[janflyborg]

Thus we are already really happy about Go 1.8 still supporting ARMv5, opening at least the option of sticking at 1.8 for this hardware version, although I'm not sure if then it is worth the risk.

@ThomasKurz Please note that maintaining a 1.8 based solution for five years is only possible if Go 1.8 is going to receive security patches for at least the same amount of time. Previously I have heard "until at least Feb 2018", so without an increased time scope for the security patches, that will leave us stranded next year anyway.

[davecheney]

The support cycle for previous releases is listed here, https://github.com/golang/go/wiki/Go-Release-Cycle, in the section marked "Release Maintenance"

On Tue, 14 Feb 2017, 00:43 janflyborg notifications@github.com wrote:

Thus we are already really happy about Go 1.8 still supporting ARMv5, opening at least the option of sticking at 1.8 for this hardware version, although I'm not sure if then it is worth the risk.

@ThomasKurz https://github.com/ThomasKurz Please note that maintaining a 1.8 based solution for five years is only possible if Go 1.8 is going to receive security patches for at least the same amount of time. Previously I have heard "until at least Feb 2018", but without an increased time scope for the security patches, that will leave us stranded anyway.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/golang/go/issues/17082#issuecomment-279395636, or mute the thread https://github.com/notifications/unsubscribe-auth/AAAcA_lFH4lcvyzzzjCuGAQl4TTPOxQJks5rcF3vgaJpZM4J7K4x .

[janflyborg]

So this means that Go 1.8 will receive security patches up until August 1 2018? That is still far away from the five years that @ThomasKurz needs to be able to support his customers and the company I work for has exactly the same problem.

[bradfitz]

That is still far away from the five years that @ThomasKurz needs to be able to support his customers and the company I work for has exactly the same problem.

Fortunately you have customers paying you money, so you can use that money to pay people to support Go 1.8 longer than we're supporting it for free. We just don't have the resources to support all old branches for years.

[rsc]

Tentatively tossing back into the proposal pile for further discussion. It would be nice to figure out the general policy for architectures; see also the thread on golang-dev about big-endian power5 ppc64.

[janflyborg]

Fortunately you have customers paying you money, so you can use that money to pay people to support Go 1.8 longer than we're supporting it for free. We just don't have the resources to support all old branches for years.

@bradfitz Has anyone talked about "supporting all old branches for years"? My remark was only about Go 1.8, which will be the last release that supports a very commonly used CPU (if this proposal is accepted).

The purpose of my posting was just to emphasize that if you are having an existing solution based on ARMv5, it is not an alternative to just freeze your code base at Go 1.8 (since it will not receive any security updates 1.5 year from now).

If it could only be arranged for Go 1.8 to receive crucial updates for a longer time period than normal, then I don't think anyone would have any problems with this proposal at all.

[bradfitz]

@janflyborg, we are going to support Go 1.8 for one year. We will support no branch for more than that. Certainly not five.

Like I said before, if you have revenue and are promising support to users, you need to budget in the cost of paying somebody to do security updates for all your dependencies which includes but is not limited to your OS and Go. Maybe a bunch of companies with similar needs can get together and collectively maintain a long-term stable branch of Go. Or maybe you just need to use a version of Go that comes with Ubuntu LTS and rely on Ubuntu or your OS of choice's maintenance team to do the security backports for you.

But the answer is not upstream (us) doing security maintenance for five years. We can definitely answer questions on the mailing list about backports with conflicts, to the extent we remember, but we're not in the business of maintaining old software.

[ThomasKurz]

I should apologize for having pushed this discussion into a somewhat off-topic direction yesterday by mentioning the option of "sticking with 1.8" (but with the restriction "not worth the risk"). Go's value is so much more than the compiler. Many of the awesome (standard) libs have grown and got new features with each version. Think of the "legacy-limited + current-full" hardware platform scenario I mentioned above, and imagine you couldn't use the same import "context" in your application, and no third-party libs relying on it, only because one of the platforms sticks with pre-1.7 ... I do not want to keep 1.8 alive for years, as this would completely cut innovation for the non-legacy platforms (besides the fact that there is no direct profit from legacy platform support, which makes it hard to budget).

What's imho much more interesting is the question of an ARM "backend split", as this touches Go's no.1 rule of simplicity.
As I understood it, Go currently supports only one single ARM code generation target, with the option to select different modes of FPU support. For me professionally developing code for ARM CPUs since >16 yrs and having seen their core capabilities grow and change so much over time, this seems to be overly simplistic. I certainly admit that it's a necessary tradeoff if "ease of use" is your primary concern. But you will regularly have to "cut the old pigtails" then, as development won't stop with ARMv8. I think there are many markets where hardware half-life time is short enough for this approach, ours isn't.

I would thus strongly opt for a more flexible, perhaps feature-based code generation strategy without the need to decide between "target split" and "cut off" with each new, but incompatible architectural feature.

Would it be hard to make ARMv6k the default target, but keep code for ARMv5 around (in the compiler) for those who really need it, and at the same time become open for new optimizations on not-yet mainstream CPU variants?

P.S.: Since I first learned to know Go I am dreaming of using it for bare-metal/RTOS development on Cortex-M CPUs - how could that ever come true with the current "there is but one" paradigm?

[davecheney]

Would it be hard to make ARMv6k the default target, but keep code for ARMv5 around (in the compiler) for those who really need it, and at the same time become open for new optimizations on not-yet mainstream CPU variants?

This is sort of true and sort of not true.

Ben Shi has submitted a number of CLs that add support in the arm backend for new ARMv6 and v7 instructions. However those instructions are only better ways to expressing the same operation, so if the chip doesn't support it, we don't generate the instructions.

The problem with ARMv5 is there is no support for atomic operations like LDREX which significantly complicate papering over the lack of this instruction in the backend.

P.S.: Since I first learned to know Go I am dreaming of using it for bare-metal/RTOS development on Cortex-M CPUs - how could that ever come true with the current "there is but one" paradigm?

Thumb2 support is an entirely different kettle of fish, it would probably be a different backend target, perhaps linux/thumb2. Thats how it was done on plan9

On Tue, Feb 14, 2017 at 8:24 PM, Thomas Kurz notifications@github.com wrote:

I should apologize for having pushed this discussion into a somewhat off-topic direction yesterday by mentioning the option of "sticking with 1.8" (but with the restriction "not worth the risk"). Go's value is so much more than the compiler. Many of the awesome (standard) libs have grown and got new features with each version. Think of the "legacy-limited + current-full" hardware platform scenario I mentioned above, and imagine you couldn't use the same import "context" in your application, and no third-party libs relying on it, only because one of the platforms sticks with pre-1.7 ... I do not want to keep 1.8 alive for years, as this would completely cut innovation for the non-legacy platforms (besides the fact that there is no direct profit from legacy platform support, which makes it hard to budget).

What's imho much more interesting is the question of an ARM "backend split", as this touches Go's no.1 rule of simplicity. As I understood it, Go currently supports only one single ARM code generation target, with the option to select different modes of FPU support. For me professionally developing code for ARM CPUs since >16 yrs and having seen their core capabilities grow and change so much over time, this seems to be overly simplistic. I certainly admit that it's a necessary tradeoff if "ease of use" is your primary concern. But you will regularly have to "cut the old pigtails" then, as development won't stop with ARMv8. I think there are many markets where hardware half-life time is short enough for this approach, ours isn't.

I would thus strongly opt for a more flexible, perhaps feature-based code generation strategy without the need to decide between "target split" and "cut off" with each new, but incompatible architectural feature.

Would it be hard to make ARMv6k the default target, but keep code for ARMv5 around (in the compiler) for those who really need it, and at the same time become open for new optimizations on not-yet mainstream CPU variants?

P.S.: Since I first learned to know Go I am dreaming of using it for bare-metal/RTOS development on Cortex-M CPUs - how could that ever come true with the current "there is but one" paradigm?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/golang/go/issues/17082#issuecomment-279652750, or mute the thread https://github.com/notifications/unsubscribe-auth/AAAcA5_MFNRJCir9xTgaSSorwgrFL4A8ks5rcXLIgaJpZM4J7K4x .

[ThomasKurz]

Thanks a lot for clarification, @davecheney ! Can you point me to a document about Go's current backend architecture to help me get better understanding?

[davecheney]

I'm afraid it's "use the source, Luke"

On Tue, 14 Feb 2017, 21:33 Thomas Kurz notifications@github.com wrote:

Thanks a lot for clarification, @davecheney https://github.com/davecheney ! Can you point me to a document about Go's current backend architecture to help me get better understanding?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/golang/go/issues/17082#issuecomment-279670371, or mute the thread https://github.com/notifications/unsubscribe-auth/AAAcA37Uu30L7X1zGCmgPk2ARgJuTWjHks5rcYMLgaJpZM4J7K4x .

[janflyborg]

@bradfitz OK I see. Thanks for the answer.

[4ad]

Thumb2 support is an entirely different kettle of fish, it would probably be a different backend target, perhaps linux/thumb2. Thats how it was done on plan9

Thumb support in Go would definitely be a new target, but Plan 9 never had a thumb port. There are thumb compilers for Inferno though.

[kortschak]

If you still use Go on ARMv5E systems, please help by listing the processor model used.

ARM926EJ-S (TI AM1808).