lox
commented
3 years ago Are there any workarounds for this in the meantime?
Closed jbeker closed 8 months ago
lox
commented
3 years ago Are there any workarounds for this in the meantime?
cguess
commented
3 years ago Has there been any updates on this? With the launch of even more M1 ARM-based Macs it's quickly becoming the soon-to-be-default architecture, and if you're running a VM there's no option except aarch64 for Linux (can't emulate x86).
korbin
commented
3 years ago Has there been any updates on this? With the launch of even more M1 ARM-based Macs it's quickly becoming the soon-to-be-default architecture, and if you're running a VM there's no option except aarch64 for Linux (can't emulate x86).
You could use QEMU to run Sorbet, probably.
I was able to make ARM64 builds work for Linux aarch64 targets:
https://github.com/sorbet/sorbet/compare/master...angellist:arm64
I am not familiar enough with Bazel to add all of the conditionals necessary to make this properly support cross-platform builds, however, my findings:
Bazel upstream isn't publishing installers for other architectures now. I've opted to directly install platform Bazel binaries.
libb2 depends on SSE2 instructions. I removed it in favor of the existing blake2/blake2 implementation which seems to have optional SSE support.
Inline x86 assembly can be changed (safe to merge upstream now I assume): __asm__("int $3"); -> __builtin_debugtrap();
I also attempted to get Darwin ARM64 support working (and it probably will if these caveats are fixed), however:
The LLVM project does not publish Darwin ARM64 binaries. Homebrew clang or system clang will need to be used. (I am not familiar enough with Bazel yet to build this escape hatch cleanly.)
jemalloc needs to be downloaded from the dev branch.
Hopefully this helps enable upstream ARM support!
cguess
commented
3 years ago @korbin QEMU can emulate x64, but it is SLOW, pretty much unusable.
klardotsh
commented
2 years ago this is a blocker to teams using sorbet in docker as well - let me know if there's anything I can do to help move this forward; as it stands I believe my workaround has to be forcing platform: linux/amd64 for our containers, which is unideal.
EDIT: nope, that doesn't work because of the lack of inotify in qemu and the unsuitability of the "Experimental Virtualization Backend" for our purposes. I'll be working to try to build arm64 native sorbet for my team.
froydnj
commented
2 years ago let me know if there's anything I can do to help move this forward
Can you at least submit a PR for https://github.com/sorbet/sorbet/commit/a7cdf0b8c24754338b7a5700e89823d6d056516b ? I think the other commits would require some reworking (all the s/x86_64/aarch64/-style ones) or some testing (the blake2 changes), but the __builtin_debugtrap seems uncontroversial, assuming GCC supports it (i.e. we can build in Stripe's CI)
froydnj
commented
2 years ago I think we'd take a PR for the bazel script changes too!
klardotsh
commented
2 years ago A brief end of week update on my end: I haven't fully inspected the generated gems or run the test suite against them, but building on the work from AngelList above, I seem to have at least runnable aarch64 sorbet ELF binaries generated on an aarch64 host (read: not cross-compiled from amd64). The logic roughly being that usually I find on-host builds easier to reason about than cross-compiled, so I decided to head down the path I was most comfortable with in general (I've dealt with ARM Linux stuff for years, but almost never cross-compiled).
The commit logs are a little messy (especially on the bazel-toolchain side), there's decidedly some hackery going on that needs cleaned up because I don't particularly know Bazel well, and just broadly it'll need work before it'll be in a state to work with @korbin @froydnj and co. to upstream it, but it's progress and I'll likely continue poking at this Tuesday, if not over the weekend.
The below output is copied from an M1 Mac, but I was also able to use Docker BuildX to "cross-compile" (very, very, miserably slowly) from my AMD 3900X rig running Void Linux.
joshklar@twg-mba-klar sorbet % make release-linux-arm64
# this is PROBABLY dangerous in the event your system already has bin
# handlers or something but whatever, those are *super* uncommon,
# especially on CI rigs
[ "$(uname -s) $(uname -m)" == "Linux x86_64" ] && docker run --rm --privileged multiarch/qemu-user-static --reset -p yes || true
docker buildx build --platform linux/arm64 .
[+] Building 516.1s (10/10) FINISHED
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 573B 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [internal] load metadata for docker.io/library/ubuntu:focal 1.1s
=> [1/5] FROM docker.io/library/ubuntu:focal@sha256:626ffe58f6e7566e00254b638eb7e0f3b11d4da9675088f4781a50ae288f3322 0.0s
=> [internal] load build context 5.6s
=> => transferring context: 195.39MB 5.5s
=> CACHED [2/5] RUN apt update && apt install -y build-essential curl perl unzip python autoconf 0.0s
=> CACHED [3/5] RUN apt update && apt install -y libffi-dev libgmp-dev libtinfo5 libtinfo-dev python python3 0.0s
=> [4/5] COPY . . 0.8s
=> [5/5] RUN ./bazel build //main:sorbet --config=release-linux 491.5s
=> exporting to image 17.1s
=> => exporting layers 17.1s
=> => writing image sha256:aa08645ee8b33bba601a9140a4d33faeb23fe2a71cc592d60e2eb0f03af19624 0.0s
# ...
joshklar@twg-mba-klar sorbet % docker run --rm -it aa08645ee8b33bba601a9140a4d33faeb23fe2a71cc592d60e2eb0f03af19624
root@478d1ec030c9:/# file ./root/.cache/bazel/_bazel_root/6666cd76f96956469e7be39d750cc7d9/execroot/com_stripe_ruby_typer/bazel-out/aarch64-opt/bin/main/sorbet
./root/.cache/bazel/_bazel_root/6666cd76f96956469e7be39d750cc7d9/execroot/com_stripe_ruby_typer/bazel-out/aarch64-opt/bin/main/sorbet: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, for GNU/Linux 3.7.0, BuildID[md5/uuid]=121e6e884edc997ac510ce7630cc2037, with debug_info, not stripped
root@478d1ec030c9:/# ./root/.cache/bazel/_bazel_root/6666cd76f96956469e7be39d750cc7d9/execroot/com_stripe_ruby_typer/bazel-out/aarch64-opt/bin/main/sorbet
You must pass either `-e` or at least one folder or ruby file.
Typechecker for Ruby
Usage:
sorbet [OPTION...] <path 1> <path 2> ...
-e, string Parse an inline ruby string (default: "")
-q, --quiet Silence all non-critical errors
-v, --verbose Verbosity level [0-3]
-h, Show short help
--help Show long help
--version Show version
root@478d1ec030c9:/# ./root/.cache/bazel/_bazel_root/6666cd76f96956469e7be39d750cc7d9/execroot/com_stripe_ruby_typer/bazel-out/aarch64-opt/bin/main/sorbet -e 'puts "yay"'
No errors! Great job. 
joaolrpaulo
commented
2 years ago 👋 Any news about this topic?
Will there be a new release of the sorbet-static for aarch64 anytime soon?
vinistock
commented
2 years ago Our team is working on support for the M1 macs tracked in this milestone. Based on the investigations posted here, I believe the changes needed for M1 support overlap with most of what is needed for Linux ARM64 as well - other than some configuration bits.
The changes in this experimental branch are enough to compile Sorbet to ARM64 on an M1 machine for both debug and release mode. However, we still need to figure out some build failures for the custom Ruby build and emscripten.
Additionally, that branch uses the bazel-toolchain version that has support for the ARM64 clang toolchains, which is still in a PR and not yet merged upstream.
After all of the code changes are merged and the configuration API is finalized on bazel-toolchain, we should be able to start releasing the new binaries.
klardotsh
commented
2 years ago please feel free to holler if there's anything we (Dockwa) can do to help, even if it's testing a quick thing. we've been using the linux/amd64 and linux/arm64 gems+docker images generated in https://github.com/dockwa/sorbet/commits/josh/arm64-more for a bit now and, short a PEBKAC issue I introduced with versioning at one point, haven't had issues (mind you, we fall back to upstream gems on Darwin-bare-metal because I never figured out cross-compilation, with Bazel or - experimentally at one point - with Zig's cross-compile-as-a-first-class-citizen toolchain)
excited to see this getting upstream traction!
froydnj
commented
2 years ago Note that getting all the necessary changes in to compile on arm64 is one step (and an important one!), but not the complete picture: we also need to cross-compile the appropriate bits on Buildkite (AFAICT, Buildkite does not seem to have native arm64 machines/containers available).
klardotsh
commented
2 years ago Certainly native arm64 machines would be ideal, but if those fail, a (somewhat slow) option (that we're using in our fork) is to QEMU around the problem - Docker BuildX sets up a binfmt_misc handler to route cross-arch builds through QEMU wrappers, simulating a "native" arm64 build. From what I can tell, Rust's https://github.com/cross-rs/cross does this as well. Certainly not saying it's ideal (our arm64 images take about 5-10x longer than our amd64 images, building on an AMD 3900X with 24 threads), but it's a potential workaround if the cross-compile-directly story gets hairy. Happy to help there as well as time allows, if needed.
klardotsh
commented
2 years ago Chiming in here again, curious to see what the blockers here are, and what my org could do to help. At this point we're still running my fork from the winter, which means v0.5.9613 (the last version I could get to cleanly rebase), complete with all of our packaging layers to get there. Simplifying back to a simple "bundle install" would be great.
louiscb
commented
2 years ago @klardotsh can you publish your forked version that works with arm64?
klardotsh
commented
2 years ago @klardotsh can you publish your forked version that works with arm64?
Ah sorry, I realize I'd talked about the WIP above and never mentioned that all this work has been public all along... just without instructions available. Here goes!
master of https://github.com/dockwa/sorbet is at v0.5.9613 (this limits you to Tapioca 0.7.x if you use check-shims [^1] [^2]). This is a set of 12 commits over AngelList's fork above; essentially all are packaging and build system related.
In theory one should be able to apply this to bare-metal aarch64+glibc Linux; pulling the linux/arm64 OCI image (docker pull blahblah --platform linux/arm64 works I think maybe possibly, otherwise dig into the multiarch manifest) and dumping it to a tarball [^3] should allow portability outside of containers.
To apply some duck-typing theory: this looks like a hack, walks like a hack, and quacks like a hack, so it's definitely a hack. With some caveats however (notably, that things like Tapioca RBI generation must be done in a consistent environment - for us that's linux/amd64 with RAILS_ENV=development - due to two Tapioca issues ^4), it's been working for us at Dockwa since Dec/Jan ish.
[^2]: I've tried rebasing this branch onto something newer and Tapioca 0.8.x compatible and while the diffs eventually apply cleanly, I get Bazel-induced build failures. Bazel is a monstrously complex beast I have thus far consistently failed to wrap my head around (especially debugging it), and thus I gave up for now - I have too many non-Sorbet responsibilities at work to dedicate the requisite time to this.
[^3]: docker docs, podman docs
klardotsh
commented
2 years ago I should, however, explicitly note now: my org is removing Sorbet from our codebase in the near future; I will likely no longer be maintaining the above forks or Docker images any further, and in particular, will not be continuing the efforts to rebase onto newer Sorbet releases. I have no intentions quite yet of removing the repos, and of course the contributions are all under the same license as Sorbet upstream, so feel free to make your own forks or otherwise remix my work so far [^1] :) Cheers y'all!
[^1]: I'll see what I can do about open-sourcing some of our internal scripts and helper tooling around this, perhaps in a GitHub Gist or something, but no promises.
ethankhall
commented
2 years ago For those interested, I took @klardotsh's changes and made them compatible with main (as of Sep 26, 2022) and able to build a static sorbet for linux-arm. You can find the changes here https://gist.github.com/ethankhall/516ddcefd22297a6cb9f736648386292.
technicalpickles
commented
2 years ago @bhuga this is the ARM64 issue I was telling you about
Nealsoni00
commented
1 year ago Is there any expectation that ARM Linux will be supported in the future?
The issue makes Sorbet inoperable for a small team like mine developing on ARM Macs with everything unified into docker 😞. Our CI is also running using ARM and has the same challenge. I would LOVE to add the benefit of sorbet's type system!
v2kovac
commented
1 year ago Is there any expectation that ARM Linux will be supported in the future?
The issue makes Sorbet inoperable for a small team like mine developing on ARM Macs with everything unified into docker 😞. Our CI is also running using ARM and has the same challenge. I would LOVE to add the benefit of sorbet's type system!
@Nealsoni00 our team got around this by having a separate gemfile for your local machine and docker, you only need sorbet locally. eg:
# local gemfile dont add stuff here
eval_gemfile './Gemfile.docker'
gem 'sorbet'
gem 'tapioca'And then you can use the env variable BUNDLE_GEMFILE to specify which to use where. After this one time setup we've had no problems, been doing this since december.
tonybruess
commented
1 year ago We worked around this issue using install_if:
#
# Until Sorbet supports ARM64 we can only use the `tapioca` and `srb`
# CLI tools on Mac/Linux, but not in a Docker VM on Mac
#
# https://github.com/sorbet/sorbet/issues/4119
#
# These versions are locked because bundler fails to resolve otherwise.
# Please upgrade these gems monthly.
#
install_if -> { RUBY_PLATFORM =~ /darwin/ || RUBY_PLATFORM =~ /x86_64/ } do
gem 'sorbet', '0.5.10597'
gem 'tapioca', github: 'Shopify/tapioca', branch: 'main'
end
# We need the runtime gem on Docker VM and it works
gem 'sorbet-runtime', '0.5.10597'
farhansaed
commented
1 year ago We are running sorbet in CI on ARM and use sorbet-static and getting following
Could not find gem 'sorbet-static (= 0.5.10878)' with platforms 'aarch64-linux',
Any updates when this will be solved.
darkBuddha
commented
1 year ago This becomes more relevant every day
simoleone
commented
1 year ago So just for curiosity's sake, I just tried building for aarch64-linux on the latest release 0.5.10991.20230829150527. It worked fine.
I was able to build sorbet-static and the sorbet-static gem, and run through a few tests and everything seemed fine? Is this at a point where it simply needs to be published alongside other builds, or is there more to the story here?
zhouyuanzhen
commented
1 year ago Hi @simoleone , can you share the steps how to build the sorbet-static and the sorbet-static gem? Many thanks!
simoleone
commented
1 year ago @zhouyuanzhen I pretty much just followed the steps in the README. There's nothing special to do. Check out the code or download a recent release tarball, and follow the instructions.
If you want to create a release build, there's almost nothing special to do, except comment out the sandybridge optimizations line in .bazelrc. Or suppose you could add a new build config, either would work. After that you can run use --config=release-linux like it says in the README to build the binary.
To build the gem, loosely follow the existing buildkite script. The main thing you need to do is make sure to edit the version in the gemspec before running gem build.
tonybruess
commented
1 year ago Sorbet has been buildable on aarch64 since this PR: https://github.com/sorbet/sorbet/pull/6862
To my knowledge the only thing left here is we need Stripe folks to setup a buildkite agent that can build and release the binaries.
noizwaves
commented
1 year ago To my knowledge the only thing left here is we need Stripe folks to setup a buildkite agent that can build and release the binaries.
@tonybruess There are also some changes required to the sorbet-build-image to be buildable and runnable on arm64, which the arm64 Buildkite agents could use. I was tinkering with this a few weeks together, and put together https://github.com/sorbet/sorbet-build-image/pull/8
tonybruess
commented
1 year ago I built and released an aarch64-linux and arm64-darwin-22 version if anyone wants to take it for a test run.
source 'https://gem.fury.io/tonybruess/' do
gem 'sorbet-static', '0.5.10965'
gem 'sorbet', '0.5.10965'
end
jweir
commented
1 year ago @tonybruess looking good on Ubuntu 22.04 aarch64
Thank you very much.
mlarraz
commented
1 year ago I was tinkering with this a few weeks together, and put together sorbet/sorbet-build-image#8
This PR was just merged yesterday, so hopefully the only thing blocking this now is getting an ARM64 Buildkite agent set up
nalgenewaterbottle
commented
1 year ago What can we do to help set up the ARM64 Buildkite agent?
mlarraz
commented
1 year ago What can we do to help set up the ARM64 Buildkite agent?
I believe this has to be done by somebody at Stripe. FYI @jez
tonybruess
commented
1 year ago Two months ago the last comment on this subject was from @froydnj in Slack:
we don’t use sorbet-build-image for builds inside Stripe
Sorbet releases on linux-arm64 (and darwin-arm64) haven’t been a priority for us at the moment
Until this is prioritized internally at Stripe it feels like we shouldn't expect to see arm64 releases. We may want to consider setting up a community run agent and repo.
froydnj
commented
1 year ago We (well, @jez) has been trying to set up a buildkite agent for arm64 builds, but the current status is that the build is getting peculiar errors while setting up the required Ubuntu PPA repos (it looks like it's trying to fetch from an entirely different PPA than the one specified in the Dockerfile). Things are moving, just maybe not as quickly as one might like.
noizwaves
commented
1 year ago Many thanks for this @froydnj & @jez !
it looks like it's trying to fetch from an entirely different PPA than the one specified in the Dockerfile
Do y'all have any build logs that you can share? Because of the work on https://github.com/sorbet/sorbet-build-image/pull/8 I may be able to help.
froydnj
commented
1 year ago This is the end of the log:
#8 53.15 Cannot add PPA: 'ppa:~ubuntu-toolchain-r/ubuntu/test'.
#8 53.15 The team named '~ubuntu-toolchain-r' has no PPA named 'ubuntu/test'
#8 53.15 Please choose from the following available PPAs:
#8 53.15 * 'aarch64': AArch64
#8 53.15 * 'binutils': binutils
#8 53.15 * 'dpkg-lto': dpkg defaulting to LTO flags
#8 53.15 * 'glibc': glibc test builds
#8 53.15 * 'golang-fips': Go FIPS
#8 53.15 * 'lunar-o3': lunar O3
#8 53.15 * 'lunar-o3-v3': lunar O3 v3
#8 53.15 * 'lunar-v3': lunar v3
#8 53.15 * 'p9': POWER9
#8 53.15 * 'power8': POWER8
#8 53.15 * 'ppa': Toolchain builds
#8 53.15 * 'security': Security Uploads
#8 53.15 * 'test': Toolchain test builds
#8 53.15 * 'volatile': Volatile Packages
#8 53.15 * 'x86-64-v2': GCC x86-64-v2
#8 53.15 * 'x86-64-v3': GCC x86-64-v3Thanks for prioritizing this @jez and @froydnj! We all greatly appreciate it.
That's a strange error. Where is the docker build being run?
noizwaves
commented
1 year ago Thanks for the extra info @froydnj & @jez. We really appreciate the effort here!
The latest code from GitHub builds for me on an M1 laptop with docker build --no-cache., so I suspect something environmental is contributing. I have a couple of theories/suggestions:
add-apt-repository may have trouble in some HTTP(S)_PROXY setups, and reports the error in a general way. Is the image being built on a machine where a proxy is being used? If so, try passing the proxy env var directly to the docker build command like this: docker build --build-arg HTTP_PROXY="$HTTP_PROXY" ..
docker build --build-arg HTTP_PROXY="$HTTP_PROXY" . instead.env | grep _PROXY will show them.apt-add-repository command that achieve a similar (see patch below). Even if this doesn't work, it should yield a different error message 🤞 .diff --git a/Dockerfile b/Dockerfile
index 1a79916..4de9908 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -17,7 +17,9 @@ RUN apt-get update && \
echo "deb http://apt.llvm.org/bionic/ llvm-toolchain-bionic-9 main" | tee /etc/apt/sources.list.d/llvm.list && \
apt-get update && \
apt-get install --no-install-recommends -y nodejs yarn clang-9 && \
- add-apt-repository --yes ppa:ubuntu-toolchain-r/test && \
+ apt-key adv --no-tty --keyserver keyserver.ubuntu.com --recv-keys 1E9377A2BA9EF27F && \
+ echo "deb http://ppa.launchpad.net/ubuntu-toolchain-r/test/ubuntu bionic main" | tee /etc/apt/sources.list.d/ubuntu-toolchain-r-ubuntu-test-bionic.list && \
+ echo "# deb-src http://ppa.launchpad.net/ubuntu-toolchain-r/test/ubuntu bionic main" | tee -a /etc/apt/sources.list.d/ubuntu-toolchain-r-ubuntu-test-bionic.list && \
apt-get update && \
apt-get install --yes --only-upgrade libstdc++6 && \
cd bazel_loader && \
eric-christian
commented
1 year ago Hello,
Will this solve the problems for people on OSX with M1 chip, too? I'm able to install sorbet locally with gem install sorbet but it fails within a rails project when using bundler. So i'm a little confused about (in)compatibility.
❯ uname -a
Darwin mymacbookpro 23.0.0 Darwin Kernel Version 23.0.0: Fri Sep 15 14:41:34 PDT 2023; root:xnu-10002.1.13~1/RELEASE_ARM64_T8103 arm64
❯ bundle -v
Bundler version 2.4.19
❯ rails -v
Rails 7.1.1
❯ ruby -v
ruby 3.2.2 (2023-03-30 revision e51014f9c0) [arm64-darwin22]
❯ bundle install
Fetching gem metadata from https://rubygems.org/.........
Resolving dependencies...
Resolving dependencies...
Could not find gem 'sorbet-static' with platforms 'aarch64-linux', 'ruby', 'x86_64-linux' in rubygems repository https://rubygems.org/ or installed locally.@eric-christian What does bundle config say? I've seen this issue if force_ruby_platform: true
eric-christian
commented
1 year ago @eric-christian What does
bundle configsay? I've seen this issue ifforce_ruby_platform: true
Yes, that was indeed on. Without, it works. Thank you!
Wonder why i had that on. Has probably something to do with a rails app i work on that requires all gems for all supported platforms to be packed within the vendor/cache. 🤔
nalgenewaterbottle
commented
12 months ago This sounds close! Thank you again to those that are working on this 😊
noizwaves
commented
11 months ago Hey @froydnj & @jez - hope you are both doing well. I just wanted to check in about this Docker image build issues you were experiencing, and if there were any new blockers?
I have some bandwidth to work on this at Gusto, and would love to help out where I can.
froydnj
commented
11 months ago Hey @froydnj & @jez - hope you are both doing well. I just wanted to check in about this Docker image build issues you were experiencing, and if there were any new blockers?
I have some bandwidth to work on this at Gusto, and would love to help out where I can.
Thanks for checking in @noizwaves ! Whatever issue we were experiencing seemed to be transient; I couldn't reproduce and jez couldn't reproduce it a day or so later.
I think the part we got stuck on was tagging the resulting images on Docker Hub. Our current practice right now is tagging the x86-64 image as latest, but that...probably won't work to tag both the x86-64 and arm64 images as latest? So we have to have different tags for each one, or is there magic that ensures we can have multiple architectures tagged as latest? (None of us are Docker experts in this respect.)
JamieMagee
commented
11 months ago You can make the latest tag into a manifest list. That allows Docker to pull the correct architecture for the platform it's on without you having to specify.
You'll need to create, tag, and push two architecture specific images, probably latest-amd64 and latest-arm64 but you can call them whatever you'd like. Then you can use the docker manifest command to create and push the manifest list. Something like:
docker manifest create sorbet:latest \
--amend sorbet:latest-amd64 \
--amend sorbet:latest-arm64
docker manifest push sorbet:latestThere's a way to use the same tag for both architectures and many images out there do, but it's a bit of a pain to set up and troubleshoot to put it mildly.
A few random tips having dealt with this recently:
docker buildx --platform .... --push can take a comma-separated list of platforms. When you do this it will join them together into a single manifest list automatically and push them straight away. On the other hand, buildx can be horrendously slow to cross-build.skopeo is a useful tool if you need to multi-platform "promote" images once they're already pushed, or push them around between repos, etc.noizwaves
commented
11 months ago I'd second @JamieMagee 's approach. Failing that, it's also possible to skip the multi-platform tag and just use the platform specific tags. This does require some parametrization of the build pipeline, but that can be easier to get working.
noizwaves
commented
11 months ago The docs on docker manifest create include a very relevant example too.
Problem
We are starting to need to use ARM based Linux (using Ubuntu) but we can't use Sorbet since there is not a binary version of Sorbet that will run. When performing a
bundle installthat includessorbetwe get the following error:This is running on Ubuntu 20 LTS
Proposed solution
Add a linux
aarch64pre-built sorbet-static.