Add arm32v7 builder

[gleich]

Create a new file called arm.Dockerfile that allows you to run it on arm32v7 CPUs such as the Raspberry Pi's CPU.

[nhooyr]

Could you elaborate on what the current problem is and how this helps?

[gleich]

At the moment the docker image for code-server only works on x86 processors (desktop processor) and returns the following error when you try to run on an arm32v7 processor (such as the Raspberry Pi):

standard_init_linux.go:211: exec user process caused "exec format error"

This is due to the fact that the docker image is built on an x86 processor. In this case, we don't even need to change the base image thanks to the fact that node has an arm image already available. So, in theory, this adds support for running code-server on a Raspberry Pi!

One problem with this is that in order for it to work on a arm32v7 processor it needs to be built on an arm32v7 processor. Sadly github-actions and travis-ci both don't support you to run your CI on arm32 CPUs. There are a few solutions to this tough:

1. Run a Jenkins server on a Raspberry Pi that you or someone on the team buys. I made a docker image that comes with Jenkins and docker installed https://hub.docker.com/repository/docker/mattgleich/arm32v7jenkins-docker
2. Try building it on an x86 with a CPU arm emulation tool such as QEMU

I am going to work to get this working and will submit a PR once done

[sr229]

@nhooyr we tried building it over at Drone but it keeps OOMing, we're still investigating what we can do with the build toolchain since it should work under normal circumstances since we run with 128GB RAM in Drone Cloud

[nhooyr]

That is unfortunate news. Not sure what we can do.

[nhooyr]

https://discourse.drone.io/t/drone-cloud-arm32-ooms-when-building/6622

[nhooyr]

Ah so we need to compile for ARM32 on ARM64.

[sr229]

I'll see on how to compile the project using 32-bit libs but AFAIK documentation to do it for Node.js is quite scarce but I'll keep looking.

[sr229]

@nhooyr AFAIK we should be compiling this on a 32-bit capable container, preferrably Raspbian, in a ARM64 environment (Raspbian can run 32-bit userland against a 64 bit kernel so it shouldn't be an issue).

[aptalca]

@sr229 I'm with linuxserver.io and we publish a code-server image. We use native builders with our jenkins and have a bunch of arm32v7 devices. If you point me to your specific build instructions, I'll try to build it and let you know.

[aptalca]

I tried a build on an odroid xu4, running armbian buster (arm32v7), kernel 4.14.165. Ran in an ubuntu bionic docker container with node 12.15.0 and yarn 1.21.1 Followed the steps from https://github.com/cdr/code-server#development yarn watch resulted in the following error:

root@3d29582a8c4c:/tmp/vscode/src/vs/server# yarn watch
yarn run v1.21.1
$ cd ../../../ && yarn watch
$ gulp watch --max_old_space_size=4095
[15:29:46] Node flags detected: --max_old_space_size=4095
[15:29:46] Respawned to PID: 8612

<--- Last few GCs --->

[8612:0x1e47950]     2561 ms: Mark-sweep 20.7 (29.5) -> 20.7 (29.5) MB, 52.8 / 0.0 ms  (average mu = 0.606, current mu = 0.001) allocation failure GC in old space requested
[8612:0x1e47950]     2614 ms: Mark-sweep 20.7 (29.5) -> 20.7 (22.0) MB, 53.4 / 0.0 ms  (average mu = 0.437, current mu = 0.001) last resort GC in old space requested
[8612:0x1e47950]     2669 ms: Mark-sweep 20.7 (22.0) -> 20.7 (22.0) MB, 54.6 / 0.0 ms  (average mu = 0.278, current mu = 0.001) last resort GC in old space requested

<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x35b0e235 <JSObject>
    0: builtin exit frame: compileFunction(this=0x2b042315 <JSGlobal Object>,0x2fd0a595 <JSArray[5]>,0x2fd0a585 <JSArray[0]>,0x25340275 <undefined>,0x253403bd <false>,0x25340275 <undefined>,0,0,0x2fd1f5e9 <String[#53]: /tmp/vscode/node_modules/typescript/lib/typescript.js>,0xb0580089 <Very long string[7797825]>,0x2b042315 <JSGlobal Object>)

    1: _compile [0x4407861d] [internal/modules/cjs/loader....

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
Aborted (core dumped)
error Command failed with exit code 134.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
error Command failed with exit code 134.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

[nhooyr]

@aptalca yea thats the same error @sr229 was getting. We need to cross compile on 64bit arm for 32bit arm.

[aptalca]

Oh I see now, my mistake. I hadn't read the linked thread on drone and incorrectly assumed drone was using arm64 builders for arm32 work and that cross compile was the issue. That's why I tried on native arm32.

[nhooyr]

Some discussion at https://github.com/cdr/code-server/issues/35

[deftdawg]

My process for cobbling together builds is as follows:

1. Build Code Server on aarch64 docker container on a X86_64 host using qemu-aarch64-static via binfmt (linking needs ~12GB of RAM; impossible on a 32-bit arch which can address a max of 4GB per process)
2. Shipping the compiled directory to a native arm6 system (steps are the same for arm32v7); in my case an RPi Zero W / RPi 2
3. Going into code-server-*-built and running npm rebuild on everything (node arm6 unofficial build needed)

That produces something that can be run via the node arm6 unofficial build via node ./build/code-server*/out/vs/server/main.js --auth none

The packing can't be done on a RPi2(1GB), because build.ts is trying to pass the entire packed binary to fs.writeFile as a buffer which easily exceeds the 1GB; from what I've read it should be possible to use streams to reduce that footprint to something more sane (streams -> https://medium.com/dev-bits/writing-memory-efficient-software-applications-in-node-js-5575f646b67f)

I'm not sure if streams could also help with the 12GB ram footprint of the build phase; I don't even know where to start investigating that one as build has way more going on.

[sr229]

I'm going to assume the build script is passing the entire thing as a buffer hence the OOM errors. I think this is something @code-asher would likely be happy to investigate to optimize the builds.

[code-asher]

The next version will be out soon and it no longer packages into a single binary so that should simplify building a bit.

[jsco2t]

Question (and if this needs to be a new issue just let me know):

I noticed that some arm binaries are being produced as part of the official releases. Ex: https://github.com/cdr/code-server/releases/tag/2.1698

I grabbed the ARM64 versions and deployed them onto a RaspberryPi just to see if they would work. I'm getting a bad image format style error:

code-server2.1698-vsc1.41.1-linux-arm64 $ ./code-server
-bash: ./code-server: cannot execute binary file: Exec format error

This leads me to believe that the current ARM binaries aren't compiled for the version of ARM the raspberryPi I am using runs (I have access to a Pi Zero W, as well as a Pi 4). Is there any chance we can get full build instructions for ARM if we want to try to compile and run the server on a raspberry pi.

• I know that running the server on a RaspberryPi device, especially the 2.x code versions, will result in an OOM error - even if I managed to get the binary to run. I also understand the experience will be substandard. That being said - I still love the idea of being able to run this on ARM devices.

[code-asher]

Hopefully with v3 we'll be able to distribute arm64 releases that work a bit better but in the meantime check out https://github.com/cdr/code-server/blob/master/doc/CONTRIBUTING.md#build for how to build.

[skabber]

I tried building on my RaspberryPi 4 according to the CONTRIBUTING.md file but ran into this error.

pi@raspberrypi:~/Projects/code-server $ yarn build
yarn run v1.22.4
$ yarn runner build
$ cd ./ci && NODE_OPTIONS=--max_old_space_size=32384 ts-node ./build.ts build
[default] rootPath is "/home/pi/Projects/code-server"
[default] codeServerVersion is "3.0.1"
[build] cleaning up old build...took 8ms
[build] building vs code...failed
Command failed: yarn gulp compile-build
/usr/bin/node[16083]: ../src/node_platform.cc:414:std::shared_ptr<node::PerIsolatePlatformData> node::NodePlatform::ForIsolate(v8::Isolate*): Assertion `data' failed.
Aborted
error Command failed with exit code 134.

error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command. 

[code-asher]

That's a new one. Are you building with Node 12?

[soyflourbread]

That's a new one. Are you building with Node 12?

Same problem, node 10.17.0, yarn 1.22.4, debian buster armhf

[skabber]

Once I switched from node 10 to 12, I get a JavaScript heap out of memory error.

[b00gizm]

Once I switched from node 10 to 12, I get a JavaScript heap out of memory error.

Same for me.

Node: v12.16.1 Yarn: 1.22.4

[deftdawg]

Gents, kindly read the entire thread before "me too" posting... Per previous posts (https://github.com/cdr/code-server/issues/1337#issuecomment-585522283), it requires 12GB of RAM to build.

[nhooyr]

Ya'll can test the npm package: yarn global add code-server. 🎉

[jguittard]

@nhooyr can you elaborate on this great news please? ^^

[nhooyr]

You can install node on a arm32v7 CPU and then run npm install -g @coder/code-server to install and use code-server.

[jguittard]

Awesome! Which exact sem-ver of node is required?

[nhooyr]

There is no restriction, the native modules are compiled against whatever version of node you have installed :)

[jguittard]

I’ll finally be able to use my iPad as a dev device with my pi4 🎉

[skabber]

Works for me! Thanks

[dreamorosi]

For future reference, the installation process requires also yarn and if you don’t have libx11-dev, libxkbfile-dev and libsecret-1-dev it won’t compile.

Other than that it works great, finally we can run it on Raspberry Pi!

[jguittard]

@dreamorosi came up to first two dependencies, thanks for the third ;)

[Sebobo]

Works great thanks !

[dreamorosi]

Has anyone been able to run this version on raspberry pi via HTTPS and use it on iPadOS?

The HTTP without certificate works fine but as we know with limitations (no clipboard). If I add certificates or I use reverse proxy it works until the login screen but then it shows a blank screen.

I think the issue is localized to iPadOS because if I had some up to access thru laptop it works fine also with HTTPS.

@jguittard @Sebobo

[nhooyr]

@dreamorosi See https://github.com/cdr/code-server/issues/715

[dreamorosi]

@dreamorosi See https://github.com/cdr/code-server/issues/715

What should I look at exactly? In that thread there are mentions to an inexistent app, a paid 3rd party service and ssh access.

[nhooyr]

Sorry linked the wrong issue at first. #715 is what I meant to link. You need a certificate signed by a real CA for iPadOS, can't use a self signed certificate.

[nhooyr]

The npm package is now just code-server. https://www.npmjs.com/package/code-server

We were able to acquire it from the previous owner.

[nhooyr]

The npm PR is now official, see https://github.com/cdr/code-server#npm for docs.

Keeping this issue open until we have a armv7 builder for static releases.

[radokristof]

@nhooyr Thanks it works great on Raspberry. Wonder if you could publish images on Docker Hub built for arm32v7 as well?

Thanks!

[aptalca]

If anyone's still interested, the linuxserver.io image is now multi-arch, supporting amd64, arm32v7 and arm64v8. It also has a bunch of mods you can install for things like docker&docker-compose, python2&3, nodejs, powershell, golang, etc. https://github.com/linuxserver/docker-code-server

EDIT: well, we tested 3.3.0 with multi-arch and it worked great on both arm32 and arm64. 3.3.1 got released right before we merged and published and unfortunately it doesn't seem to work on arm. Also noticed 3.3.1 added a whole lot of node modules as dependencies as the image size grew considerably between 3.3.0 and 3.3.1: https://github.com/cdr/code-server/issues/1704 Sorry to announce here prematurely. Our dev image with 3.3.0 seems to work still, which you can find at this docker hub endpoint: lsiodev/code-server:v3.3.0-pkg-4c123a3c-dev-9137da0f154cbad6946846cce08ee42ddd603a1a

[nhooyr]

@aptalca That's awesome!

I'll be sure to add to the README under the docker section.

Any ideas why 3.3.1 doesn't work on ARM? The release is tested on arm64 by CI so it should be good.

And I've responded re #1704

[aptalca]

@nhooyr Thanks. I'll open a new issue with logs from the arm image of 3.3.1

[aptalca]

With a little bit of help from @nhooyr we fixed up the images and pushed a new build of 3.3.1 to https://github.com/linuxserver/docker-code-server This one works on arm (passed basic gui tests)

[nhooyr]

@aptalca Congrats! Did you manually test or do you have some automated tests? I'd love to check them out if possible.

[nhooyr]

And do you guys host your own arm32 builders?

[aptalca]

Our jenkins does automated checks, just a basic docker run, take a screenshot and get the logs and packages, but for some reason in this case I guess it took the screenshot a little too early, before the gui came up: https://lsio-ci.ams3.digitaloceanspaces.com/linuxserver/code-server/v3.3.1-ls43/index.html

Yes, we have arm32 and arm64 builders: https://ci.linuxserver.io/ It also spins up some aws slaves (x86_64 and arm64) when there is a backlog

[nhooyr]

That's awesome, we'd love to adopt that test in our CI with credit of course. Can you share the source?

[aptalca]

Sure it's all open on github This is the overview of our CI: https://github.com/linuxserver/pipeline-triggers The jenkinsfiles are templated and this is the builder: https://github.com/linuxserver/docker-jenkins-builder It takes in jenkins-vars.yml (and others) and generates the Jenkinsfile In the Jenkinsfile, this is the part that does the test: https://github.com/linuxserver/docker-code-server/blob/master/Jenkinsfile#L483-L531 And it basically runs this tester script: https://github.com/linuxserver/docker-ci/blob/master/ci/ci.py and that uses this image to take the screenshot: https://github.com/linuxserver/docker-tester

You probably just need to rip out the test part and the screenshot parts. We had to centralize it and template it all since we use them over a lot of repos.