arm64 deb packaging has 32-bit binaries

[vielmetti]

I'm trying to run this on a Packet Type 2A server (96-core ARMv8, 64-bit only).

The installed package seems to have 32-bit binaries only, installed with

dpkg -i code-oss_1.14.0-1497385076_arm64.deb

code-oss: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-armhf.so.3, for GNU/Linux 2.6.26, BuildID[sha1]=14b6a9ab1263520c73d76aafa59a326d291a44f8, stripped

[headmelted]

The code-oss binary is ARMv7 (i.e. Electron, as it doesn't currently build out to ARMv8), all other dependencies should be compiled to ARMv8 - this explains what you're seeing but not why.

ARMv8 should be backwards compatible via the Aarch32 extensions, although it's possible (if unlikely) that the ThunderX chips don't implement it - but I can't find any indication that this is the case.

It's unlikely, but it could be disguising an otherwise unrelated dependency problem. Can you try adding the repository (if that's an option for you) and performing an apt-get install code-oss??

If nothing else it would confirm where the problem lies here.

[OtherCrashOverride]

this explains what you're seeing but not why.

ARMv8 is backwards compatible in that it can run AArch64 and AArch32 with an AArch64 kernel. However, the 32bit "compatible" dependencies are not usually installed on a 64bit ARMv8 OS. You have to setup multi-arch: https://help.ubuntu.com/community/MultiArch

This typically introduces multiple issues since some vendor provided binaries may not be available in both AArch32 and AArch64 for a system. Furthermore, the 64bit ARMv8 kernel may not have all the required "compat" IOCTLS to function properly when a vendor device is used from 32bit userland.

For the above reasons, an ARMv8 native version of Electron (that does not exist) is required.

[headmelted]

I've had another look to see how far out a fully ARMv8 stack of dependencies is.

I think the thread to watch is:

https://bitbucket.org/chromiumembedded/cef/issues/1990/linux-add-arm-build-support

It looks like it's close enough that it should be possible to add the patches to get CEF building out, then Electron should be easy enough after that (then this repository).

I don't have enough time to maintain parallel builds of CEF and Electron as well as VS Code but I'll take a run at this next chance I get to see if I can get it working and send a PR back (there's no guarantee that Spotify would add it to their build server, so it may come back down to running the builds on a fork).

In the meantime, if you guys dpkg --add-architecture armhf does this run for you?

(Obviously the caveat of having the requisite IOCTLS remains, but are there a lot of kernels around that don't have this?)

As I've been testing on the same machine I've been working on I have multiarch configured already (although now that I've moved the builds to docker I may remove it all so that I'm seeing what you folks are seeing).

[OtherCrashOverride]

if you guys dpkg --add-architecture armhf does this run for you?

That did not work for me. You still need to install the 32bit version of the required dependencies: sudo apt install something:armhf

[OtherCrashOverride]

After going one by one installing dependencies:

sudo apt install libgtk2.0-0:armhf
sudo apt install libx11-xcb1:armhf
sudo apt install libxtst6:armhf
sudo apt install libxss1:armhf
sudo apt install libgconf-2-4:armhf
sudo apt install libnss3:armhf
sudo apt install libasound2:armhf
sudo apt install libstdc++6:armhf
sudo apt install libatk-adaptor:armhf

The execution fails:

[28255:0620/060141.005166:FATAL:udev_loader.cc(38)] Check failed: false.
#0 0x0000011cd9dc <unknown>
#1 0x0000011b9d0a <unknown>
#2 0x000001fd9fd8 <unknown>
#3 0x000001fda260 <unknown>
#4 0x000001fda3fe <unknown>
#5 0x00000160fe46 <unknown>
#6 0x0000011bc490 <unknown>
#7 0x0000011f78fe <unknown>
#8 0x0000011f7b4c <unknown>
#9 0x0000011f7dee <unknown>
#10 0x0000011be042 <unknown>
#11 0x0000011f76f2 <unknown>
#12 0x0000011e2186 <unknown>
#13 0x000001e56842 <unknown>
#14 0x000001e5692e <unknown>
#15 0x000001209010 <unknown>
#16 0x0000011eddf8 <unknown>

Aborted

The above installs a bunch of extra stuff (armhf) since the dependencies have their own dependencies. Again, a native AArch64 Electron build is needed. I posted this info for anyone that wants to further experiment. The "workaround" is still "use a different IDE". ;-)

[vielmetti]

I found this issue for Electron

https://github.com/electron/electron/issues/5706

with the existence of a commit that gives arm64 support at

https://github.com/electron/electron/pull/9547

[headmelted]

Just saw this, good spot!

I haven't had a chance to pick this up yet due to some family commitments but I'll be able to get back to it very soon.

The ideal of course would be that Electron start pushing arm64 binaries and we can then just import those and have fully 64-bit arm builds immediately.

Will wait for a reply on the other thread.

On 7 Jul 2017 22:15, "Edward Vielmetti" notifications@github.com wrote:

I found this issue for Electron

electron/electron#5706 https://github.com/electron/electron/issues/5706

with the existence of a commit that gives arm64 support at

electron/electron#9547 https://github.com/electron/electron/pull/9547

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/headmelted/codebuilds/issues/25#issuecomment-313794843, or mute the thread https://github.com/notifications/unsubscribe-auth/AKYVoItJMI6nE4qV3j3mJ1EJ7CTzglxbks5sLp_bgaJpZM4N5HJu .

[vielmetti]

There's some progress here

https://github.com/electron/libchromiumcontent/issues/334

if you're keeping score.

[mauritslamers]

Would it be useful to remove the arm64 builds from packagecloud.io to prevent confusion until a real arm64 build is available?

[magic-k]

electron 1.8.0 beta with arm64 support has been released. https://github.com/electron/electron/releases/tag/v1.8.0 We're getting closer :-)

[headmelted]

I should be able to use this immediately.

Will get a look this week at implementing as soon as I get a chance. Exciting times!! :-)

On 4 Sep 2017 10:05, "magic-k" notifications@github.com wrote:

electron 1.8.0 beta with arm64 support has been released. https://github.com/electron/electron/releases/tag/v1.8.0 We're getting closer :-)

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/headmelted/codebuilds/issues/25#issuecomment-326908364, or mute the thread https://github.com/notifications/unsubscribe-auth/AKYVoFbai_CvGLG9Zt2qgZWa5s5EG0ZNks5se71kgaJpZM4N5HJu .

[mauritslamers]

Any tips on how to contribute to the build process? I have been trying to cross-compile vscode, but running into problems with references to electron npm packages, which all refer to older versions which do not support arm64.

[mauritslamers]

Update on my progress. I managed to compile on arm64 by using an arm64 environment instead of cross-compiling.

I took the master branch of https://github.com/microsoft/vscode and edited the package.json to include electronVersion: "1.8.0" instead of 1.7.7. To get rid of the same kind of errors I encountered during my attempts to cross-compile, I commented out the line reading npmInstall('test'smoke'); in build/npm/postinstall.js. This line starts the installation of the test environment, and as part of the dependencies it tries to download the current version of electron through its npm package. There is no package for 1.8.0 nor will it run on arm64, so it doesn't make sense (yet) to fix this. The same kind of problem arose with a direct dependency called vscode-ripgrep, which is a npm package downloading the correct binary from github. As there is no arm64 version release of this package, I had to manually compile it.

I still don't have a .deb package though, because for some reason the minifying process of vscode (npm run gulp vscode-linux-arm-min as indicated on https://github.com/Microsoft/vscode/wiki/Cross-Compiling-for-Debian-Based-Linux) tries to contact a microsoft server through https but encounters a "Connection reset by peer" error, which might be caused by that site not provide a correct certificate. (It tries to contact https://13.85.19.92/ which serves a certificate for market.visualstudio.com, but that domain doesn't exist it seems).

Edit: documenting steps and add detailed information

[magic-k]

Funny, at first that IP gave me also an cerificate error but on second try i got directly to marketplace.visualstudio.com with a green https.

[mauritslamers]

Tiny tidbit I forgot to mention: vscode does run! For me the IP address doesn't change for marketplace.visualstudio.com when looking it up with a browser. I have found that the problem originates from the builtin extensions of which the process tries to get information from the marketplace.visualstudio.com API (build/lib/extensions.js#L67) but why this fails is yet unclear.

[jkol]

@mauritslamers - How did you go about manually compiling vscode-ripgrep such that the npm install script doesn't fail?

[mauritslamers]

I didn't make the npm install script pass, I just let it fail but let it keep the installed files in. Then I compiled vscode-ripgrep on the side, and replace the binary in the npm package.

[headmelted]

Hi all,

I'm sorry for not having checked in the last couple of weeks - had a few personal matters come up out of the blue that couldn't wait, and it just hasn't left a moment to look into this further.

@mauritslamers I'd be eager to discuss the exact steps you followed in greater detail as it doesn't seem like anything that we shouldn't be able to work into the scripts. The edits described should be possible with the cross-compiler toolchain unless I'm mistaken - and I'd really like to leverage the work you've done to get proper ARMv8 support into the builds.

I'm getting a look through the scripts at the minute to see if I can make any headway here :-)

[vielmetti]

@headmelted - any progress on arm64? Let us know if the timing is good to make another run at this.

[vielmetti]

There is now an Electron CI set up on Works on Arm for arm64, by the way.

https://github.com/WorksOnArm/cluster/issues/58

[magic-k]

I was able to build vscode in an 64 bit chroot on my arm64 chromebook. Runs out of the box with current master.

[vielmetti]

@magic-k do you think things are stable enough to build a .deb for this?

[magic-k]

Didn't build a .deb yet. I would wait for the 1.26 release that is imminent. That release contains the necessary electron2.x integration. The arm builds were always quite stable though.

[vielmetti]

Thanks @magic-k - is there a specific place where that 1.26 release is being talked through?

[magic-k]

Iteration plan: https://github.com/Microsoft/vscode/issues/53850 They are a bit behind schedule-

[headmelted]

I'm working as quickly as possible to get true ARM64 builds up-and-running (and to restore the 32-bit builds to running nightly) but I'm hitting issues at the Microsoft side.

Unfortunately ARM support in the upstream code conflates all ARM variants to "arm" which causes the kinds of issues we're seeing in this thread. That being said, Electron now has official 32-bit and 64-bit ARM support that classifies the different variants correctly enough that we can proceed ("armv7l" and "arm64" at their end) - and I know the steps I need to take to get nightly 100% 64-bit ARM builds going, so I just need to get someone at Microsoft nudged to work with me on this.

Incidentally, if anyone has a point of contact on the code team it would be a big help in getting this done.

[headmelted]

Effectively, I need to seperate out "arm" into two seperate builds that are "armhf" and "arm64" (or "armv7l" and "aarch64" for rpm users). I have the patches needed to do this ready all down the stack, but Microsoft hosts some tools it uses in other repositories that I can't patch (as I'd need to patch Microsoft's code to call a fork of those tools, and it quickly becomes a mess that prevents me from submitting a pull request for this to Microsoft in the future).

[headmelted]

@magic-k

It's great that we can now build Code in a chroot on a native box, but considering how little extra work would be involved to support cross-compiling I'd really like it if we could also cross-compile to ARM64 in the CI system to provide continuous delivery of binaries to users on that architecture (not least because I have quite a few non-tech-savvy folks depending on these builds for an educational setting where Chromebooks have been rolled out en masse). It also brings us closer to having official binary support for people that need that.

I'm glad in any case that you were able to get a working build for the Chromebook though as I feel bad that I've been unable to get the ARM64 builds up-and-running nightly yet. 👍

[vielmetti]

@headmelted If you can identify any specific people or projects I might reach out to help you sort out the Microsoft side of things, happy to help.

[headmelted]

I believe the latest deb should be fully aarch64. That's not to say that it's working of course (I haven't had a chance to test yet).

Can you try this:

https://packagecloud.io/headmelted/codebuilds/packages/ubuntu/xenial/code-oss_1.28.0-1537541296_arm64.deb

And let me know?

[c0d1f1ed]

Tested on NVIDIA Jetson Xavier. Works like a charm!

[headmelted]

Success!! 😁👍

Thanks for letting me know!

On Fri, Sep 21, 2018, 18:24 Nicolas Capens notifications@github.com wrote:

Tested on NVIDIA Jetson Xavier. Works like a charm!

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/headmelted/codebuilds/issues/25#issuecomment-423612328, or mute the thread https://github.com/notifications/unsubscribe-auth/AKYVoM23zXhA-BdNeLxK7WW3ASgRELoiks5udSDTgaJpZM4N5HJu .

[headmelted]

Nightlies should be back up-and-running again now, will check over scripts at weekend to make sure no bugs related to the 64-bit builds

On Fri, Sep 21, 2018, 18:25 Jay Rodgers headmelted@gmail.com wrote:

Success!! 😁👍

Thanks for letting me know!

On Fri, Sep 21, 2018, 18:24 Nicolas Capens notifications@github.com wrote:

Tested on NVIDIA Jetson Xavier. Works like a charm!

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/headmelted/codebuilds/issues/25#issuecomment-423612328, or mute the thread https://github.com/notifications/unsubscribe-auth/AKYVoM23zXhA-BdNeLxK7WW3ASgRELoiks5udSDTgaJpZM4N5HJu .

[mauritslamers]

Can confirm that the 64bit ARM runs. Running it on a Pinebook with 64bit Ubuntu.

[mauritslamers]

One thing: it complains about "no extension gallery service configured", so no extensions can be installed. Is this done on purpose?

[vielmetti]

Confirmed that it's running on a Cavium ThunderX system (with remote X terminal to my laptop across the continent, so not exceptionally quickly). I also see the "no extension gallery service configured" error.

[headmelted]

I have patches to bring that up, didn't want to apply them until I got it working - I'll get them restored within the next couple of days :-)

On Fri, Sep 21, 2018, 20:05 Edward Vielmetti notifications@github.com wrote:

Confirmed that it's running on a Cavium ThunderX system (with remote X terminal to my laptop across the continent, so not exceptionally quickly). I also see the "no extension gallery service configured" error.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/headmelted/codebuilds/issues/25#issuecomment-423640760, or mute the thread https://github.com/notifications/unsubscribe-auth/AKYVoJ6AD8CP326Grpxbzb9Eu-ZArDaIks5udTiFgaJpZM4N5HJu .