🐞 aarch64 userspace is now appearing for some machines as of M110, so need to rebuild all packages for aarch64...

[satmandu]

There is a limited armv7l userspace on the new aarch64 userspace enabled builds, so we need to figure out how to move forward with these new machines.

I would suggest NOT creating a hacked up multilib build of old chromebrew packages and new ones for aarch64. Instead, why don't we figure out how to bootstrap pure aarch64 binaries for those machines which now have these new base images?

I would suggest NOT having an upgrade path, aside from suggesting running the installer again on these machines.

Config files can probably be kept at this point.

Here is a docker image built from lazor, without chromebrew installed: crewbase-lazor-aarch64.m110.sh

#!/bin/bash
# crewbase-lazor-aarch64.m110.sh
if [ -n "$SSH_CLIENT" ] || [ -n "$SSH_TTY" ]; then
  SESSION_TYPE=remote/ssh
elif pstree -p | egrep --quiet --extended-regexp ".*sshd.*\($$\)"; then
  SESSION_TYPE=remote/ssh
else
  case $(ps -o comm= -p $PPID) in
    sshd|*/sshd) SESSION_TYPE=remote/ssh;;
  esac
fi
X11=-e
X11+=" "
X11+=DISPLAY=${DISPLAY:-:0.0}
X11+=" "
if ! [[ $SESSION_TYPE == remote/ssh ]] && [ -d /tmp/.X11-unix ]; then
  X11+=" -v /tmp/.X11-unix:/tmp/.X11-unix "
fi
if [ -f "$HOME"/.Xauthority ]; then
  X11+=--volume=$HOME/.Xauthority:/home/chronos/user/.Xauthority:rw
  X11+=" "
  X11+=--volume=$HOME/.Xauthority:/home/chronos/.Xauthority:rw
fi
docker pull --platform linux/arm64 satmandu/crewbase:lazor-armv7l.m110
docker pull tonistiigi/binfmt
docker run --privileged --rm tonistiigi/binfmt --install all
docker run --platform linux/arm64 --rm --net=host ${X11}  -v $(pwd)/pkg_cache:/usr/local/tmp/packages -v $(pwd):/output -h $(hostname)-aarch64 -it satmandu/crewbase:lazor-armv7l.m110 /bin/bash -c "passwd -d chronos ; echo 'chronos ALL=(ALL) NOPASSWD: ALL' >> /etc/sudoers ; sudo -i -u chronos /bin/bash -i"

[satmandu]

Also we need to shift our architecture detection from kernel architecture to userspace architecture... since an aarch64 kernel could now mean either an aarch64 userspace or an armv7l userspace...

[uberhacker]

Sounds complicated but I agree. We also need to split out amd64 and x86_64 binaries to account for the processor differences unless we can figure out something compatible with both.

[satmandu]

Steps in the container above to get ruby working, by first installing nix: This may also work for AMD machines after you have disabled the sudo password...

export PAGER=more
sh <(curl -L https://nixos.org/nix/install) --no-daemon
source /nix/var/nix/profiles/per-user/chronos/profile/etc/profile.d/nix.sh
nix-channel --add https://nixos.org/channels/nixpkgs-unstable
nix-channel --update
nix-env -iA nixpkgs.ruby_3_1
nix-env -iA nixpkgs.git
nix-env -iA nixpkgs.llvmPackages_15.libllvm
nix-env -iA nixpkgs.llvmPackages_15.clang-unwrapped
nix-env -iA nixpkgs.lld_15
nix-env -iA nixpkgs.mold
nix-env -iA nixpkgs.rsync
nix-env -iA nixpkgs.vim
nix-env -iA nixpkgs.pixz
nix-env -iA nixpkgs.openssl_1_1

To get a list of available nix packages: nix-env -qaP Example of searching packages using a keyword: nix-env -qaP ruby Example of installing a package: nix-env -iA nixpkgs.ruby_3_1

[satmandu]

This is one way of setting LIB_SUFFIX on both aarch64 and x86_64 userspace:

LIB_SUFFIX=
# See https://superuser.com/a/1369875
# If /bin/bash is 64-bit, then set LIB_SUFFIX, as this is true on both
# x86_64 and aarch64 userspace
[ $(od -An -t x1 -j 4 -N 1  /bin/bash) == "02" ] && LIB_SUFFIX='64'

[satmandu]

We also need to figure out now how we want to redo our architectures...

This is how I see it now. I suggest we designate architectures by compatible userspace...

Kernel	Userspace	CPU	Compatibility	Current crew Designation	Notes	Suggested Designation
x86	x86	Intel	A	i686		i686
x86_64	x86_64	Intel	B	x86_64		x86_64
x86_64	x86_64	AMD	C	x86_64 (broken!)	Built-in libraries use AMD specific instructions (superset of x86_64). Also, we don't have any emulators which can handle this instruction set when on an intel machine.	amd64
armv7l	armv7l		D	armv7l		armv7l
aarch64	armv7l		D	aarch64		armv7l
aarch64	aarch64		E		Device rollout is as of 3/2023 limited (e.g., lazor).	aarch64

[thomas725]

since I'm affected by this issue https://github.com/chromebrew/chromebrew/issues/8090 and you have the "help wanted" label applied here - is there anything I can do to help get this done? I don't have too much experience with ruby sadly, but I know my way around linux systems in general and am handy with bash scripts if those skills could help?

[satmandu]

We need to have a design discussion about how we want to delineate the new architecture and how we want to split it out, and then we need to bootstrap the install for this architecture, building a toolchain and so forth, since the default install doesn't give us much.

One way to help would be if you could tar up what dev_install installs to /usr/local, as that might help us bootstrap an install in a container.

sudo /usr/bin/dev_install should do that for you. (It doesn't work in a container, sadly.)

[thomas725]

Okey, since you say it installs to /usr/local/ the first thing I'll do is clear out the last broken chromebrew installation: rm -Rf /usr/local/* (no hidden folders exists since installing actual packages didn't work)

After running for ~10 seconds the command you mentioned asks:

Install virtual/target-os-dev package now? (y/N)

Should I accept the default no option or do you want me to enter yes?

Here's the state of /usr/local/ while it's waiting for an answer to that prompt: https://ufile.io/dtc3h1gr (~70mb, link will expire after "a maximum of 30 days", whatever that means, just used the first registration free upload service that duckduckgo gave me..)

[satmandu]

Thanks! I suspect that you will want to hit y on that question, but let me see what this does too...

[thomas725]

Okey, I tried selecting yes, but it seems it couldn't comply with that request:

Install virtual/target-os-dev package now? (y/N) y                                               
!!! PORTAGE_BINHOST unset, but use is requested.
!!! Problem with sandbox binary. Disabling...

Calculating dependencies... done in 0:00:00.012595

emerge: there are no binary packages to satisfy "virtual/target-os-dev".

emerge: searching for similar names...
emerge: Maybe you meant virtual/os-headers?
ERROR dev_install: [dev_install.cc(546)] Could not install virtual/target-os-dev

[satmandu]

Hmm. Do these steps work for you?

sudo su -
emerge sshfs-fuse

(Those steps are taken from https://www.chromium.org/chromium-os/how-tos-and-troubleshooting/install-software-on-base-images/ )

[satmandu]

This is a WIP... Before starting the container:

mkdir -p aarch64_build
curl -OLf https://gitlab.com/api/v4/projects/26210301/packages/generic/dev_install/lazor-m110_aar
ch64/dev_install-lazor-m110-chromeos-aarch64.tar.gz

Start the container in the first post.

./crewbase-lazor-aarch64.m110.sh

Extract the aforementioned devinstall package:

sudo -s
tar fxv /output/aarch64_build/dev_install-lazor-m110-chromeos-aarch64.tar.gz -C /

Move files in /usr/share/dev-install/portage/make.profile/package.provided out of the way.

mv /usr/share/dev-install/portage/make.profile/package.provided/* /tmp/

Create some portage dirs...

mkdir -p /etc/portage/repos.conf
nano /etc/portage/repos.conf/gentoo.conf

Enter the following from https://wiki.gentoo.org/wiki/Project:Portage/Sync :

[gentoo]
location = /var/db/repos/gentoo
sync-type = rsync
sync-uri = rsync://rsync.gentoo.org/gentoo-portage
auto-sync = yes

Comment out EMERGE_DEFAULT_OPTS:

sed -i 's/EMERGE_DEFAULT_OPTS/#EMERGE_DEFAULT_OPTS/' /usr/share/dev-install/portage/make.profile/make.defaults

Then try to update portage information:

emerge-webrsync -x -v
emerge --sync
emerge -qav python
USE='-rsync-verify' emerge -qavO portage
emerge -qav portage

[thomas725]

emerge sshfs-fuse

the sudo su works in giving me a root shell, but this one doesn't look like it did what you hoped:

# emerge sshfs-fuse
!!! PORTAGE_BINHOST unset, but use is requested.
!!! Problem with sandbox binary. Disabling...

!!! Problem with sandbox binary. Disabling...

Calculating dependencies... done in 0:00:00.027463

emerge: there are no binary packages to satisfy "sshfs-fuse" for /usr/local/.

emerge: searching for similar names... nothing similar found.

Got a read-only filesystem there:

# mkdir -p /etc/portage/repos.conf
mkdir: cannot create directory '/etc/portage/repos.conf': Read-only file system

the folder contains this:

# find /etc/portage/                 
/etc/portage/
/etc/portage/make.profile
/etc/portage/make.profile/package.provided
/etc/portage/make.profile/package.provided/toolchain.conf
/etc/portage/make.profile/package.provided/board-profile.conf
/etc/portage/make.profile/make.defaults

[satmandu]

Currently stuck here after the above steps trying to update portage or install gcc or glibc...

rpi4b-aarch64 /usr/share/dev-install/portage # USE=glibc emerge -qav sys-apps/portage
!!! Problem with sandbox binary. Disabling...

 * IMPORTANT: 4 news items need reading for repository 'gentoo'.
 * Use eselect news read to view new items.

!!! All ebuilds that could satisfy "sys-apps/portage" have been masked.
!!! One of the following masked packages is required to complete your request:
- sys-apps/portage-9999::gentoo (masked by: missing keyword, invalid: RDEPEND: USE flag 'elibc_glibc' referenced in conditional 'elibc_glibc?' is not in IUSE)
- sys-apps/portage-3.0.45.3-r1::gentoo (masked by: invalid: RDEPEND: USE flag 'elibc_glibc' referenced in conditional 'elibc_glibc?' is not in IUSE)
- sys-apps/portage-3.0.45.2-r1::gentoo (masked by: invalid: RDEPEND: USE flag 'elibc_glibc' referenced in conditional 'elibc_glibc?' is not in IUSE)
- sys-apps/portage-3.0.44-r1::gentoo (masked by: invalid: RDEPEND: USE flag 'elibc_glibc' referenced in conditional 'elibc_glibc?' is not in IUSE)
- sys-apps/portage-3.0.43-r1::gentoo (masked by: invalid: RDEPEND: USE flag 'elibc_glibc' referenced in conditional 'elibc_glibc?' is not in IUSE)

For more information, see the MASKED PACKAGES section in the emerge
man page or refer to the Gentoo Handbook.

[thomas725]

So ChromeOS is based on Gentoo? Or did it just borrow their package-manager?

[satmandu]

So ChromeOS is based on Gentoo? Or did it just borrow their package-manager?

It is very much based upon Gentoo, and uses their package manager for setup. But as far as I can tell a lot of the gentoo functionality is dependent upon being setup inside google. :/ Also dev_install has been sporadically been broken on and off for the last several years. See https://issuetracker.google.com/issues/187793205

[satmandu]

Also, some of the steps I am taking will ONLY work in a container unless you go full dev-mode and make your rootfs R/W. (I haven't done that to ANY of my ChromeOS devices...)

[satmandu]

(The container I am using is the docker container in the first post... and you need to run that on a different machine... since I don't have docker working yet inside chromeos...)

[thomas725]

okey. I'm unsure if I should maybe try Debian or Manjaro on my Chromebook... ChromeOS in the state it has on my Chromebook is not really making me content with it anymore :(

[satmandu]

The problem with other distributions is that you need to make sure you have kernel support for your hardware... The ChromeOS kernel is open source, but it doesn't automatically lend itself to being used with other distributions. There was a project using ChromeOS kernels with Debian/Ubuntu distributions on ChromeOS x86_64 hardware, but that has more or less been abandoned.

[satmandu]

Also worth noting that the ONLY publicly accessible aarch64 Chrome Browser builds to date are in these M110 builds of ChromeOS for the lazor boards.

There is no aarch64 build of Chrome Browser available for other aarch64 Linux distributions.

[thomas725]

okey, so if I understood your comments correctly, that means trying a generic arm manjaro build, for example: https://github.com/manjaro-arm/generic-images/releases/download/23.02/Manjaro-ARM-xfce-generic-23.02.img.xz on my device will not work or at least not well?

Though probably if it works at all, to get acceptible touch screen support I guess wayland -> kde plasma or gnome should be prefered over xfce: https://github.com/manjaro-arm/generic-images/releases/download/23.02/Manjaro-ARM-kde-plasma-generic-23.02.img.xz

[satmandu]

You can try a nixos install too in /usr/local... (Since that has an aarch64 install)...

But using any distribution on bare hardware is dependent upon the kernel being patched to support the hardware properly...

With arm hardware you might also need the appropriate dtb files to work with the kernel.

[thomas725]

I naively thought I'll just boot from an usb installation media like on any other PC, but it seems that's not as easy as I thought on Chromebooks. I've stumbled upon https://mrchromebox.tech/ but he seems to not support arm based Chromebooks at all :(

[thomas725]

You can try a nixos install too in /usr/local... (Since that has an aarch64 install)...

do you have a set of instructions on how to achieve that? I've tried sh <(curl -L https://nixos.org/nix/install) --no-daemon (and also the deamon version) but both ended with this error:

/dev/fd/63: 117: /dev/fd/63: /tmp/nix-binary-tarball-unpack.oQwLrnuGa1/unpack/nix-2.13.3-aarch64-linux/install: Permission denied

I guess the problem is that /tmp/ is a no-exec mount on chromeOS if I understood that correctly, so I'd somehow need to make the nixos package manager installation script to put it's executeables into /usr/local

[satmandu]

You can remount tmp without that flag...

[thomas725]

hah! you're right, should have tought of that myself ;)

mount -o remount,exec /tmp/

but strangely, even though now the "noexec" flag is gone:

# mount | grep /tmp
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,relatime,nosymfollow,seclabel)

I still get the same error:

/dev/fd/63: 117: /dev/fd/63: /tmp/nix-binary-tarball-unpack.oQwLrnuGa1/unpack/nix-2.13.3-aarch64-linux/install: Permission denied

[BitRanger]

I naively thought I'll just boot from an usb installation media like on any other PC, but it seems that's not as easy as I thought on Chromebooks. I've stumbled upon https://mrchromebox.tech/ but he seems to not support arm based Chromebooks at all :(

I stumbled across this https://github.com/hexdump0815/imagebuilder project that has builds of debian/ubuntu for chromebooks and you can check if your chromebook is supported in the readme.

[thomas725]

hey @Nitro01010 - yes I've found that too and successfully booted one of those images on my Chromebook, see https://github.com/hexdump0815/imagebuilder/issues/47 for details. (Though my interest has shifted towards other [not chromebook related] projects for now)

[satmandu]

Thanks to @mio-19 and https://github.com/chromebrew/chromebrew/issues/8398#issuecomment-1594025222 I've started to get access to the dev environment for the aarch64 machines.

After installing the tarball of the dev_install at that link, I then ran:

source /etc/profile
emerge -a virtual/target-chromium-os-dev

Now to figure out how to get gcc emerged...

[satmandu]

The big question is really how to get the toolchain packages: gcc, glibc, binutils

For instance, in /usr/share/dev-install/portage/make.profile/package.provided/toolchain.conf We have this:

sys-devel/gcc-10.2.0-r30
sys-libs/glibc-2.35-r17
dev-lang/go-1.20-r2

It would be really nice to find the binary packages for those.

[satmandu]

 emerge sys-libs/gcc
!!! PORTAGE_BINHOST unset, but use is requested.
!!! Problem with sandbox binary. Disabling...

!!! Problem with sandbox binary. Disabling...

Calculating dependencies... done in 0:00:00.284262

emerge: there are no binary packages to satisfy "sys-libs/gcc" for /usr/local/.

emerge: searching for similar names...
emerge: Maybe you meant any of these: sys-libs/gdbm, sci-libs/gsl?
rpi4b-aarch64 /usr/local/etc/portage 

[satmandu]

Sigh...

rpi4b-aarch64 / # env | grep PORTAGE
PORTAGE_CONFIGROOT=/usr/local
PORTAGE_BINHOST=https://commondatastorage.googleapis.com/chromeos-dev-installer/board/strongbad/15393.58.0/packages
rpi4b-aarch64 / # emerge sys-devel/gcc-10.2.0-r30
!!! PORTAGE_BINHOST unset, but use is requested.
!!! Problem with sandbox binary. Disabling...

!!! Problem with sandbox binary. Disabling...

!!! 'sys-devel/gcc-10.2.0-r30' is not a valid package atom.
!!! Please check ebuild(5) for full details.
rpi4b-aarch64 / # emerge sys-devel/gcc           
!!! PORTAGE_BINHOST unset, but use is requested.
!!! Problem with sandbox binary. Disabling...

!!! Problem with sandbox binary. Disabling...

Calculating dependencies... done in 0:00:00.284224

emerge: there are no binary packages to satisfy "sys-devel/gcc" for /usr/local/.

emerge: searching for similar names...
emerge: Maybe you meant any of these: sys-devel/gdb, sys-devel/patch, sys-devel/flex?
rpi4b-aarch64 / # emerge sys-devel/gdb
!!! PORTAGE_BINHOST unset, but use is requested.
!!! Problem with sandbox binary. Disabling...

!!! Problem with sandbox binary. Disabling...

Calculating dependencies... done in 0:00:00.372437

emerge: there are no binary packages to satisfy "dev-libs/gmp:0/10.4=" for /usr/local/.
(dependency required by "sys-devel/gdb-11.2-r3::chromiumos" [binary])
(dependency required by "sys-devel/gdb" [argument])
rpi4b-aarch64 / # emerge sys-devel/gcc-10.2.0    
!!! PORTAGE_BINHOST unset, but use is requested.
!!! Problem with sandbox binary. Disabling...

!!! Problem with sandbox binary. Disabling...

!!! 'sys-devel/gcc-10.2.0' is not a valid package atom.
!!! Please check ebuild(5) for full details.

[mio-19]

Sigh...

rpi4b-aarch64 / # env | grep PORTAGE
PORTAGE_CONFIGROOT=/usr/local
PORTAGE_BINHOST=https://commondatastorage.googleapis.com/chromeos-dev-installer/board/strongbad/15393.58.0/packages
rpi4b-aarch64 / # emerge sys-devel/gcc-10.2.0-r30
!!! PORTAGE_BINHOST unset, but use is requested.
!!! Problem with sandbox binary. Disabling...

!!! Problem with sandbox binary. Disabling...

!!! 'sys-devel/gcc-10.2.0-r30' is not a valid package atom.
!!! Please check ebuild(5) for full details.
rpi4b-aarch64 / # emerge sys-devel/gcc           
!!! PORTAGE_BINHOST unset, but use is requested.
!!! Problem with sandbox binary. Disabling...

!!! Problem with sandbox binary. Disabling...

Calculating dependencies... done in 0:00:00.284224

emerge: there are no binary packages to satisfy "sys-devel/gcc" for /usr/local/.

emerge: searching for similar names...
emerge: Maybe you meant any of these: sys-devel/gdb, sys-devel/patch, sys-devel/flex?
rpi4b-aarch64 / # emerge sys-devel/gdb
!!! PORTAGE_BINHOST unset, but use is requested.
!!! Problem with sandbox binary. Disabling...

!!! Problem with sandbox binary. Disabling...

Calculating dependencies... done in 0:00:00.372437

emerge: there are no binary packages to satisfy "dev-libs/gmp:0/10.4=" for /usr/local/.
(dependency required by "sys-devel/gdb-11.2-r3::chromiumos" [binary])
(dependency required by "sys-devel/gdb" [argument])
rpi4b-aarch64 / # emerge sys-devel/gcc-10.2.0    
!!! PORTAGE_BINHOST unset, but use is requested.
!!! Problem with sandbox binary. Disabling...

!!! Problem with sandbox binary. Disabling...

!!! 'sys-devel/gcc-10.2.0' is not a valid package atom.
!!! Please check ebuild(5) for full details.

Yes I basically can't find any binary packages on my system for some reason

[github-actions[bot]]

This issue has gone 120 days without comment. To avoid abandoned issues, it will be closed in 21 days if there are no new comments. If you're the original submitter of this issue, please comment confirming if this issue still affects you in the latest release with crew update && yes | crew upgrade, or close the issue if it has been fixed. If you're another user also affected by this bug, please comment confirming so. Either action will remove the stale label. This bot exists to prevent issues from becoming stale and forgotten. Chromebrew is always moving forward, and bugs are often fixed as side effects of other changes. We therefore ask that bug report authors remain vigilant about their issues to ensure they are closed if fixed, or re-confirmed - perhaps with fresh logs or reproduction examples - regularly.

[Zopolis4]

The issue is still present, so it shouldn't be auto closed.

[satmandu]

Mark is as confirmed.