arm64 support

[MarkoSagadin]

Context

I have created a NCS docker image, however for the first release I will disable support for the arm64 architecture, since some of the packages that I expected to install (gcc-multilib) are not available there.

Since I expect more quirks with this setup I would appreciate some help from an engineer with MacOS laptop.

@vid553 @NejcKle can one of you help me testing this?

Test steps

1. Clone the repo.
2. Go to ncs-zephyr folder.
3. Remove gcc-multilib from the list of the installed apt packages in Dockerfile (found at the top).
4. Decide on a NCS version, correct it in the build_ci.sh and build_dev.sh scripts.
5. Run those two scripts in ci, dev order. I expected that something will go wrong at this setup.
6. If images successfully build, use ./run.sh <path to west top dir of the ncs repo> to enter docker environment.
7. Try using various west commands for build a project, flashing, debugging, testing (twister), etc.

Ok, probably anything related to the device access won't work, USB pass-through Docker support in MacOS is a long standing issue.

Tell me how this goes and what needs to be changed.

[MarkoSagadin]

Not sure about this, if docker selects the correct platform correctly when building, but just to be sure add --platform linux/arm64 flag to the docker build command in the build scripts.

[vid553]

I have performed the test, observations and results:

• you could have provided the modified script files (for example in a branch) instead of writing what to change for quicker evaluation
• on MacOS, the docker engine needs to be started before it can be used, consider adding this info to instructions docs or check it (and start it if not running) in the script
• you were correct that gcc-multilib is not installable on macos (16.02 E: Package 'gcc-multilib' has no installation candidate)
• the platform appears to be picked up correctly by docker (even without specifying the platform flag) as I saw ubuntu packages getting installed with arm64 specifier getting printed in various URIs
• the script however fails when getting the nrfutil, where it appears that the platform is not picked up correctly (even with the platform flag), see attached printout
• the same happens with both scripts (build and dev)
• attached is also a screenshot showing the modified docker build command - for double check

Files:

build_ci-error.txt

[vid553]

In regards to USB pass-through issues on MacOS:

• there seems to be some movement from Apple, as they are adding USB to their virtualization API (https://developer.apple.com/documentation/virtualization/vzvirtualmachineconfiguration/4360569-usbcontrollers) for the next MacOS release, not sure what this means for docker though
• Parallels, the virtualisation software that I use, has USB pass-through build in and it works seamlessly somehow (tested in Windows 11 VM, not Linux VM however), so I'm fine with using that to run this software (if you need I can test the above there, as the Linux VM that is installed is arm64 architecture)

[MarkoSagadin]

@vid553

Please change this line in the Dockerfile:

  wget https://developer.nordicsemi.com/.pc-tools/nrfutil/x64-linux/nrfutil \

to this:

wget https://developer.nordicsemi.com/.pc-tools/nrfutil/universal-osx/nrfutil \

and rebuild.

[vid553]

So, with your help we managed to get it to run on MacOS. I have succesfully build fw for our project that runs on NCS v2.6.1. The build task however is VERY slow (which is kinda expected due to the fact that docker runs containers inside Linux VM on Mac OS). So containers are built and running as amd64 arch.

I have commited the changes we did into feature/mac-os-support branch.

Additional links (maybe useful):

• https://docs.docker.com/desktop/settings/mac/?uuid=536a5338-5c41-4f25-acd8-8870da3e3301%0A#virtual-file-shares
• https://stackoverflow.com/questions/56784492/permissionerror-errno-13-permission-denied-manage-py (mentioned fix about full disk access did not made a difference, I have disabled it back and confirmed that the process still works)

[MarkoSagadin]

@vid553 please try another thing: with your current setup on the feature/mac-os-support branch add back gcc-multilib app package (at the top of the Docker file) and try running ./build_ci.sh.

The correct --platform might have fixed this.

[vid553]

You are correct, both build scripts now succeed. I can build fw for the above mentioned NCS.

❯ ./build_ci.sh
[+] Building 87.1s (7/7) FINISHED                                                                                                                                                                      docker:desktop-linux
 => [internal] load build definition from Dockerfile                                                                                                                                                                   0.0s
 => => transferring dockerfile: 3.83kB                                                                                                                                                                                 0.0s
 => [internal] load metadata for docker.io/library/ubuntu:22.04                                                                                                                                                        0.5s
 => [internal] load .dockerignore                                                                                                                                                                                      0.0s
 => => transferring context: 2B                                                                                                                                                                                        0.0s
 => [ci 1/3] FROM docker.io/library/ubuntu:22.04@sha256:340d9b015b194dc6e2a13938944e0d016e57b9679963fdeb9ce021daac430221                                                                                               0.0s
 => CACHED [ci 2/3] RUN   apt-get -y update   && apt-get -y upgrade   && apt-get -y install --no-install-recommends   wget curl unzip lcov gcc-multilib make libffi7 ca-certificates   && apt-get clean   && rm -rf /  0.0s
 => [ci 3/3] RUN   wget https://developer.nordicsemi.com/.pc-tools/nrfutil/x64-linux/nrfutil   && ls -al   && chmod +x nrfutil   && mv nrfutil /usr/bin   && nrfutil self-upgrade --to-version 7.11.1   && nrfutil i  78.5s
 => exporting to image                                                                                                                                                                                                 7.8s 
 => => exporting layers                                                                                                                                                                                                7.8s 
 => => writing image sha256:240b0a38dbb302507b6f4e8d73979fc5fcbaca36683ff3f96095a319459b4cfb                                                                                                                           0.0s 
 => => naming to docker.io/irnas/ncs-zephyr-v2.6.1-ci:latest                                                                                                                                                           0.0s 

What's next:                                                                                                                                                                                                                
    View a summary of image vulnerabilities and recommendations → docker scout quickview 

[MarkoSagadin]

@vid553 Nice! That is great to know. Please commit this.

I have an another test for you:

1. Revert back the UID and GID arguments to their original values.
2. Change the docker run command in the run.sh to the below one and run ./run.sh.

docker run -it --rm \
    --privileged \
    --volume "${HOME}":"${HOME}" \
    --volume "${1}":"${HOME}"/workdir \
    --volume /dev:/dev \
    --workdir "${HOME}"/workdir \
    --volume "/etc/group:/etc/group:ro" \
    --volume "/etc/passwd:/etc/passwd:ro" \
    --volume "/etc/shadow:/etc/shadow:ro" \
    --user "$(id -u):$(id -g)" \
    --device-cgroup-rule='c 166:* rmw' \
    irnas/ncs-zephyr-v2.6.1-dev:latest

There shouldn't be any errors at this point and you should be logged in the container with your host user, so running whoami outside and inside the container should return the same result.

[MarkoSagadin]

Nevermind my last comment, that command is not ok. Mounting the entire home folder is not a good idea.

---

I am giving up on arm64 support for now.

The main issue is that the UID and GID that are declared in the Dockerfile must match those of the host user, otherwise we get into the permission problems, where files that are created by the container user are not writable by the host user.

So, the UID and GID must be modified to match the ones of the host.

I know that this is possible in two ways:

• When building the docker image with ./build_dev.sh add the following flags: --build-arg UID=${id -u} --build-arg GID=${id -g} to the docker build command. (If building on ARM MacOS also add --platform linux/amd64). With that you should be able to run the image on the MacOs. However, this means that user needs to build the image locally and not use the defacto image provided by the project. I don't like that.
• Modify the UID and GID of the existing container user to match the UID and GID of the host user as a part of the entrypoint script as described in this blog. This looks like correct solution, however I did not test it yet.

Just to highlight one thing: this is not MacOS specific problem. Any Linux distro that creates a user with UID and GID set to something else than 1000:1000 will have this problem.

[vid553]

Required changes to make it work manually are in branch feature/mac-os-support. Besides them, the only thing that needs to be changed are the UID and GID numbers in Dockerfile as described in the last comment above.