kas-container: Can not start Bitbake server Operation not permitted

[skycaptain]

Hi,

I'm trying to build core-image-minimal on a Mac using kas-container with docker and docker-machine. However, when running a very minimal example, I'm getting the following error message:

$ ./kas-container build kas.yml
2020-12-19 22:35:49 - INFO     - kas 2.3.3 started
2020-12-19 22:35:49 - INFO     - /repo$ git rev-parse --show-toplevel
2020-12-19 22:35:49 - INFO     - /repo$ git rev-parse --show-toplevel
2020-12-19 22:35:49 - INFO     - Using /repo as root for repository meta-custom
2020-12-19 22:35:49 - INFO     - /work$ git clone -q https://git.yoctoproject.org/git/poky /work/poky
2020-12-19 22:37:11 - INFO     - Repository poky cloned
2020-12-19 22:37:11 - INFO     - /repo$ git rev-parse --show-toplevel
2020-12-19 22:37:11 - INFO     - Using /repo as root for repository meta-custom
2020-12-19 22:37:11 - INFO     - /work/poky$ git status -s
2020-12-19 22:37:12 - INFO     - /work/poky$ git rev-parse --verify -q origin/thud
2020-12-19 22:37:12 - INFO     - e52122a3e6912575ff401a4af6ac1bf3070092bc
2020-12-19 22:37:12 - INFO     - /work/poky$ git checkout -q e52122a3e6912575ff401a4af6ac1bf3070092bc -B thud
2020-12-19 22:37:17 - INFO     - /repo$ git rev-parse --show-toplevel
2020-12-19 22:37:18 - INFO     - Using /repo as root for repository meta-custom
2020-12-19 22:37:18 - INFO     - /work/poky$ /tmp/tmp3plp2byg/get_bb_env /work/build
2020-12-19 22:37:18 - INFO     - /repo$ git rev-parse --show-toplevel
2020-12-19 22:37:18 - INFO     - Using /repo as root for repository meta-custom
2020-12-19 22:37:18 - INFO     - /repo$ git rev-parse --show-toplevel
2020-12-19 22:37:18 - INFO     - Using /repo as root for repository meta-custom
2020-12-19 22:37:18 - INFO     - /work/build$ /work/poky/bitbake/bin/bitbake -k -c build core-image-minimal
NOTE: Retrying server connection... (Traceback (most recent call last):
  File "/work/poky/bitbake/lib/bb/main.py", line 446, in setup_bitbake
    server = bb.server.process.BitBakeServer(lock, sockname, configuration, featureset)
  File "/work/poky/bitbake/lib/bb/server/process.py", line 402, in __init__
    self.sock.bind(os.path.basename(sockname))
PermissionError: [Errno 1] Operation not permitted
)
WARNING: /work/poky/bitbake/lib/bb/main.py:481: ResourceWarning: unclosed <socket.socket fd=12, family=AddressFamily.AF_UNIX, type=SocketKind.SOCK_STREAM, proto=0>
  logger.info("Retrying server connection... (%s)" % traceback.format_exc())

NOTE: Reconnecting to bitbake server...
NOTE: Previous bitbake instance shutting down?, waiting to retry...
NOTE: Retrying server connection...
ERROR: Unable to connect to bitbake server, or start one
2020-12-19 22:37:39 - ERROR    - Command returned non-zero exit status 1

Contents of kas.yml:

# Every file needs to contain a header, that provides kas with information
# about the context of this file.
header:
  # The `version` entry in the header describes for which configuration
  # format version this file was created for. It is used by kas to figure
  # out if it is compatible with this file. The version is an integer that
  # is increased on every format change.
  version: 1
# The machine as it is written into the `local.conf` of bitbake.
machine: qemux86-64
# The distro name as it is written into the `local.conf` of bitbake.
distro: poky
repos:
  # This repo contains the kas.yml file - there is no need to fetch it again.
  # It's current revision will be used to perform the build.
  meta-custom:
  # Here we include a list of layers from the poky repository to the
  # bblayers.conf:
  poky:
    url: https://git.yoctoproject.org/git/poky
    refspec: thud
    layers:
      meta:
      meta-poky:
      meta-yocto-bsp:

[jan-kiszka]

Sorry, I've no experience with docker on Mac, specifically around traps and pitfalls of using unix sockets (which is apparently where bitbake stumbles). I could imagine that your host-side filesystem which we pass through via kas-container is not compatible with unix sockets. Maybe search the web around that first and/or try running

import socket
sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
sock.bind('test-socket-file')

natively in python on the Mac fs you use as workdir.

If that works, the issue is likely around how docker on Mac passes the fs through. But, again, no idea where that could come from (no one I know is doing Mac-hosted embedded Linux development...).

If you post this issue also to the kas mailing list, chances are higher that someone else reads it and has an idea.

[gramss]

I think I was having a slightly similar issue.

I wanted to use docker inside a Debian docker and was unable to connect to my docker engine. I solved it by forwarding /var/usr/docker.sock (path might be wrong) via -v directly into my docker. So I used the sock from the Mac host.

Maybe this could help you as well? I'm also trying to use kas on my Mac to build the meta-iot2050 image natively..

But I had the limitation that MacOS is just not a Debian lacking some basic tools like apt that are used in scripts here. So it seems I need to go down the full Debian docker road. Or try the Parallels Desktop Preview and work on a Debian VM..

Hope that helped a bit..! 😊 Let me know if you solved your problem. I'm eager to see if and how the M1 Chip boosts the build time (especially for the kernel headers for arm boards that take a lot of time on x86 platforms due to other reasons)

[jan-kiszka]

I think we cannot help with Macs from a kas perspective here. If you think otherwise, please reopen.

[skycaptain]

@jan-kiszka I found the issue to be the accient character limit on socket paths and the way Docker constructs it's paths. So on Mac the path to sockets is limited to 104 characters (107 characters on Linux). When Bitbake starts, it opens a socket at ${TOPDIR}/bitbake.sock. Now, the issue is that even when you are working under /work/build/... inside the container with the Docker volume mount, the actual path to the socket has your local machine path prepended. So it's more like <your-local-path>/work/build/...; which could easily exceed the character limit. This paths also seems to be hardcoded; so it's an issue on the Bitbake side, not allowing to create the socket in e.g. /tmp or /var/run inside the container. As a workaround, use a shorter path on the host machine. BTW the same issue exists with VSCode devcontainer.

[Jak-o-Shadows]

I can confirm that I encountered this issue on WSL, on a sshfs mounted remoted folder, using Xilinx Petalinux. My path was: /mnt/z/Myuser/programming/ebaz4205/petalinux/projects/wsltest I resolved this issue by moving the path to /home/jak/petalinux/petalinux/projects/wsltest