Failure building packages.docker.bootstrap

[chadrik]

Hi, I started kicking the tires on spk, and my first impression is that I love that the project is already all setup to build within a container, but I'm running into an error running make packages.docker.bootstrap.

I'm building on Apple M2 chip, with MacOS 13.5.

First I ran:

make rpms

This put the rpms into ./dist/rpm/RPMS/aarch64/. Note that this is different from the expected location of x86_64 (more info here). I modified the Makefile to get past this (I will contribute back some fixes once I'm done).

Then I ran:

make packages.docker.bootstrap

Here's the output:

if [ ! -f ../dist/rpm/RPMS/*/spk-*.rpm ]; then \
    echo "Please run 'make rpm' or download the latest spk rpm from github"; \
    echo "and place it into dist/rpm/RPMS/<arch>/ before continuing"; \
    exit 1; \
    fi
cp builder.spfs.toml ../dist
docker build ../dist \
        --build-arg SPFS_VERSION=0.34.6 \
        --build-arg SPK_VERSION=0.38.0 \
        -t spk-package-builder -f Dockerfile
[+] Building 0.0s (13/13) FINISHED                                                                                                                                docker:desktop-linux
 => [internal] load .dockerignore                                                                                                                                                 0.0s
 => => transferring context: 2B                                                                                                                                                   0.0s
 => [internal] load build definition from Dockerfile                                                                                                                              0.0s
 => => transferring dockerfile: 768B                                                                                                                                              0.0s
 => [internal] load metadata for docker.io/library/centos:7                                                                                                                       0.0s
 => [internal] load build context                                                                                                                                                 0.0s
 => => transferring context: 500B                                                                                                                                                 0.0s
 => [1/8] FROM docker.io/library/centos:7                                                                                                                                         0.0s
 => CACHED [2/8] RUN yum install -y     autoconf     autoconf-archive     autogen     automake     bison     coreutils     file     flex     gcc     gcc-c++     gettext     get  0.0s
 => CACHED [3/8] RUN yum update -y                                                                                                                                                0.0s
 => CACHED [4/8] COPY rpm/RPMS/*/spfs-0.34.6-*.rpm /tmp/                                                                                                                          0.0s
 => CACHED [5/8] COPY rpm/RPMS/*/spk-0.38.0-*.rpm /tmp/                                                                                                                           0.0s
 => CACHED [6/8] RUN yum install -y /tmp/*.rpm                                                                                                                                    0.0s
 => CACHED [7/8] RUN mkdir -p /spfs-storage /spfs-origin/{objects,payloads,tags}                                                                                                  0.0s
 => CACHED [8/8] COPY builder.spfs.toml /etc/spfs.toml                                                                                                                            0.0s
 => exporting to image                                                                                                                                                            0.0s
 => => exporting layers                                                                                                                                                           0.0s
 => => writing image sha256:c286a1bc174dbfd78eba3891f7091f3cdad4876356b374386d5158577b229a37                                                                                      0.0s
 => => naming to docker.io/library/spk-package-builder                                                                                                                            0.0s

What's Next?
  View summary of image vulnerabilities and recommendations → docker scout quickview
docker run --privileged --rm -i \
    -e SPFS_FILESYSTEM_TMPFS_SIZE=10G \
    -e RUST_LOG \
    -v $PWD/../build/packages/:/spfs-storage \
    -v $PWD/..:/work spk-package-builder bash -ex -c "\
    cd /work && \
    make packages.bootstrap"
+ cd /work
+ make packages.bootstrap
SPFS_VERSION=0.34.6 SPK_VERSION=0.38.0 make -C packages bootstrap
make[1]: Entering directory `/work/packages'
spk info -r local stdfs/stdfs.spk.yaml@sources > /dev/null 2>&1 || spk make-source -v stdfs/stdfs.spk.yaml
spk mkb -r origin -v stdfs/stdfs.spk.yaml
DEBUG relaunching under spfs
/etc/profile.d/lang.sh: line 19: warning: setlocale: LC_CTYPE: cannot change locale (en_US.UTF-8): No such file or directory
/etc/profile.d/lang.sh: line 20: warning: setlocale: LC_COLLATE: cannot change locale (en_US.UTF-8): No such file or directory
/etc/profile.d/lang.sh: line 23: warning: setlocale: LC_MESSAGES: cannot change locale (en_US.UTF-8): No such file or directory
/etc/profile.d/lang.sh: line 26: warning: setlocale: LC_NUMERIC: cannot change locale (en_US.UTF-8): No such file or directory
/etc/profile.d/lang.sh: line 29: warning: setlocale: LC_TIME: cannot change locale (en_US.UTF-8): No such file or directory
 INFO saving package recipe for stdfs/1.0.0
 INFO building binary package(s) for stdfs/1.0.0
 INFO building variant:
Options: {arch=aarch64, centos=7, distro=centos, os=linux}
Additional Requirements:  None
DEBUG variant options: {arch: aarch64, centos: 7, distro: centos, os: linux}
DEBUG   build options: {}
DEBUG Resolving source package for build
DEBUG All Impossible Checks solver has finished first. Stopped remaining solver tasks.
> RESOLVE stdfs/1.0.0/src  (requested by stdfs/1.0.0/src source build)
 INFO The All Impossible Checks solver found a solution, but its output was disabled. To see its output, rerun the spk command with '--solver-to-run checks'
Installed Packages:
  local/stdfs:src/=1.0.0/src highest (required by stdfs/1.0.0/src source build) 
 Number of Packages: 1
DEBUG Resolving build environment
DEBUG Unchanged solver has finished first. Stopped remaining solver tasks.
Nothing Installed
ERROR Failed to create directory /spfs/spk/pkg/stdfs/1.0.0/3I42H3S6

Caused by:
    Function not implemented (os error 38)
make[1]: *** [stdfs/stdfs.spk] Error 1
make[1]: Leaving directory `/work/packages'
make: *** [packages.bootstrap] Error 2
make[1]: *** [docker.bootstrap] Error 2
make: *** [packages.docker.bootstrap] Error 2

Any idea what's going wrong here, or suggestions on how to debug? I was considering trying to force the docker platform to x86_64 in case the architecture is the issue, but thought I'd seek some guidance before I go stabbing in the dark.

Thanks!

[jrray]

What this looks like to me is the step

# check that source package exists || create it
spk info -r local stdfs/stdfs.spk.yaml@sources > /dev/null 2>&1 || spk make-source -v stdfs/stdfs.spk.yaml

doesn't do anything (since I see no output there), but this step is supposed to be creating a source package, then the next step

# `spk make-binary` assuming source package already exists
spk mkb -r origin -v stdfs/stdfs.spk.yaml

is failing because it can't find the source package (see other comment). Perhaps something has changed since these makefile rules were written to make that first spk info command not exit with an error code anymore? I don't know why the makefile was written this way with two steps, versus just doing spk build which would make the source package and then build it in one go.

If you want to experiment, try replacing these two lines in bootstrap/Makefile:

-spk info -r local $<@sources > /dev/null 2>&1 || spk make-source -v $<
-spk mkb -r origin -v $<
+spk build -r origin -v $<

Maybe @rydrman can comment on why the makefile is written the way it is.

[rydrman]

Is @sources right? It might be @source - I have a vague memory of needing to fix this line a long time ago but maybe it never made it into a branch. I'm out of town until Wednesday but can take a closer look after that if that's not the issue

[jrray]

Sorry, I think I misinterpreted your output.

> RESOLVE stdfs/1.0.0/src  (requested by stdfs/1.0.0/src source build)
 INFO The All Impossible Checks solver found a solution, but its output was disabled. To see its output, rerun the spk command with '--solver-to-run checks'
Installed Packages:
  local/stdfs:src/=1.0.0/src highest (required by stdfs/1.0.0/src source build) 
 Number of Packages: 1

This looks like it did find a source package, I misread the "impossible checks" output as a solver failure, but it was a solver success.

It's this output that is unusual:

Nothing Installed
ERROR Failed to create directory /spfs/spk/pkg/stdfs/1.0.0/3I42H3S6

Caused by:
    Function not implemented (os error 38)

Something unsupported on MacOS it seems. Not enough context here to know exactly what's going wrong.

[chadrik]

Something unsupported on MacOS it seems

Most of this is taking place inside the container, so the types of issues we'd be looking for would be distant edge cases, like uname reporting aarch64 instead of x86_64.

This is pretty much the extent of what's running on Mac, outside of docker:

docker.%:
    if [ ! -f ../dist/rpm/RPMS/*/spk-*.rpm ]; then \
    echo "Please run 'make rpm' or download the latest spk rpm from github"; \
    echo "and place it into dist/rpm/RPMS/<arch>/ before continuing"; \
    exit 1; \
    fi
    cp builder.spfs.toml ../dist
    docker build ../dist \
        --build-arg SPFS_VERSION=$(SPFS_VERSION) \
        --build-arg SPK_VERSION=$(SPK_VERSION) \
        -t spk-package-builder -f Dockerfile
    docker run --privileged --rm -i \
    -e SPFS_FILESYSTEM_TMPFS_SIZE=10G \
    -e RUST_LOG \
    -v $$PWD/../build/packages/:/spfs-storage \
    -v $$PWD/..:/work spk-package-builder bash -ex -c "\
    cd /work && \
    make packages.$*"

From there, make packages.bootstrap is running within docker.

Is @sources right? It might be @source

I tried switching this, no luck.

I'm out of town until Wednesday but can take a closer look after that if that's not the issue

It's not urgent, happy to wait. It would be good to confirm that this is working correctly for others, as a starting point.

[rydrman]

Circling back on this and it seems to be a syscall that we are making that is not supported in the aarch64 architecture. My mac access is limited, but I suspect that you could strace the spk process to discover what call returned the 38 errno - I'll post back if I am able to get far enough on my macbook to get more info

[rydrman]

I tried this on my Mac and, interestingly enough, had no issue running the make packages.docker target and building well past the bootstrap packages... the only possible difference is that I built the rpm with -j 1 in the cargo command to avoid overloading my machine.

Maybe it would be worth trying to regenerate the rpm?

[chadrik]

I’ll try again from scratch. Is your Mac running Apple Silicon?

[rydrman]

Ya, I'm running an M1

[chadrik]

I'm on holiday right now, so I don't have a ton of time to dig into this, but here's what I did:

From the main branch, first I made these edits:

diff --git a/packages/Dockerfile b/packages/Dockerfile
index 0ddb7890..ff864b94 100644
--- a/packages/Dockerfile
+++ b/packages/Dockerfile
@@ -31,7 +31,7 @@ RUN yum install -y \

 RUN yum update -y

-COPY rpm/RPMS/x86_64/*.rpm /tmp/
+COPY rpm/RPMS/*/*.rpm /tmp/
 RUN yum install -y /tmp/*.rpm

 RUN mkdir -p /spfs-storage /spfs-origin/{objects,payloads,tags}
diff --git a/packages/Makefile b/packages/Makefile
index fc2dfcf4..3c3da0d8 100644
--- a/packages/Makefile
+++ b/packages/Makefile
@@ -101,9 +101,9 @@ lint: $(shell find . -name "*.spk.yaml")
        spk lint $^

 docker.%:
-       if [ ! -f ../dist/rpm/RPMS/x86_64/spk-*.rpm ]; then \
+       if [ ! -f ../dist/rpm/RPMS/*/spk-*.rpm ]; then \
        echo "Please run 'make rpm' or download the latest spk rpm from github"; \
-       echo "and place it into dist/rpm/RPMS/x86_64/ before continuing"; \
+       echo "and place it into dist/rpm/RPMS/<arch>/ before continuing"; \
        exit 1; \
        fi
        cp builder.spfs.toml ../dist

Then I created the rpms from scratch:

rm -rf dist
make rpms
ll dist/rpm/RPMS/aarch64/
total 110088
drwxrwxrwx@ 4 chad  staff       128 Oct 12 08:18 .
drwxrwxrwx@ 3 chad  staff        96 Oct 12 08:08 ..
-rwxrwxrwx@ 1 chad  staff  24478388 Oct 12 08:18 spfs-0.34.6-1.aarch64.rpm
-rwxrwxrwx@ 1 chad  staff  31879616 Oct 12 08:08 spk-0.38.0-1.aarch64.rpm

Then I tried to bootstrap:

make packages.docker.bootstrap

I got what I think is the same result:

make -C packages bootstrap
make[1]: Entering directory `/work/packages'
spk info -r local stdfs/stdfs.spk.yaml@sources > /dev/null 2>&1 || spk make-source -v stdfs/stdfs.spk.yaml
spk mkb -r origin -v stdfs/stdfs.spk.yaml
DEBUG relaunching under spfs
/etc/profile.d/lang.sh: line 19: warning: setlocale: LC_CTYPE: cannot change locale (en_US.UTF-8): No such file or directory
/etc/profile.d/lang.sh: line 20: warning: setlocale: LC_COLLATE: cannot change locale (en_US.UTF-8): No such file or directory
/etc/profile.d/lang.sh: line 23: warning: setlocale: LC_MESSAGES: cannot change locale (en_US.UTF-8): No such file or directory
/etc/profile.d/lang.sh: line 26: warning: setlocale: LC_NUMERIC: cannot change locale (en_US.UTF-8): No such file or directory
/etc/profile.d/lang.sh: line 29: warning: setlocale: LC_TIME: cannot change locale (en_US.UTF-8): No such file or directory
 INFO saving package recipe for stdfs/1.0.0
 INFO building binary package(s) for stdfs/1.0.0
 INFO building variant:
Options: {arch=aarch64, centos=7, distro=centos, os=linux}
Additional Requirements:  None
DEBUG variant options: {arch: aarch64, centos: 7, distro: centos, os: linux}
DEBUG   build options: {}
DEBUG Resolving source package for build
DEBUG All Impossible Checks solver has finished first. Stopped remaining solver tasks.
> RESOLVE stdfs/1.0.0/src  (requested by stdfs/1.0.0/src source build)
Installed Packages:
  local/stdfs:src/=1.0.0/src highest (required by stdfs/1.0.0/src source build) 
 Number of Packages: 1
 INFO The All Impossible Checks solver found a solution, but its output was disabled. To see its output, rerun the spk command with '--solver-to-run checks'
DEBUG Resolving build environment
DEBUG Unchanged solver has finished first. Stopped remaining solver tasks.
Nothing Installed
ERROR Failed to create directory /spfs/spk/pkg/stdfs/1.0.0/3I42H3S6

@rydrman Are you seeing the same aarch64 rpms on your end as well? What changes did you make to get past the arch differences (just in case I'm doing something wrong)?

[chadrik]

btw, this failed in the same way with -j1 as well:

diff --git a/spfs.spec b/spfs.spec
index e008a293..34c0d8e8 100644
--- a/spfs.spec
+++ b/spfs.spec
@@ -25,7 +25,7 @@ Filesystem isolation, capture, and distribution.
 %setup -q

 %build
-cargo build --release -p spfs -p spfs-cli-main -p spfs-cli-clean -p spfs-cli-enter -p spfs-cli-join -p spfs-cli-monitor -p spfs-cli-render --verbose --all --features=server,spfs/protobuf-src,fuse-backend-rhel-7-9
+cargo build -j1 --release -p spfs -p spfs-cli-main -p spfs-cli-clean -p spfs-cli-enter -p spfs-cli-join -p spfs-cli-monitor -p spfs-cli-render --verbose --all --features=server,spfs/protobuf-src,fuse-backend-rhel-7-9

 %install
 mkdir -p %{buildroot}/usr/local/bin
diff --git a/spk.spec b/spk.spec
index f88c6636..051f7c9c 100644
--- a/spk.spec
+++ b/spk.spec
@@ -31,7 +31,7 @@ Package manager and a software runtime for studio environments
 %setup -q -n %{name}-%{version}

 %build
-cargo build --release --all --features=server,spfs/protobuf-src,fuse-backend-rhel-7-9
+cargo build -j1 --release --all --features=server,spfs/protobuf-src,fuse-backend-rhel-7-9

 %install
 mkdir -p %{buildroot}/usr/local/bin

[rydrman]

diff --git a/packages/Dockerfile b/packages/Dockerfile
index 0ddb7890..dd1dfe4a 100644
--- a/packages/Dockerfile
+++ b/packages/Dockerfile
@@ -31,7 +31,7 @@ RUN yum install -y \

 RUN yum update -y

-COPY rpm/RPMS/x86_64/*.rpm /tmp/
+COPY rpm/RPMS/aarch64/*.rpm /tmp/
 RUN yum install -y /tmp/*.rpm

 RUN mkdir -p /spfs-storage /spfs-origin/{objects,payloads,tags}
diff --git a/packages/Makefile b/packages/Makefile
index fc2dfcf4..f74449c9 100644
--- a/packages/Makefile
+++ b/packages/Makefile
@@ -101,7 +101,7 @@ lint: $(shell find . -name "*.spk.yaml")
    spk lint $^

 docker.%:
-   if [ ! -f ../dist/rpm/RPMS/x86_64/spk-*.rpm ]; then \
+   if [ ! -f ../dist/rpm/RPMS/aarch64/spk-*.rpm ]; then \
    echo "Please run 'make rpm' or download the latest spk rpm from github"; \
    echo "and place it into dist/rpm/RPMS/x86_64/ before continuing"; \
    exit 1; \
diff --git a/spk.spec b/spk.spec
index f88c6636..c68d689c 100644
--- a/spk.spec
+++ b/spk.spec
@@ -31,7 +31,7 @@ Package manager and a software runtime for studio environments
 %setup -q -n %{name}-%{version}

 %build
-cargo build --release --all --features=server,spfs/protobuf-src,fuse-backend-rhel-7-9
+cargo build --release --all -j 1 --features=server,spfs/protobuf-src,fuse-backend-rhel-7-9

 %install
 mkdir -p %{buildroot}/usr/local/bin

Ya, my patch was nothing fancy - are you also running M1 silicone or something newer?

[rydrman]

hey @chadrik - I'm still sad that I couldn't reproduce and figure this out for you. Were you able to get a hold of a linux machine or otherwise get past this issue?