RFC: add a qemu big endian platform to CI

joholl commented 3 years ago

I don't think we explicitly support big endian platforms (correct me if I'm wrong). As it turns out, the TSS does not only build and run on big endian machines - apparently, we are packaged for such platforms, as well. At least for me, that was a surprise.

However, we don't have a CI big endian platform, which makes it hard to reproduce and fix issues (see #2125). When fixing the tcti-pcap, I had to set up a qemu machine with an exotic processor architecture from scratch.

Therefore my RFC: what do you think about spinning up a big endian platform in the CI pipeline? Not only would we prevent regressions in the future, but also we'd have a "documented" (at least in code :D) way on how test the TSS on big endian platforms. (If we explicitly support big endian platforms is a separate question, though).

As for the technical part: Since I've already spent some time setting up a s390x qemu machine running alpine (at least to the point where the unit tests run), I think we're half-way there. For reference, see my notes below:

# based on https://wiki.alpinelinux.org/wiki/S390x/Installation 
if [ ! -f alpine-standard-3.10.0-s390x.iso ]; then 
    wget -q 'http://dl-cdn.alpinelinux.org/alpine/v3.10/releases/s390x/alpine-standard-3.10.0-s390x.iso' 
fi 

# create virtual disk
qemu-img create alpine_disk.qcow2 5G

# start vm
sudo qemu-system-s390x -M s390-ccw-virtio \ 
       -m 1024 -smp 2 -nographic \ 
       -nic user \ 
       -hda alpine_disk.qcow2 \ 
       -boot d -cdrom alpine-standard-3.10.0-s390x.iso 

# Login: root 

# ============ run inside vm (this needs to be scripted) ============ 
setup-alpine -q  # interactive: choose all defaults (can be scripted by providing an answers file)

apk update 
apk add acl automake autoconf coreutils cmocka-dev g++ git json-c-dev curl-dev libtool libuuid yaml-dev m4 openssl-dev pkgconfig make linux-headers wget 

# autoconf-archive 
autoconf_archive=autoconf-archive-2018.03.13 
wget "http://mirror.kumi.systems/gnu/autoconf-archive/$autoconf_archive.tar.xz" 
tar -xf $autoconf_archive.tar.xz; rm $autoconf_archive.tar.xz 
cd $autoconf_archive 
./configure --prefix=/usr && make -j $(nproc) && make install 
cd .. 

# tpm_server 
ibmtpm_name=ibmtpm1637 
wget -L "https://downloads.sourceforge.net/project/ibmswtpm2/$ibmtpm_name.tar.gz" 
mkdir -p $ibmtpm_name && tar xv --no-same-owner -f $ibmtpm_name.tar.gz -C $ibmtpm_name && cd $ibmtpm_name/src 
make -j$(nproc) 
ln -s $PWD/tpm_server /bin/tpm_server 
cd ../.. 

# build and test TSS
git clone https://github.com/tpm2-software/tpm2-tss.git && cd tpm2-tss 
touch groupadd useradd && chmod +x groupadd useradd 
./bootstrap && PATH=$PATH:. ./configure --enable-unit
# make fails, so we run it in a loop... we probably need to up the RAM
while ! make check -j TESTS=""; do echo "trying again..."; done
make -j check

williamcroberts commented 3 years ago

I've spun up machines like this for testing the SE Linux kernel in Travis. However, last I knew GH CI runners didn't have nested virt turned on.

JuergenReppSIT commented 1 year ago

@joholl You did only enable the unit tests. Was it possible to start tpm_server and the integration tests?

joholl commented 1 year ago

@JuergenReppSIT When I did this, I only needed the unit tests. Although I (unnecessarily) built the mssim simulator (tpm_server), I did not run the integration tests. For that, I would have to had to install more dependencies.

joholl commented 1 year ago

With issue #2649, I revisited this.

Github Action `run-on-arch-action`

I tried uraimo/run-on-arch-action, which is qemu-based (but idk how exactly). Currently I am struggeling with the slowness (granted, based on a vanilla ubuntu22-04).

ubuntu22.04 s390x
The job running on runner GitHub Actions 3 has exceeded the maximum execution time of 360 minutes.

Github Actions

[.github/workflows/main.yml](https://github.com/joholl/tpm2-tss/blob/93ebc9485afaf0014e22fd65a4d4f51a3179d5f6/.github/workflows/main.yml): ```yaml on: [push, pull_request] jobs: build_job: # The host should always be linux runs-on: ubuntu-22.04 name: ${{ matrix.distro }} ${{ matrix.arch }} # Run steps on a matrix of 4 arch/distro combinations strategy: matrix: include: - arch: aarch64 distro: ubuntu22.04 - arch: armv7 distro: ubuntu22.04 - arch: s390x distro: ubuntu22.04 steps: - uses: actions/checkout@v3 - uses: uraimo/run-on-arch-action@v2 name: Build and test id: build with: arch: ${{ matrix.arch }} distro: ${{ matrix.distro }} run: | set +x uname -a apt-get update apt-get install -y \ autoconf-archive \ curl \ libcmocka0 \ libcmocka-dev \ net-tools \ build-essential \ git \ pkg-config \ gcc \ g++ \ m4 \ libtool \ automake \ libgcrypt20-dev \ libssl-dev \ autoconf \ gnulib \ wget \ doxygen \ libdbus-1-dev \ libglib2.0-dev \ clang \ clang-tools \ pandoc \ lcov \ libcurl4-openssl-dev \ dbus-x11 \ vim-common \ libsqlite3-dev \ iproute2 \ libtasn1-6-dev \ socat \ libseccomp-dev \ expect \ gawk \ libjson-c-dev \ libengine-pkcs11-openssl \ default-jre \ default-jdk \ sqlite3 \ libnss3-tools \ python3 \ python3-pip \ libyaml-dev \ libmbedtls-dev \ uuid-dev \ opensc \ gnutls-bin \ rustc \ acl \ libjson-glib-dev \ libusb-1.0-0-dev \ libftdi-dev \ uthash-dev \ swtpm #python3 -m pip install --upgrade pip #pkgs="cryptography==$PYCRYPTO_VERSION pyyaml cpp-coveralls pyasn1 pyasn1_modules python-pkcs11 bcrypt==$PYBCRYPT_VERSION setuptools"; \ # pkgs=$(echo "$pkgs" | sed -E 's/==\s+/ /g'); \ # python3 -m pip install $pkgs ./bootstrap ./configure --enable-unit --enable-integration --disable-tcti-libtpms --enable-self-generated-certificate make check -j$(($(nproc)-1)) CFLAGS="-Wno-error" TESTS="" make check -j$(($(nproc)-1)) || { find ../ -name test-suite.log -exec cat {} +; exit 1; } ```

multiarch/qemu-user-static

Another alternative would be multiarch/qemu-user-static which seems to be even more promising. Basically, with docker run --privileged it registers qemu-user-static binary interpreters in the host system. We could then directly run our foreign-arch docker images (assuming we switch to multi-arch builds):

ARG ARCH=
FROM ${ARCH}debian:buster-slim

# rest of Dockerfile

JuergenReppSIT commented 11 months ago

I could run all integration tests on s390x/ubuntu:jammy with the following branch: https://github.com/JuergenReppSIT/tpm2-tss/tree/test-use-libtpms-on-big-endian I will create a pull request after main-fapi.c will be adapted to the latest rework of the tests. The docker image used for the tests was created with:

docker pull s390x/ubuntu
sudo apt-get install -y qemu-system-s390x
docker run --rm --privileged multiarch/qemu-user-static --reset -p yes
docker build -t s390x_ubuntu_jammy_update -f s390x-ubuntu-22.04.docker .

The docker file for the update was created from: https://github.com/JuergenReppSIT/tpm2-software-container/blob/s390x-ubuntu-22.04/s390x-ubuntu-22.04.docker.m4

joholl commented 11 months ago

Cool!

I'm surprised integration_tcti in your configure.ac is not overwritten by the tcti autodetect further down:

https://github.com/JuergenReppSIT/tpm2-tss/blob/bf5f5be6ae72c9b745f523ff5cb03c3a5c2b4588/configure.ac#L450-L453

In any case, I wanted to have something like this for the configuration step anyway. We might also want to replace --with-device=/dev/tpm0 with --with-integrationtcti=device:/dev/tpm0 later.

What is missing now is the integration in our ci. I've played around with this as well. In my case I tried to separate distro and arch a little bit into a (half-baked) matrix build. In any case, we need something like this in tpm2-software/ci.

https://github.com/joholl/ci/blob/5a1b3cb02a129865fbf3b0238589a55094d3a0bf/scripts/ci.sh#L43-L60