[feature request] Multi-arch Docker support

[jmreicha]

I am building a homelab with arm64 (rock64) boards, it would be great to get freeipa working on arm.

[adelton]

Based on https://www.freeipa.org/page/ARM, that is well possible. Is there anything specific that is needed on the containerization side?

[jmreicha]

Not that I'm aware of. I think the trick is to either emulate arm builds on x86 with something like qemu or build directly on arm servers with something like https://www.worksonarm.com/cluster/.

[jmreicha]

As a quick test I built an image using the qemu method. The image seems to build without any problems, but when I try running the container it is slow and doesn't ever fully start, I'm guessing because of the way I built my image?

Anyway, here's the output.

systemd 234 running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN2 +IDN default-hierarchy=hybrid)
Detected virtualization container-other.
Detected architecture arm64.
Set hostname to <rancher.thepracticalsysadmin.com>.
Initializing machine ID from random generator.
Failed to open netlink: Protocol not available
Failed to install release agent, ignoring: File exists
systemd-journald.service: Start operation timed out. Terminating.
systemd-journald.service: State 'stop-sigterm' timed out. Killing.
systemd-journald.service: Main process exited, code=killed, status=31/SYS
container-ipa.target: Job container-ipa.target/start failed with result 'dependency'.
systemd-journald.service: Unit entered failed state.
systemd-journald.service: Failed with result 'timeout'.
systemd-journald.service: Service has no hold-off time, scheduling restart.
systemd-journald.service: Start operation timed out. Terminating.
systemd-journald.service: State 'stop-sigterm' timed out. Killing.
systemd-journald.service: Main process exited, code=killed, status=31/SYS
systemd-journald.service: Unit entered failed state.
systemd-journald.service: Failed with result 'timeout'.
systemd-journald.service: Service has no hold-off time, scheduling restart.

Here's the modified Dockerfile I made to build for arm64.

FROM arm64v8/fedora:27
# Enable builds from x86 based systems
COPY qemu-aarch64-static /usr/bin
...

[adelton]

This however is likely generic issue of running systemd inside of container on aarch64, under qemu, right? If that's so, it needs to be reported to those projects. Here we really want to focus on anything that is specific about running FreeIPA in container. While we might be able to help debug some of the issues external to FreeIPA in container, I guess we are getting very far from what we are able to help with at this point.

[jmreicha]

There seem to be other issues going on as well. I tried building directly on an arm64 board (rock64) to try to remove the qemu issue and got the following error.

...
Step 7/47 : RUN sed -i '/installutils.verify_fqdn(config.master_host_name, options.no_host_dns)/s/)/, local_hostname=False)/' /usr/lib/python2.7/site-packages/ipaserver/install/server/replicainstall.py && python -m compileall /usr/lib/python2.7/site-packages/ipaserver/install/server/replicainstall.py
 ---> Running in 86ecaf98e7c8
sed: can't read /usr/lib/python2.7/site-packages/ipaserver/install/server/replicainstall.py: No such file or directory
The command '/bin/sh -c sed -i '/installutils.verify_fqdn(config.master_host_name, options.no_host_dns)/s/)/, local_hostname=False)/' /usr/lib/python2.7/site-packages/ipaserver/install/server/replicainstall.py && python -m compileall /usr/lib/python2.7/site-packages/ipaserver/install/server/replicainstall.py' returned a non-zero code: 2

[jmreicha]

It looks like this error was just fixed in latest master. I am now able to successfully build and start the container on arm64.

@adelton Would there be any interest in adding this to the the Docker official image project? Leveraging this project would allow use of their build infrastructure, making various cross platform architecture images easy to build without creating much extra work.

[adelton]

Nod, the python 2 issue was fixed via https://github.com/freeipa/freeipa-container/pull/240.

[adelton]

As for adding FreeIPA arm container images to the Docker official image project, the problem is not adding it, it's the maintenance I'm concerned about. We've had Travis CI enabled on this repo for a couple of months and it's been very helpful. Are you aware of any CI offering which supports arm, which we could use for testing PRs and branches?

[jmreicha]

Hmm I haven't heard of any but I'll look around.

I think the idea for using the Docker official repo would be to submit a PR there when the changes made here are stable, so things could be vetted on the normal build/test flow here and then build cross platform images when things look good.

[jmreicha]

@adelton If you're interested, apparently Shippable offers a free tier for public repos and offers support for building on arm machines. I don't know how easy it is to build arm Docker images though 😞

[jmreicha]

@adelton I am attempting to use Shippable to build an arm64 PoC but the build is failing when running the tests/run-master-and-replica.sh script and I'm not sure why, maybe you could shed some light? Here's the link the failed build, https://app.shippable.com/github/jmreicha/freeipa-container/runs/9/1/console.

As far as I can tell, the only difference between this build and successful builds is the named service failed to start line just before the job fails.

[jmreicha]

Ugh, it looks like I was building against the wrong Dockerfile.

After I pointed to the right Dockerfile I got the build working and tests passing for arm64 if you're interested https://app.shippable.com/github/jmreicha/freeipa-container/runs/12/1/console.

[jmreicha]

@adelton I got a PoC working with using Shippable https://github.com/jmreicha/freeipa-container/blob/master/shippable.yml. This could be adapted to build and push an arm64 only version only (with build matrix) if you wanted.

[adelton]

Hello @jmreicha, I tried to look into the Shippable approach, basically trying to reproduce what we do with Travis CI. However, with plain

language: none
branches:
  only:
    - shippable

build:
  ci:
    - tests/run-partial-tests.sh Dockerfile.fedora-29
    - docker ps -aq | while read i ; do docker rm -f $i ; done
    - docker build -t local/freeipa-server:fedora-29 -f Dockerfile.fedora-29 .
    - tests/run-master-and-replica.sh local/freeipa-server:fedora-29

I'm getting wait_for_exit failure (but not timeout) after cca 7 to 10 minutes -- for example https://app.shippable.com/github/adelton/freeipa-container/runs/52/1/console. The tests/systemd-container-ipa-server-install-data.sh gets killed (?) in the docker exec freeipa-server-container-fedora-29 id bob step or after the MACHINE_ID was assigned and we try to do another docker exec.

Have you got any idea how to debug this further?

[abbra]

@adelton at first, I suspected it might be https://pagure.io/freeipa/issue/7860 which I'm intending to fix next week. In short, 389-ds stopped using /etc/sysconfig to store own systemd configuration snippets and we need to adopt as we did already for httpd in past.

However, when running the same test on up to date F29 I've got no problem at all:

$ docker="sudo podman" tests/run-partial-tests.sh Dockerfile.fedora-rawhide
....
Done configuring DNS key synchronization service (ipa-dnskeysyncd).
Restarting ipa-dnskeysyncd
Restarting named
Updating DNS system records
Configuring client side components
Using existing certificate '/etc/ipa/ca.crt'.
Client hostname: ipa.example.test
Realm: EXAMPLE.TEST
DNS Domain: example.test
IPA Server: ipa.example.test
BaseDN: dc=example,dc=test
Configured sudoers in /data/etc/nsswitch.conf
Configured /etc/sssd/sssd.conf
Systemwide CA database updated.
SSSD enabled
Configured /etc/openldap/ldap.conf
/etc/ssh/ssh_config not found, skipping configuration
/etc/ssh/sshd_config not found, skipping configuration
Configuring example.test as NIS domain.
Client configuration complete.
The ipa-client-install command was successful
This program will set up FreeIPA client.
Version 4.7.2

The ipa-server-install command was successful

==============================================================================
Setup complete

Next steps:
    1. You must make sure these network ports are open:
        TCP Ports:
          * 80, 443: HTTP/HTTPS
          * 389, 636: LDAP/LDAPS
          * 88, 464: kerberos
          * 53: bind
        UDP Ports:
          * 88, 464: kerberos
          * 53: bind

    2. You can now obtain a kerberos ticket using the command: 'kinit admin'
       This ticket will allow you to use the IPA tools (e.g., ipa user-add)
       and the web user interface.
    3. Kerberos requires time synchronization between clients
       and servers for correct operation. You should consider enabling chronyd.

Be sure to back up the CA certificates stored in /root/cacert.p12
These files are required to create replicas. The password for these
files is the Directory Manager password
Password for admin@EXAMPLE.TEST: 
----------------
Added user "bob"
----------------
  User login: bob
  First name: Bob
  Last name: Nowak
  Full name: Bob Nowak
  Display name: Bob Nowak
  Initials: BN
  Home directory: /home/bob
  GECOS: Bob Nowak
  Login shell: /bin/sh
  Principal name: bob@EXAMPLE.TEST
  Principal alias: bob@EXAMPLE.TEST
  Email address: bob@example.test
  UID: 959400001
  GID: 959400001
  Password: False
  Member of groups: ipausers
  Kerberos keys available: False
uid=959400001(bob) gid=959400001(bob) groups=959400001(bob)
-rw-------. 1 root root            3893740 Feb 23 08:56 /data/var/log/ipaserver-install.log
-rw-r-----+ 1 root systemd-journal 8388608 Feb 23 08:56 /data/var/log/journal/f97fe68f64d2d6adb945a4ef680ee2c2/system.journal
ls: cannot access '/var/log/journal/f97fe68f64d2d6adb945a4ef680ee2c2': No such file or directory
C /etc
C /etc/machine-id
A /lextab.py
A /yacctab.py
OK tests/systemd-container-ipa-server-install-data.sh.
OK tests/run-partial-tests.sh.

So I suspect this could be podman version issue in Ubuntu build used in travis CI.

[adelton]

@abbra, did you mean to put this reply to https://github.com/freeipa/freeipa-container/issues/186?

[abbra]

@adelton yes, sorry, put to wrong one.

[adelton]

Thanks. I've hidden it here to minimize confusion.

[jmreicha]

@adelton It's been awhile since I've looked at this but I will give it a poke and see if I can reproduce.

[jmreicha]

@adelton So I have been poking around a little bit. I have a different shippable.yaml file now that I'm using to test, which is getting farther but still getting hung up.

Here is my console output - https://app.shippable.com/github/jmreicha/jobs/build_arm64/builds/5c785c5eb2f57f060000a9f1/console

Here the shortened output.

Restarting ipa-dnskeysyncd
Restarting named
named service failed to start
Updating DNS system records
Configuring client side components
This program will set up FreeIPA client.
Version 4.7.2

FreeIPA server configuration failed.
The container has exited with .State.ExitCode [123].
Thu Feb 28 22:39:17 UTC 2019

I vaguely remember seeing something similar when I got this working originially, unfortunately I don't remember what the fix was 😭 I'm going to keep digging here but that should at least show you what I'm doing at this point.

[jmreicha]

How much memory do the tests need to run? It looks like the default free tier builds machines have 2 cpus and 3.75 GB of memory.

I was able to get the tests to pass locally for ARM, but I have a much beefier setup locally.

[adelton]

Red Hat's recommendations for IdM in RHEL at https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/linux_domain_identity_authentication_and_policy_guide/installing-ipa#server-hw-recomendations states 3 GB of RAM and 1 GB swap should work.

[jmreicha]

Well that might be it. According to Shippable there's not a way to tell what kind of resource utilization is happening on the arm nodes.

I might look at a different method, multiarch support would be really nice.

[msnelling]

Has anyone tried using the BuildKit feature? https://dev.to/zeerorg/build-multi-arch-docker-images-on-travis-5428

[adelton]

I guess the point is not so much building the images as is testing the images and changes that go to this repository.

[jwhb]

An arm64 variant would be great to run IPA on Apple Silicon Podman Machine.