implement workaround for systemd-tmpfiles failures

[frasertweedale]

Note: please rebase-merge, not squash-merge.

45fc7b2 (Fraser Tweedale, 2 hours ago)
   replace systemd-tmpfiles with tmpfiles.py

   Overwrite /bin/systemd-tmpfiles with our tmpfiles.py program.  This is
   needed because the FreeIPA installer explicitly invokes systemd-tmpfiles in
   some places.  Those invocations fail, which causes installation to fail.

   This is expected to be a temporary change until FreeBSD provides a way to
   select an alternative tmpfiles implementation via the ipapython.paths
   facility.

6c62d48 (Fraser Tweedale, 2 hours ago)
   tmpfiles.py: implement --prefix

   Implement the --prefix option for restricting the set of paths that are
   acted upon.

   Also add the --create, --remove and --clean options to conform to the
   interface of systemd-tmpfiles(8).  --remove and --clean are not 
   implemented, but the CLI options are recognised.

43e5b90 (Fraser Tweedale, 5 hours ago)
   tmpfiles.py: handle missing 'chattr' program

   If chattr(1) doesn't exist on the system (container), emit a notice instead
   of crashing the program.

e9c2f4d (Fraser Tweedale, 26 hours ago)
   use tmpfiles.py instead of systemd-tmpfiles

   Convert to tmpfiles.py instead of systemd-tmpfiles for initialising the
   tmpfiles.  This avoids the path processing problems encountered by
   systemd-tmpfiles due to emptyDir mounts being owned by unmapped users when
   running the workload in an isolated user namespace.

   Remove the unit test, because it appears the bats-mock framework cannot
   handle mocking a program specified by full path (i.e. containing slashes).

f048b7b (Fraser Tweedale, 5 days ago)
   add tmpfiles.py

209ce3b (Fraser Tweedale, 2 days ago)
   add tmpfiles config

   The IPA container needs a few extra tmpfiles directives related to the
   /data volume - mainly to solve some dangling symlinks.  Add a dedicated
   tmpfiles config file that runs early.

4554475 (Fraser Tweedale, 2 weeks ago)
   tolerate chmod/chown failure when populating templates

   In some container environments it is not possible to chown and chmod some
   volumes.  In particular, Kubernetes/OpenShift emptyDir mounts for /tmp,
   /run, etc, when running in user namespaces, will be owned by root on the
   host, which is unmapped in the user namespace.

   Tolerate the failure of chmod/chown operation.  We do this by
   "modifying" the /usr/local/bin/populate-volume-from-template program. 
   Rather than writing a modified version of the program, we do it via sed and
   pipe the result to the /bin/sh for execution.

   The result of the sed program is to transform the lines:

       chown --reference="$VOLUME-template" "$VOLUME"
      chmod --reference="$VOLUME-template" "$VOLUME"

   into (folded for readability):

       chown --reference="$VOLUME-template" "$VOLUME" \
          || ( echo "Failed to chown $VOLUME" ; ls -ld "$VOLUME" )
      chmod --reference="$VOLUME-template" "$VOLUME" \
          || ( echo "Failed to chmod $VOLUME" ; ls -ld "$VOLUME" )

da5e879 (Fraser Tweedale, 2 weeks ago)
   rename "populate tmp" step

   Use a more accurate name for the step that populates "tmp".  In particular,
   the goal is to disambiguate it from the helper routine that actually
   invokes /usr/local/bin/populate-volume-from-template.

77dd638 (Fraser Tweedale, 2 weeks ago)
   container-step-volume-update: tolerate chown/chmod failure

   In some container environments it is not possible to chown and chmod 
   particular volumes.  In particular, emptyDir mounts for /tmp, /run etc on
   OpenShift, when running in user namespaces, will be owned by root on the
   host, which is unmapped in the user namespace.

   Tolerate the failure of the chmod/chown operations.  Instead, merely emit a
   notice when it fails.

[frasertweedale]

@tiran thanks for the review. PR updated.

[tiran]

@frasertweedale, @avisiedo and I spent the better half of the afternoon debugging a problem with the script. IPA server installation is failing with systemd-tmpfiles replacement. Something related to Apache, Kerberos, GSS-Proxy, or IPA API is misbehaving.

During our investigation I realized two problems:

• chattr is not available, which resulted in silent errors.
• the script does not implement specifiers (variable substitution), see https://www.freedesktop.org/software/systemd/man/tmpfiles.d.html SPECIFIER

IMO it's going to be much more work to re-implement systemd-tmpfiles than we initially anticipated. I have a proposal for a different implementation:

• Rename the original implementation mv /usr/bin/systemd-tmpfiles /usr/bin/systemd-tmpfiles.orig
• Copy our replacement to /usr/bin/systemd-tmpfiles
• Call the original implementation from our Python implementation with all supported arguments
• Finally run our replacement code. Only perform additional fixups and apply differences that are necessary for our container requirements. Would this work?

[frasertweedale]

The failure was found to be due to 'd' (and other directory-related commands) only performing chmod/chown when it creates the directory. If the directory already existed, chmod/chown was not performed. This diverges from systemd-tmpfiles behaviour and was the cause of the failure (specifically related to directives in /usr/lib/tmpfiles.d/tmp.conf). The behaviour has been corrected and @avisiedo successfully launched the container in a non-user-ns environment.