TKLBAM - restore from v14.x Proxmox (LXC) built to v14.x EC2 build not working as expected

[JedMeister]

It was recently reported on the forums that a TKLBAM backup from a v14.0 PostgeSQL appliance on Proxmox, restored to a v14.2 EC2 PostgeSQL appliance does not work as expected.

I'm almost certain that this applies to all v14.x appliances migrating data from/to LXC builds via TKLBAM. For all but a few specific appliances, the differences between v14.x builds should be near enough to not cause issues. Due to a bug in the Debian Jessie SystemD version, it does not play nice on LXC (so we had to continue using Sysvinit on Proxmox/LXC) - and I suspect that is a primary cause of issues.

However, according to the report, there may also be other issues related to LXC.

[dafyddj]

I have come up against this bug while migrating my LXC v14 servers to LXC v15 (and while testing in VirtualBox VMs installed from ISO). I believe the main issue is that the TKLBAM profile downloaded to the server is not consistent with the initial state of the LXC containers. This means that, on migration, system files and packages are changed, removed or installed that seriously break the new server.

My workaround for this was to generate an LXC specific TKLBAM profile and manually install it onto my v14 servers before running further backups that I can use to migrate to other servers. Here I use the Core appliance; others would need to generate a profile for the particular appliance that they use by downloading the container tar file from here. My steps are as follows:

mkdir workdir
cd workdir/
git clone https://github.com/turnkeylinux/tklbam-profiles.git
wget http://mirror.turnkeylinux.org/turnkeylinux/images/proxmox/debian-8-turnkey-core_14.1-1_amd64.tar.gz
mkdir rootfs outdir
tar -C rootfs -xf debian-8-turnkey-core_14.1-1_amd64.tar.gz
tklbam-profiles/bin/make-profile --profiles-conf tklbam-profiles/ rootfs/ outdir/
ssh root@old-server "tar -C /var/lib/tklbam/profile -xzvf -" <outdir/turnkey-core-14.1-jessie-amd64.tar.gz

Out of interest the difference in packages between an LXC install and an ISO install for v14 were: (the +s are LXC)

--- packages.bak        2019-02-14 01:23:18.929891999 +0000
+++ packages    2019-02-14 01:24:25.013891999 +0000
-authbind
+cgmanager
-dehydrated
+libcgmanager0
+libdbus-1-3
+libglib2.0-0
+libnih-dbus1
+libnih1
+libnl-3-200
+libnl-genl-3-200
+libtirpc1
-linux-image-3.16.0-4-amd64
-linux-image-amd64
-ntp
-python-bottle
+quota
-resolvconf
-systemd
-systemd-sysv
+systemd-shim
+sysvinit-core

I hope this is of some help to anyone else coming up against this problem.

[JedMeister]

Hi @dafyddj - Thanks so much for posting this! I'm sure that will assist other users.

Is this a a diff between v14.x ISO and v14.x LXC? Or between v15.x ISO and v14.x LXC?

Re the packages:

The systemd related packages (systemd-shim & sysvinit-core on v14.x LXC; and systemd & systemd-sysvon v14.x ISO and v15.x ISO & LXC) are different between v14.x LXC and v14.x ISO & v15.x ISO and LXC because of issues with the Jessie/v14.x version of SystemD within containers. As of v15.0, all TurnKey servers should havesystemd& systemd-sysv only (and not systemd-shim & sysvinit-core).

The authbind, dehydrated & python-bottle packages are all components of our Let's Encrypt implantation - introduced in v14.2.

Exclusion of ntp, resolvconf & linux-image-amd64 (and the specific kernel version; linux-image-3.16.0-4-amd64 in your case) are common to all LXC/Proxmox images. Obviously on a non-LXC instance, you'll need a kernel and most likely want ntp (for updating system time) and resolvconf (for networking config). quota was required in the OpenVZ templates (and is option in LXC) and has since been removed from TurnKey LXC images (starting v15.0 IIRC).

TBH, I'm not 100% sure about the remaining packages:

cgmanager
libcgmanager0
libdbus-1-3
libglib2.0-0
libnih-dbus1
libnih1
libnl-3-200
libnl-genl-3-200
libtirpc1

[dafyddj]

The diff was between LXC 14.1 (downloaded from archive) and ISO 14.2 (what I had available for testing), so that explains the dehydrated stuff. cgmanager and dependencies replace parts of systemd I think, so are probably required in its place.