TKLDev should check for new bootstrap (and other bits)

This issue has been re-purposed - see https://github.com/turnkeylinux/tracker/issues/1296#issuecomment-460837194

Background: Version 15.0 TKLDev running in an LXC v3 (LXD) container. Ran tkldev-setup to get the latest code from the git repositories. Failure: Whenever I run make or make root.patched on LAMP or any appliance based on LAMP, I get the following error.

fab-chroot build/root.patched --script /turnkey/fab/common/conf/mysql
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  216k  100  216k    0     0   266k      0 --:--:-- --:--:-- --:--:--  265k
/
[....] Starting MariaDB database server: mysqldinstall: invalid user 'mysql'
/usr/share/fab/product.mk:535: recipe for target 'build/stamps/root.patched' failed
make: *** [build/stamps/root.patched] Error 1

When I check root.patched I find that the mysql user has not been defined. I think it may be related to the issue mentioned in LAMP's changelog.

turnkey-lamp-15.1 (1) turnkey; urgency=low

  * Rebuild to resolve inadvertant removal of mariadb during sec-updates
    - part of #1246.

 -- Jeremy Davis <jeremy@turnkeylinux.org>  Wed, 21 Nov 2018 18:29:42 +1100

Apparently the issue has something to do with the way I created the v15.0 TKLDev container. I was able to create a new container based on the v15.1 TKLDev Proxmox image. Once I got that running properly I'm able to build the LAMP appliance without getting the mysql error. Along the way, I discovered that while running tkldev-setup will update from the git repositories, it will not update the bootstraps if they are already in place. I was using a set of stretch bootstraps from Oct 11 2017. TODO: Feature request to have tkldev-setup check the date of the bootstrap files and if newer files are available, then backup the old files and directory, before downloading and installing newer bootstraps.

Hi @Dude4Linux - I'm inclined to re-purpose and re-open this issue as a feature request to have tkldev-setup to check for a newer version of the bootstrap.

However, I have a few further thoughts:

As all packages should be updated during the build process, having an outdated bootstrap should be immaterial. The exception is building offline (with cached packages, i.e. using Chanko and Pool with FAB_POOL=y set). But under that scenario, you likely wouldn't have internet access anyway...
If the downloaded bootstrap file (e.g. bootstrap-stretch-amd64.tar.gz) has been removed, getting a date of the current bootstrap may be unreliable. As the default is to retain the downloaded bootstrap, re-downloading if that doesn't exist, or there is a newer one available would be reasonably sensible IMO.
It's possibly of limited value for most users as historically, we have only ever provided a single bootstrap per major TurnKey/Debian release.

Regarding your old bootstrap, I can only assume that the Stretch bootstrap you had was pre v15.0 (i.e. while we were testing), as the only "official" bootstrap is dated 2018-04-09 (see on the mirror).

FWIW though, during the build process, all packages should be updated. So unless you were also building offline (i.e. setting FAB_POOL=y and using Pool & Chanko) then it should be pulling the latest packages from the Debian apt repo. So if that wasn't the case for you, I doubt that it was that issue.

Other possibilities that come to mind:

Stale package stored by polipo - which should have been auto refreshed during the build (i.e. some sort of failure within the build process).
Perhaps you already had it previously built to root.build prior? I recommend always running make clean before building, unless you are currently working on something.
An issue with the build process that you are using/have devised. IIRC you have updated your TKLDev to build straight to LXC right? (I.e. not build ISO first as buildtasks does).

PS Apologies that I still haven't done anything with the work you did late last year. I really need to circle back to that so we can include the changes and rebuild LXC. I've just been pretty snowed under lately and the squeaky wheels get oiled first...

On 2/5/19 3:55 PM, Jeremy Davis wrote:

Hi @Dude4Linux https://github.com/Dude4Linux - I'm inclined to re-purpose and re-open this issue as a feature request to have tkldev-setup to check for a newer version of the bootstrap.

However, I have a few further thoughts:

As all packages should be updated during the build process, having an outdated bootstrap should be immaterial. The exception is building offline (with cached packages, i.e. using Chanko and Pool with |FAB_POOL=y| set). But under that scenario, you likely wouldn't have internet access anyway...

If the downloaded bootstrap file (e.g. |bootstrap-stretch-amd64.tar.gz|) has been removed, getting a date of the current bootstrap may be unreliable. As the default is to retain the downloaded bootstrap, re-downloading if that doesn't exist, or there is a newer one available would be reasonably sensible IMO.

It's possibly of limited value for most users as historically, we have only ever provided a single bootstrap per major TurnKey/Debian release.

Regarding your old bootstrap, I can only assume that the Stretch bootstrap you had was pre v15.0 (i.e. while we were testing), as the only "official" bootstrap is dated |2018-04-09| (see on the mirror http://mirror.turnkeylinux.org/turnkeylinux/images/bootstrap/).

You are correct. It was the testing bootstrap that I downloaded from your development site, replacing the custom bootstrap that I had created.

The problems I was experiencing come from the 'unusual' development setup that I have created. While travelling, I'm using LXC/LXD version 3.0.3 on Ubuntu 16.04. I've been afraid to upgrade to 18.04 for fear of breaking my system.

In this setup, I have three containers running TKLDev. The first was created from the version 14.2 Proxmox image and it appears to be working okay. Because the v15.0 TKLDev appliance wasn't released (for Proxmox) when I wanted to do development on Stretch, I created my own bastardized appliance by starting with installing the v14.2 appliance and then upgrading it to Stretch. I thought I had caught most of the changes that were needed, but I must have missed something. I've since found that tkldev-setup was the older version from Jessie.

My third container is now running TKLDev v15.1 created from the Proxmox image of that version. I had problems getting this to work because the version of LXC/LXD provided by xenial-backports has changed from v2.1 to 3.0.0, 3.0.1, 3.0.2 to the current 3.0.3. I had to update lxc-turnkey and turnkey.common.conf to create v3.x containers. This won't be needed until the Buster release of the LXC appliance, but the good news is that the work is already done. I also had to rewrite the lxc-to-lxd utility to handle the new container configuration. To avoid confusion, I changed the name to lxc-convert.

FWIW though, during the build process, all packages should be updated. So unless you were also building offline (i.e. setting |FAB_POOL=y| and using Pool & Chanko) then it should be pulling the latest packages from the Debian apt repo. So if that wasn't the case for you, I doubt that it was that issue.

Other possibilities that come to mind:

Stale package stored by polipo - which should have been auto refreshed during the build (i.e. some sort of failure within the build process).

Perhaps you already had it previously built to |root.build| prior? I recommend always running |make clean| before building, unless you are currently working on something.

I always start with make clean then run make root.patched (as you advised previously). Next I run lxd-make-image which use deck to mount another layer mounted on build/rootfs. From there, it makes all the adjustments needed for an LXC v3 container and directly creates the LXD image. That's how I found the bug in deck that fails to confirm that build/rootfs is empty. I use this last method when I'm developing a new appliance that doesn't have a Proxmox image. Always nice to have options.

An issue with the build process that you are using/have devised. IIRC you have updated your TKLDev to build straight to LXC right? (I.e. not build ISO first as buildtasks does).

PS Apologies that I still haven't done anything with the work you did late last year. I really need to circle back to that so we can include the changes and rebuild LXC. I've just been pretty snowed under lately and the squeaky wheels get oiled first...

No apology necessary. I do have one suggestion related to the Buster release. If you could prioritize updating the development tools first, it would allow other developers to jump in and start testing sooner and perhaps cut down on your workload. The minimum requirements to run TKLDev, with new bootstraps and core would be very helpful.

Cheers, John

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/turnkeylinux/tracker/issues/1296#issuecomment-460837194, or mute the thread https://github.com/notifications/unsubscribe-auth/ABKbBvf0pWQvoW09ySeMnuGzTdiWd50_ks5vKgvZgaJpZM4ahoNV.

@JedMeister - After realizing that my 'home brew' TKLDev appliance had an older version of deck and tkldev-setup, I started to wonder why. After checking to make sure my APT sources were correct, I ran apt list --upgradable and found that a number of TurnKey related packages were out of date. I had simply neglected to run apt-get upgrade recently.

# apt list --upgradable
Listing... Done
autoversion/stretch 0.9.2+32+g1da3404 amd64 [upgradable from: 0.9.2+31+gadbc353]
base-files/stable 9.9+deb9u7 amd64 [upgradable from: 9.9+deb9u6]
chanko/stretch 0.6+108+g6329935 amd64 [upgradable from: 0.6+107+g0204db0]
confconsole/stretch 1.1.0+2+g6c2aad9 all [upgradable from: 1.0.1+24+g3c8fe0b]
curl/stable 7.52.1-5+deb9u9 amd64 [upgradable from: 7.52.1-5+deb9u8]
deck/stretch 2.0rc+6+g4929242 amd64 [upgradable from: 1.0+18+g451b87c]
deckdebuild/stretch 1.0+15+g3dbe071 amd64 [upgradable from: 1.0+13+g1f228b3]
di-live/stretch 0.9.5+32+g552c168 amd64 [upgradable from: 0.9.5+30+g3fa70dc]
fab/stretch 0.6+257+g26f91ac amd64 [upgradable from: 0.6+257+g26f91ac]
hubdns/stretch 1.0+19+gdea5291 amd64 [upgradable from: 1.0+18+g2c888e5]
inithooks/stretch 0.9+270+g179b423 all [upgradable from: 0.9+234+g7503665]
libcurl3/stable 7.52.1-5+deb9u9 amd64 [upgradable from: 7.52.1-5+deb9u8]
libcurl3-gnutls/stable 7.52.1-5+deb9u9 amd64 [upgradable from: 7.52.1-5+deb9u8]
pool/stretch 1.0.1+29+g9a6f718 amd64 [upgradable from: 1.0.1+29+g9a6f718]
pycurl-wrapper/stretch 1.2+9+g1fe96cf amd64 [upgradable from: 1.2+8+gf4d2e96]
pyproject-common/stretch 1.1+36+gf50fff4 amd64 [upgradable from: 1.1+32+g2e0205d]
python-dialog/stretch 2.7-1turnkey+20+g037798a amd64 [upgradable from: 2.7-1turnkey+19+g0ef57e7]
repo/stretch 1.0+16+gceaa948 amd64 [upgradable from: 1.0+15+ge3a56ec]
tklbam/stretch 1.4.1+37+g8117cd6 all [upgradable from: 1.4.1+31+gfece3d0]
tklbam-duplicity/stretch 0.6.18-2turnkey+28+g6495c80 amd64 [upgradable from: 0.6.18-2turnkey+25+ga1b40d4]
tklbam-python-boto/stretch 2.3.0-2turnkey+13+g29b88e1 all [upgradable from: 2.3.0-2turnkey+12+ga891657]
tklbam-squid/stretch 2.7.STABLE9-2.1turnkey+48+g37dea7b amd64 [upgradable from: 2.7.STABLE9-2.1turnkey+47+ga7982fc]
tklbam-squid-common/stretch 2.7.STABLE9-2.1turnkey+48+g37dea7b amd64 [upgradable from: 2.7.STABLE9-2.1turnkey+47+ga7982fc]
turnkey-pylib/stretch 0.5+45+gc201a49 amd64 [upgradable from: 0.5+44+g24b8281]
turnkey-sysinfo/stretch 0+2017.8.16+00.00.28+a8223b3e amd64 [upgradable from: 0+2017.7.24+16.38.10+4d3c6f63]
turnkey-version/stretch 0+2017.8.15+23.56.46+6c2decf6 amd64 [upgradable from: 0+2017.7.24+16.38.10+e6ad7143]

After running the upgrade and installing all the latest software, my 'home brew' appliance appears to work as well as the new one that I derived from the v15.1 Proxmox image.

My thoughts on the future of tkldev-setup

As you pointed out, it's unlikely to need to upgrade the bootstraps, but if in the future it is required, tkldev-setup should handle the situation gracefully. -- check for new version by downloading the signature file (.hash) and compare to the installed signature -- backup the installed bootstrap by renaming files and directory -- download and install the new bootstraps -- verify that the device files have been created properly (I've had this fail in a non-privileged container)
Continue to install/update components from the git repositories
Check a list of required apt packages and ensure they are installed and up-to-date
Respect idempotency. Only preform actions which are necessary

You should be able to install a core appliance, then run tkldev-setup and have a working TKLDev appliance.

@Dude4Linux

Hey John,

Thanks for the additional info. Your suggestions re tkldev-setup sound pretty reasonable to me!

This next bit is OT, but it does fit with your previous post and I wanted to share:

I do have one suggestion related to the Buster release. If you could prioritize updating the development tools first, it would allow other developers to jump in and start testing sooner and perhaps cut down on your workload. The minimum requirements to run TKLDev, with new bootstraps and core would be very helpful.

We're already working on Buster :smile:

The aim is to be able to provide a v16.0rc1 Core while Buster is RC (expected sometime probably late Q2). Depending on how things go, we may even do a v16.0beta before that? And/or an rc2 if it seems appropriate. I also plan to release at least one v16.0rc TKLDev, and then use that to build the v16.0 stable TKLDev (unlike v15.0, where we build that from v14.2).

We're also trying to update (at least some) of the tools that are written in Python2; to Python3. Ideally we hope to release v16.0 stable (at least of Core and TKLDev, possibly LAMP and others too?!) within days of Buster going stable.

See how we go...

Moved comment to new Issue (interested in this one though)

I am running into this myself. I just installed a LCX TKLDEV 15.0 and have had quite different errors than ISO built TKLDEV 15.0 RC1. Likely need to do a non RC1 ISO build (LCX install was sure easy on Promox).

Just playing on PROMOX in our Makerspace and found I could pull TKL LCX images. Did not see Promox ones so I will fish around looking at different ways I can build a system.

My understanding is NOT to use the LCX version of Tkldev 15.0. Is that correct?

turnkeylinux / tracker

TKLDev should check for new bootstrap (and other bits) #1296