Updating timezone data

[mw9]

Further to earlier comments, I think I have found a clean solution to updating SqueezeOS timezone data.

It is really a very early draft pull request, so I've presented it in patch form. I hope this format works.

As you will, there remains an amount of validation to do.

From ee30d383df0d58da21c38c0a66756566945f1c53 Mon Sep 17 00:00:00 2001
From: Martin Williams <martinr.williams@gmail.com>
Date: Wed, 18 May 2022 14:17:21 +0100
Subject: [PATCH] SqueezeOS - Update timeone data - Preliminary draft

---
 poky/build/conf/local.conf.sample             |  7 ++-
 .../poky-external-csl2010q1-modified.inc      | 21 +++++++
 ...ernal-csl-toolchain_2010q1-202-modified.bb | 63 +++++++++++++++++++
 3 files changed, 90 insertions(+), 1 deletion(-)
 create mode 100644 poky/meta-squeezeos/conf/distro/include/poky-external-csl2010q1-modified.inc
 create mode 100644 poky/meta-squeezeos/packages/meta/external-csl-toolchain_2010q1-202-modified.bb

diff --git a/poky/build/conf/local.conf.sample b/poky/build/conf/local.conf.sample
index 42b775b05..20070524f 100644
--- a/poky/build/conf/local.conf.sample
+++ b/poky/build/conf/local.conf.sample
@@ -118,9 +118,14 @@ PACKAGE_CLASSES ?= "package_ipk"
 # Use "oabi" for machines with kernels < 2.6.18 on ARM for example.
 # Use "external-MODE" to use the precompiled external toolchains where MODE
 # is the type of external toolchain to use e.g. eabi.
+
+# "external-csl2010q1-modified" is a modified version of the Code Sourcery
+# csl2010q1 tool chain used by the Community firmware project.
+
 #POKYMODE = "external-csl2009q1"
 #POKYMODE = "external-csl2009q3"
-POKYMODE = "external-csl2010q1"
+#POKYMODE = "external-csl2010q1"
+POKYMODE = "external-csl2010q1-modified"
 #POKYMODE = "external-csl2010.09"

 # Uncomment this to specify where BitBake should create its temporary files.
diff --git a/poky/meta-squeezeos/conf/distro/include/poky-external-csl2010q1-modified.inc b/poky/meta-squeezeos/conf/distro/include/poky-external-csl2010q1-modified.inc
new file mode 100644
index 000000000..0627a7d71
--- /dev/null
+++ b/poky/meta-squeezeos/conf/distro/include/poky-external-csl2010q1-modified.inc
@@ -0,0 +1,21 @@
+#
+# Poky configuration to use our modified external CSL 2010q1 toolchain (ARM EABI)
+#
+
+TARGET_VENDOR = "-none"
+
+PREFERRED_PROVIDER_linux-libc-headers = "external-csl-toolchain"
+PREFERRED_PROVIDER_virtual/arm-none-linux-gnueabi-gcc = "external-csl-toolchain"
+PREFERRED_PROVIDER_virtual/arm-none-linux-gnueabi-gcc-initial = "external-csl-toolchain"
+PREFERRED_PROVIDER_virtual/arm-none-linux-gnueabi-gcc-intermediate = "external-csl-toolchain"
+PREFERRED_PROVIDER_virtual/arm-none-linux-gnueabi-binutils = "external-csl-toolchain"
+PREFERRED_PROVIDER_virtual/arm-none-linux-gnueabi-libc-for-gcc = "external-csl-toolchain"
+PREFERRED_PROVIDER_virtual/libc = "external-csl-toolchain"
+PREFERRED_PROVIDER_virtual/libintl = "external-csl-toolchain"
+PREFERRED_PROVIDER_virtual/libiconv = "external-csl-toolchain"
+PREFERRED_PROVIDER_glibc-thread-db = "external-csl-toolchain"
+PREFERRED_PROVIDER_virtual/linux-libc-headers = "external-csl-toolchain"
+
+PREFERRED_VERSION_external-csl-toolchain = "2010q1-202-modified"
+
+TOOLCHAIN_OPTIONS = " --sysroot=${STAGING_DIR_HOST}"
diff --git a/poky/meta-squeezeos/packages/meta/external-csl-toolchain_2010q1-202-modified.bb b/poky/meta-squeezeos/packages/meta/external-csl-toolchain_2010q1-202-modified.bb
new file mode 100644
index 000000000..769c720db
--- /dev/null
+++ b/poky/meta-squeezeos/packages/meta/external-csl-toolchain_2010q1-202-modified.bb
@@ -0,0 +1,63 @@
+# This is a modified version of the external tool chain package
+#  'external-csl-toolchain_2010q1-202.bb'.
+
+# It eliminates the pre-compiled timezone data otherwise installed by the
+# tool chain into '/usr/share/zoneinfo', so that it can be provided by
+# package 'tzdata' instead.
+
+# It also defines a run time dependency on 'tzdata', so that
+# '/usr/share/zoneinfo' will, indeed, be populated.
+
+
+# Base package.
+
+require external-csl-toolchain_2010q1-202.bb
+
+# Override SRC_URI, because the base package incorporates a parameterized
+# version number ${PV}, which no longer matches up.
+
+SRC_URI = "https://sourcery.sw.siemens.com/public/gnu_toolchain/arm-none-linux-gnueabi/arm-2010q1-202-arm-none-linux-gnueabi-i686-pc-linux-gnu.tar.bz2 \
+        file://SUPPORTED"
+
+PR = "r0"
+
+# Remove timezone data installed by base package.
+# We leave in place the initial symbolic link /etc/localtime -> /usr/share/zonedata/Factory
+
+do_install_append() {
+   rm -r  ${D}/usr/share/zoneinfo
+}
+
+# Add run time dependency on 'tzdata' to populate '/usr/share/zoneinfo'.
+# 'tzdata-misc' contains the 'Factory' time zone, 
+
+RDEPENDS = "tzdata tzdata-misc"
+
+# Always clean our "automatic" dependency 'tzdata' whenever we clean this package.
+# (Can help avoid confusion if switching tool chain.)
+
+do_clean[depends] = "tzdata:do_clean"
+
+
+### TO DO:
+
+# Check that all required timezones are installed.
+# Verify against SqueezePlay, LMS, existingg SqueezeOS, etc.
+# E.g. datetime.lua, SetupTZApplet.lua, ...
+#RDEPENDS = "tzdata tzdata-misc tzdata-posix tzdata-right tzdata-africa \
+#    tzdata-americas tzdata-antarctica tzdata-arctic tzdata-asia \
+#    tzdata-atlantic tzdata-australia tzdata-europe tzdata-pacific"
+# How does SqueezeOS/LMS use this stuff ?
+
+# Is the generated zoneinfo database compatible with our old OS ?
+
+# Do we need posixrules -> America/New_York ? None of the tzdata packages provide it.
+# Could modifiy tzdata zic command, or perhaps add a symlink in our install.
+
+# Do we need:
+#  iso3166.tab
+#  zone.tab
+
+# Handling of /etc/localtime -> /etc/Factory
+
+
-- 
2.25.0

[ralph-irving]

Thanks for putting together the patch.

I will rebuild my qemu environment with it to try and answer some of the TODO items.

I haven't had a chance to track down my tzdata package issue yet. For now, I renamed the problem source tree and checked out a fresh copy from git and just copied over the entire sources folder.

The build I just completed with the patch has New_York is in the staging rootfs and the cramfs image.

$ file ~/source/squeezeos/poky/build/tmp-baby/rootfs/usr/share/zoneinfo/America/New_York
New_York: timezone data, version 2, no gmt time flags, no std time flags, no leap seconds, no transition times, 1 abbreviation char

$ sudo cramfsck -x x squeezeos-image-baby.cramfs
$ file x/usr/share/zoneinfo/America/New_York 
x/usr/share/zoneinfo/America/New_York: timezone data, version 2, no gmt time flags, no std time flags, no leap seconds, no transition times, 1 abbreviation char

Good question about the .tab files. I had not noticed them in /usr/share/zoneinfo before.

[ralph-irving]

Initial results would suggest that busybox does not support the new tzdata. The timezone abbreviation is completely missing.

# uname -a
Linux SqueezeboxRadio 2.6.39.4-versatile #4 PREEMPT Sun Mar 21 12:36:26 EDT 2021 armv5tejl GNU/Linux
# date
Sat May 21 15:18:24  2022

vs radio

# uname -a
Linux radio 2.6.26.8-rt16 #1 PREEMPT RT Thu Aug 19 11:11:04 EDT 2021 armv5tejl GNU/Linux
# date
Sat May 21 11:16:53 EDT 2022

[mw9]

Initial results would suggest that busybox does not support the new tzdata. The timezone abbreviation is completely missing.

I have just uploaded my own generated timezone data, and get the same result. So a little research required on that, I think. The documentation talks about 'version 2' and 'version 3', and probably being backwards compatible, but with an element of occasional doubt about it.

As regards other matters:

tzdata and tzdata-misc do not include enough timezones.

At present I am including all timezone packages except for tzdata-posix andtzdata-right.

+# Provides a few basic cities.
+RDEPENDS += "tzdata"
+# 'tzdata-misc' contains the deffault 'Factory' time zone.
+RDEPENDS += "tzdata-misc"
+# SqueezePlay uses more timezones than are provided by the basic
+# 'tzdata' package.
+RDEPENDS += "tzdata-africa tzdata-americas tzdata-antarctica \
+    tzdata-arctic tzdata-asia tzdata-atlantic tzdata-australia \
+    tzdata-europe tzdata-pacific"

Refer SetUpTZApplet.luafor some insight: https://github.com/ralph-irving/squeezeos-squeezeplay/blob/public/8.0/src/squeezeplay_squeezeos/share/applets/SetupTZ/SetupTZApplet.lua#L20

The size of the generated zoneinfo is a little less that the existing install. If tzdata-posix andtzdata-right are included, it would take it over by quite a lot, I think. I must check.

It would be very straightforward to replace the tzdata.bb package with our own, say tzdata_for_SQP.bb. That could give us exactly what we would want, including posixrules, iso3166.tab, and zone.tab. I don't know that we particularly need these files, but can we know ?

I do think this is something that will want to be done. There is regular talk of abandoning summer time in Europe, or Some Parts Of Europe, and I guess the same in North America.

[mw9]

Initial results would suggest that busybox does not support the new tzdata. The timezone abbreviation is completely missing.

A -b fat argument to zic seems to overcome it. Also worth looking at will be the -r [@lo][/@hi] argument.

I don't understand at the moment, but there seem to be 32 bit/64 bit issues in play.

[mw9]

I've made a customized 'tzdata', to be installed as poky/meta-squeezeos/packages/tzdata/tzdata-squeezeos_2021e.bb. The modified toolchain then changes to simply depend on this. Removes a certain amount of crud.

The built tzdata seems to be usable. Incidentally, I had the the same issue when using it on my Ubuntu 10.04.4 build system. So not just an issue with busybox date.

DESCRIPTION = "Timezone data for SqueezeOS"
SECTION = "base"
PRIORITY = "optional"
DEPENDS = "tzcode-native"

# This recipe is a customized version of 'tzdata', which builds the time
# zone database '/usr/share/zoneinfo'. It replaces the time zone database
# provided by the CSL tool chain.

# It is intended to be automatically invoked by recipe
#  'external-csl-toolchain_2010q1-202-modified.bb'

# The recipe 'tzdata' cannot be used 'as is' for this purpose, as it
# does not compile suitable databse files for this older 32 bit OS.

PR = "r0"

# Taken from the 'tzdata' recipe - presumably older packages.
RCONFLICTS= "timezones timezone-africa timezone-america timezone-antarctica \
             timezone-arctic timezone-asia timezone-atlantic \
             timezone-australia timezone-europe timezone-indian \
             timezone-iso3166.tab timezone-pacific timezone-zone.tab"

# This package will replace any packages provided by the 'tzdata' recipe.
RCONFLICTS += "tzdata tzdata-africa tzdata-americas tzdata-antarctica \
               tzdata-arctic tzdata-asia tzdata-atlantic \
               tzdata-australia tzdata-europe tzdata-misc \
               tzdata-pacific tzdata-posix tzdata-right"

SRC_URI = "https://data.iana.org/time-zones/releases/tzdata${PV}.tar.gz"

S = "${WORKDIR}"

# time zone data files provided by the source package
TZONES= "africa antarctica asia australasia europe northamerica southamerica  \
         factory etcetera backward \
        "

do_compile () {
        # create the timezone (tzif) files
        for zone in ${TZONES}; do \
            # Need the '-b fat' option to generate tzif files compatible with older 32 bit systems.
            ${STAGING_BINDIR_NATIVE}/zic -d ${S}${datadir}/zoneinfo -b fat -L /dev/null \
                ${S}/${zone} ; \
            ${STAGING_BINDIR_NATIVE}/zic -d ${S}${datadir}/zoneinfo/posix -b fat -L /dev/null \
                ${S}/${zone} ; \
            ${STAGING_BINDIR_NATIVE}/zic -d ${S}${datadir}/zoneinfo/right -b fat -L ${S}/leapseconds \
                ${S}/${zone} ; \
        done

    # generate a posixrules file
    ln -s America/New_York ${S}${datadir}/zoneinfo/posixrules
}

do_install () {
        # make target directories available
        install -d -m 0755 ${D}${datadir}/zoneinfo
        install -d -m 0755 ${D}${sysconfdir}

        # copy over generated zoneinfo directory - preserving permissions
        cp -pPR ${S}${datadir}/zoneinfo ${D}${datadir}

        # supplementary files
        install -m 0644 ${S}/iso3166.tab  ${D}${datadir}/zoneinfo
        install -m 0644 ${S}/zone.tab     ${D}${datadir}/zoneinfo
        install -m 0644 ${S}/zone1970.tab ${D}${datadir}/zoneinfo

        # create /etc/localtime link
        ln -s ../usr/share/zoneinfo/Factory ${D}/${sysconfdir}/localtime
}

FILES_${PN} = "${datadir}/zoneinfo ${sysconfdir}"

### TO DO

## Can we depend on a particular version of tzcode-native ?
##   This recipe was based on v 2021e

## Can we reduce the considerable growth in file size ?
##   Yes:
##   By excluding 'posix' and 'right' timezone data, if we don't need it
##   Perhaps by identifying and picking off exotics

## SetUpTZApplet.lua barely uses any of the timzones.
##   Should it be extended to bring up to date to modern needs, i.e. new time zones ?
##   We could then use the revised list to winnow down what we install.

## Check that all required timezones are installed.
## Verify against SqueezePlay, LMS, existingg SqueezeOS, etc.
## E.g. datetime.lua, SetupTZApplet.lua, ...
## How does SqueezeOS/LMS use this stuff ?

## Is the generated zoneinfo database compatible with our old OS ?

## Do we need posixrules -> America/New_York ? None of the tzdata packages provide it.

## Do we need:
#  iso3166.tab
#  zone.tab
#  zone1970.tab

[ralph-irving]

Yes, "they" are trying to keep eastern daylight savings time year round here. It's been in the works for several years, but trying to get the various states and provinces to agree is proving to be a challenge.

I haven't found any references to the three .tab files in the firmware images I checked.

I like your idea to add more timezones to the lua applet and only add the zoneinfo files to match. But how to determine which ones?

I tried to build and test the latest change proposal today but have a conflict with tzdata-misc. Unfortunately, I won't be able to get back to it until later this week.

remove_packaging_data_files
rm -rf /home/ralphy/source/squeezeos/poky/build/tmp-baby/rootfs/usr/lib/opkg/
rm -f '/home/ralphy/source/squeezeos/poky/build/tmp-baby/rootfs/usr/lib/opkg/lists/*'
log_check rootfs
set +x
log_check: Using /home/ralphy/source/squeezeos/poky/build/tmp-baby/work/baby-none-linux-gnueabi/squeezeos-image-1.0-r1/temp/log.do_rootfs.27229 as logfile
log_check: There were error messages in the logfile
log_check: Matched keyword: [ERR]
log_check: Using /home/ralphy/source/squeezeos/poky/build/tmp-baby/work/baby-none-linux-gnueabi/squeezeos-image-1.0-r1/temp/log.do_rootfs.27229 as logfile
log_check: There were error messages in the logfile
log_check: Matched keyword: [ERR]
(offline root mode: not running udev.postinst)
Configuring update-rc.d
Configuring watchdog
Configuring wireless-tools
Configuring wpa-supplicant
Collected errors:
 * ERROR: The following packages conflict with tzdata-squeezeos:
         *  tzdata-misc *
+ '[' '!' -z '' ']'
+ export D=/home/ralphy/source/squeezeos/poky/build/tmp-baby/rootfs
+ D=/home/ralphy/source/squeezeos/poky/build/tmp-baby/rootfs

[mw9]

I like your idea to add more timezones to the lua applet and only add the zoneinfo files to match. But how to determine which ones?

Good question ! I shall plan on a little research over the next couple of weeks or so to see what might be sensible to add to the lua applet.

Perhaps just removing posix and right from the zoneinfo files might be perfectly acceptable. It would be enough to get the image size back down to where it was.

I tried to build and test the latest change proposal today but have a conflict with tzdata-misc.

\<snip>

 * ERROR: The following packages conflict with tzdata-squeezeos:
         *  tzdata-misc *

Excellent ! So that worked as intended.

Have you tried bitbake -c clean tzdata ? Always worked for me...

[mw9]

I've raised a PR, https://github.com/ralph-irving/squeezeos/pull/17, to provide a concrete implementation of what has been discussed here.

[ralph-irving]

Thanks for doing this. I never managed to get the above changes to build, but was successful using those in PR,#17.

[mw9]

Issue resolved by PR Update timezone data #17.