"Cannot determine time zone" on FreeBSD

[autarch]

Migrated from rt.cpan.org #55029 (status was 'open')

Requestors:

• mschwern@cpan.org (@schwern)

Attachments:

• localtime
• 0001-Parse-etc-localtime-by-DateTime-TimeZone-Tzfile.patch

From mschwern@cpan.org (@schwern) on 2010-02-26 19:39:44:

$ perl -wle 'use DateTime; DateTime->now( time_zone => "local" )' Cannot determine local time zone

$ uname -a FreeBSD eschaton.local 7.2-STABLE FreeBSD 7.2-STABLE #0: Sat Nov 14 22:23:32 EST 2009 root@eschaton.local:/usr/obj/usr/src/sys/ESCHATON amd64

I'm also seeing it on FreeBSD 8. http://www.cpantesters.org/cpan/report/6868995

Digging into the problem, the only DateTime::TimeZone::Local::Unix method which is having any luck is /etc/localtime, the rest of the files checked do not exist, but there are no files of the same size in /usr/share/zoneinfo.

Attached is a copy of /etc/localtime from that system.

This is with 5.10.1 and DateTime::TimeZone 1.10.

[autarch]

From mschwern@cpan.org (@schwern) on 2010-02-26 19:59:45:

A further tidbit of information, /etc/localtime is not a symlink and it is a TZif file while all the applicable files in /usr/share/zoneinfo are TZif2.

One possibility is that /etc/localtime was updated as part of an update for the 2007 DST changes and the zoneinfo files were changed under it. http://www.freebsd.org/releng/dst_info.html

One possible work around is to use zdump and scrape the output. $ zdump /etc/localtime /etc/localtime Fri Feb 26 14:56:30 2010 EST

[autarch]

From autarch@urth.org (@autarch) on 2010-02-27 16:48:40:

On Fri, 26 Feb 2010, Michael G Schwern via RT wrote:

Digging into the problem, the only DateTime::TimeZone::Local::Unix method which is having any luck is /etc/localtime, the rest of the files checked do not exist, but there are no files of the same size in /usr/share/zoneinfo.

This could, in theory, be fixed by just using the binary Olson file directly, as we discussed on the list.

But in this case, it seems like your system has a problem. If there's no match in /usr/share/zoneinfo, the most likely case is that the file copied to /etc/localtime is now out of date and wrong, so actually using it will just produce wrong answers, instead of dying. I'm not sure that's an improvement.

You suggested using zdump, which I suppose could work, assuming that the combination of a zone's short name and gmt offset for a given date is unique. However, that'd require a whole bunch of new code to look this up given the list of all possible zones. This really doesn't seem worth it, when the simple fix is to symlink /etc/localtime instead of copying a file to it.

-dave

/============================================================ http://VegGuide.org http://blog.urth.org Your guide to all that's veg House Absolute(ly Pointless) ============================================================/

[autarch]

From schwern@pobox.com on 2010-02-27 21:53:13:

autarch@urth.org via RT wrote:

Digging into the problem, the only DateTime::TimeZone::Local::Unix method which is having any luck is /etc/localtime, the rest of the files checked do not exist, but there are no files of the same size in /usr/share/zoneinfo.

This could, in theory, be fixed by just using the binary Olson file directly, as we discussed on the list.

But in this case, it seems like your system has a problem. If there's no match in /usr/share/zoneinfo, the most likely case is that the file copied to /etc/localtime is now out of date and wrong, so actually using it will just produce wrong answers, instead of dying. I'm not sure that's an improvement.

You suggested using zdump, which I suppose could work, assuming that the combination of a zone's short name and gmt offset for a given date is unique. However, that'd require a whole bunch of new code to look this up given the list of all possible zones. This really doesn't seem worth it, when the simple fix is to symlink /etc/localtime instead of copying a file to it.

So far this has shown up on two independent machines, neither are mine. A FreeBSD 7.2 box provided to me by apeiron and szrezic's FreeBSD 8.0 smoke tests. If they've simply both misconfigured their machines then I'll get them to fix it. If its a systematic FreeBSD problem then as much as it sucks, working around it would be necessary.

I've contacted szrezic and apeiron to see if they can shed any light.

1. If the thought of something makes me giggle for longer than 15 seconds, I am to assume that I am not allowed to do it. -- The 213 Things Skippy Is No Longer Allowed To Do In The U.S. Army http://skippyslist.com/list/

[autarch]

From mschwern@cpan.org (@schwern) on 2010-02-27 22:47:03:

Here's what I've figured out from talking with apeiron. Its likely policy for FreeBSD and other distributions to copy zone files rather than symlink. The logic appears to be that they want the time to be correct even if /usr isn't mounted. Here's Gentoo discussing it. http://bugs.gentoo.org/110038 That discussion also indicates this is a more widespread problem than just FreeBSD.

The problem occurs like so: 1) tzsetup is run and a file from /usr/share/zoneinfo is copied to /etc/localtime. 2) The operating system is updated and new zoneinfo files are installed. 3) tzsetup is not as part of the update 4) /etc/localtime now represents no installed zoneinfo file.

Failure to re-run tzsetup is the fault of the operating system distribution, not the sysadmin. It should be handled as part of the updated mechanism and it is not. Which is to say, a lot of systems will have this problem. Since its a problem endemic to an entire major distribution, it should be worked around.

A simple fallback mechanism would be to use POSIX::strftime() to get the short form time zone name and GMT offset.

$ perl -wle 'use POSIX; print strftime("%Z %z", localtime)' EST -0500

This is enough to identify the time zone. If there's a way to get all zones which match a given short form zone then DateTime can simply pick the first one that matches the gmtoff. For modern times it shouldn't matter if you're in America/Seattle or America/Los_Angeles.

Or DT simply reads and trusts /etc/localtime which would seem to be the best long term fix.

[autarch]

From srezic@cpan.org (@eserte) on 2010-02-28 10:11:09:

On Sat Feb 27 17:47:03 2010, MSCHWERN wrote:

Here's what I've figured out from talking with apeiron. Its likely policy for FreeBSD and other distributions to copy zone files rather than symlink. The logic appears to be that they want the time to be correct even if /usr isn't mounted. Here's Gentoo discussing it. http://bugs.gentoo.org/110038 That discussion also indicates this is a more widespread problem than just FreeBSD.

The problem occurs like so: 1) tzsetup is run and a file from /usr/share/zoneinfo is copied to /etc/localtime. 2) The operating system is updated and new zoneinfo files are installed. 3) tzsetup is not as part of the update 4) /etc/localtime now represents no installed zoneinfo file.

Failure to re-run tzsetup is the fault of the operating system distribution, not the sysadmin. It should be handled as part of the updated mechanism and it is not. Which is to say, a lot of systems will have this problem. Since its a problem endemic to an entire major distribution, it should be worked around.

A simple fallback mechanism would be to use POSIX::strftime() to get the short form time zone name and GMT offset.

$ perl -wle 'use POSIX; print strftime("%Z %z", localtime)' EST -0500

This is enough to identify the time zone. If there's a way to get all zones which match a given short form zone then DateTime can simply pick the first one that matches the gmtoff. For modern times it shouldn't matter if you're in America/Seattle or America/Los_Angeles.

Or DT simply reads and trusts /etc/localtime which would seem to be the best long term fix.

Excellent analysis, and this is what happened on my system, too: I upgraded from FreeBSD 6.1 to FreeBSD 8.0, but /etc/localtime was not changed during this process.

Though there's one problem in the proposed workaround with POSIX: depending on DST being active or not one will get different results:

$ perl -wle 'use POSIX; print strftime("%Z %z", localtime)' CET +0100 $ perl -wle 'use POSIX; print strftime("%Z %z", localtime(time+86400*30))' CEST +0200

However, I think that the result of DateTime::TimeZone's heuristics was always something DST-agnostic like "Europe/Berlin".

Regards, Slaven

[autarch]

From mschwern@cpan.org (@schwern) on 2010-02-28 10:46:49:

On Sun Feb 28 05:11:09 2010, SREZIC wrote:

Though there's one problem in the proposed workaround with POSIX: depending on DST being active or not one will get different results:

$ perl -wle 'use POSIX; print strftime("%Z %z", localtime)' CET +0100 $ perl -wle 'use POSIX; print strftime("%Z %z", localtime(time+86400*30))' CEST +0200

However, I think that the result of DateTime::TimeZone's heuristics was always something DST-agnostic like "Europe/Berlin".

Yes, and the result from the heuristic would not be "CET" but would be a European time zone. It could be Europe/Berlin or Europe/Paris or any other time zone which calls itself "CET". Its not 100% accurate, the rules for one city in a time zone may be slightly different from another, but it will account for daylight savings time.

Personally I'd just trust /etc/localtime like everything else does. I've worked around it in perl5i by using DateTime::TimeZone::Tzfile to read /etc/localtime if present. Tzfile is so small the simplest solution may be to just include it in DT::TZ.

Dave?

[autarch]

From srezic@cpan.org (@eserte) on 2010-02-28 16:11:06:

On Sun Feb 28 05:11:09 2010, SREZIC wrote: [...]

$ perl -wle 'use POSIX; print strftime("%Z %z", localtime)' CET +0100 $ perl -wle 'use POSIX; print strftime("%Z %z", localtime(time+86400*30))' CEST +0200

Another observation: a list of all possible timezone names as returned by POSIX.pm may be generated with the following oneliner:

for tz in perl -nale 'next if /^#/; print $F[2]' /usr/share/zoneinfo/zone.tab |sort -u; do env TZ=$tz perl -MPOSIX -le 'print strftime("%Z", localtime)'; done|sort -u

This results in 137 timezone names on a FreeBSD machine, and 138 on a Linux machine. However, DateTime::TimeZone::Catalog knows only a small fraction of these names.

Regards, Slaven

[autarch]

From autarch@urth.org (@autarch) on 2010-02-28 20:51:50:

On Sun, 28 Feb 2010, Slaven_Rezic via RT wrote:

Another observation: a list of all possible timezone names as returned by POSIX.pm may be generated with the following oneliner:

for tz in perl -nale 'next if /^#/; print $F[2]' /usr/share/zoneinfo/zone.tab |sort -u; do env TZ=$tz perl -MPOSIX -le 'print strftime("%Z", localtime)'; done|sort -u

This results in 137 timezone names on a FreeBSD machine, and 138 on a Linux machine. However, DateTime::TimeZone::Catalog knows only a small fraction of these names.

Those are short names, which are generated based on a long name (like America/Chicago) and a specific date. For example, America/Chicago can be either CDT or CST, depending on whether daylight savings is currently in effect.

The DT::TZ catalog is generated based on the Olson database source data. This data has, for historical reasons, some short names aliased to longer names, as well as some actual zones with short names like EST or EST5EDT. It's strongly recommended that you use the Continent/City names when selecting a time zone, since these are the most likely to do what people expect.

The short names are primarily useful for output only. If you look in the datetime@perl.org list archives, you can probably find discussions about this.

-dave

/============================================================ http://VegGuide.org http://blog.urth.org Your guide to all that's veg House Absolute(ly Pointless) ============================================================/

[autarch]

From autarch@urth.org (@autarch) on 2010-03-02 18:50:49:

On Sun, 28 Feb 2010, Michael G Schwern via RT wrote:

Yes, and the result from the heuristic would not be "CET" but would be a European time zone. It could be Europe/Berlin or Europe/Paris or any other time zone which calls itself "CET". Its not 100% accurate, the rules for one city in a time zone may be slightly different from another, but it will account for daylight savings time.

Personally I'd just trust /etc/localtime like everything else does. I've worked around it in perl5i by using DateTime::TimeZone::Tzfile to read /etc/localtime if present. Tzfile is so small the simplest solution may be to just include it in DT::TZ.

I'd rather not include Tzfile. That doesn't really fix the problem, since the real problem is an out of date zone file in /etc/localtime. So including Tzfile just means we'd get wrong answers later in the process, rather than blowing up early on.

If someone wants to implement a lookup based on POSIX output, I'd probably take a patch.

-dave

/============================================================ http://VegGuide.org http://blog.urth.org Your guide to all that's veg House Absolute(ly Pointless) ============================================================/

[autarch]

From schwern@pobox.com on 2010-03-02 20:43:00:

autarch@urth.org via RT wrote:

Yes, and the result from the heuristic would not be "CET" but would be a European time zone. It could be Europe/Berlin or Europe/Paris or any other time zone which calls itself "CET". Its not 100% accurate, the rules for one city in a time zone may be slightly different from another, but it will account for daylight savings time.

Personally I'd just trust /etc/localtime like everything else does. I've worked around it in perl5i by using DateTime::TimeZone::Tzfile to read /etc/localtime if present. Tzfile is so small the simplest solution may be to just include it in DT::TZ.

I'd rather not include Tzfile. That doesn't really fix the problem, since the real problem is an out of date zone file in /etc/localtime. So including Tzfile just means we'd get wrong answers later in the process, rather than blowing up early on.

If someone wants to implement a lookup based on POSIX output, I'd probably take a patch.

We know using POSIX::strtime() is going to give us an ambiguous time zone because the short form can match many zones with the same current GMT offset but possibly different DST rules.

For example, Arizona doesn't observe DST but it'll still match MST. Here's the list of fun DST exceptions in the US alone. https://secure.wikimedia.org/wikipedia/en/wiki/Daylight_saving_time_in_the_United_States#Local_observance_of_DST

On the one hand, using POSIX::strftime output we know is never going to be quite right. We'll probably get lucky and probably the server is in a place that has normal DST rules. Probably. But "probably correct" is the same level of accuracy we have for /etc/localtime.

I'm not sure how one would efficiently map the short form time zone to a long form. You can't make it static, because a given zone changes its short form over the course of a year. America/New York can be EDT or EST depending on the date and time. You'd have to ask every time zone what its current short form is until you got a hit. This can be reduced by storing a list of all possible zones for a given short form, possibly tweaking the order to prefer normal ones over odd ones. But this is still all a complicated guess.

On the other hand, /etc/localtime might be out of date, but it might not be. A given time zone doesn't change often, and I'd gamble on /etc/localtime being correct. If its significantly out of touch the problem will show up across the whole system and the admin will fix it.

One can reverse the whole argument. Because DateTime maintains its own zone files they can just as easily fall out of date as /etc/localtime can. Can you trust the currently installed DateTime to be up to date?

DateTime getting it wrong isn't any better than /etc/localtime getting it wrong. It requires a lot more work for DateTime to get it wrong. It will be wrong in a way different from the rest of the system. I don't see this as a win.

DateTime can't solve the real problem, it is out of its control and scope. The user asked for the local time the system is configured to and that's the best that DateTime can give it.

1. Not allowed to let sock puppets take responsibility for any of my actions. -- The 213 Things Skippy Is No Longer Allowed To Do In The U.S. Army http://skippyslist.com/list/

[autarch]

From ppisar@redhat.com on 2014-09-02 16:02:15:

Dne Ne 28.úno.2010 05:46:49, MSCHWERN napsal(a):

Personally I'd just trust /etc/localtime like everything else does. I've worked around it in perl5i by using DateTime::TimeZone::Tzfile to read /etc/localtime if present.

I have the same feeling. The only problem with this approach is you cannot obtain the time zone name. I asked tzdata Fedora package maintainer to augment tzfile format to deliver the zone name, and he said that upstream had already refused any change in the format.

Tzfile is so small the simplest solution may be to just include it in DT::TZ.

Attached patch utilizes DateTime::TimeZone::Tzfile as a last resort to obtain unnamed time zone definition from /etc/localtime content.

I hit this "bug" in Fedora build system, unfortunately nobody can come with a fix on other side (tzdata package, build tools etc). You can read https://bugzilla.redhat.com/show_bug.cgi?id=1135981 and bugs referenced from there via See Also and Blocks or Depends, if you are interested in this topic.

-- Petr

[jkeenan]

I would like to add a data point to this discussion.

As part of a project to track "blead-breaks-CPAN" instances through the upcoming 5.29 development cycle, a colleague and I from ny.pm, in cooperation with New York City BSD Users Group (NYCBUG), are in the process of installing a FreeBSD-11.1 virtual machine on a FreeBSD-11.1 host provided by NYCBUG. The VM in question is Vagrant box "generic/freebsd11" (https://app.vagrantup.com/generic/boxes/freebsd11/versions/1.5.0) and the provider is virtualbox.

Last week I installed FreeBSD ports via pkg and CPAN distributions via cpan sufficient to download a perl-5.27.11 tarball, build that perl and install cpanm against it, and then use cpanm to install a list of 3000 modules derived from a "CPAN river" list. Though that process did take 8 hours, it did exit successfully -- which I expected it to, as I have been running similar process once a month on Debian Linux since November 2017.

There was one very significant difference in the results when running in this FreeBSD box, however: Dist-Zilla had test failures and failed to install. This in turn prevented the hundreds of CPAN distros in the top 3000 of the CPAN river which depend on Dist-Zilla to be skipped by cpanm. The Dist-Zilla test failures all displayed the same exception which Schwern first reported back in February 2010: Cannot determine local time zone.

I spent most of the past four days trying to diagnose this problem and develop workarounds. This problem was not related to perl-5.27.11; I could not install Dist-Zilla against the default perl (perl-5.26.2), either. While I quickly determined that the exception came from DateTime-TimeZone, for several days I thought the problem was some coding defect deep in the bowels of Dist-Zilla. It was only this morning that I began to review DateTime-TimeZone's issue queue and found this issue. I then took the following steps:

• On the host, I called Schwern's diagnostic: $ perl -wle 'use DateTime; DateTime->now( time_zone => "local" )'. It exited silently, therefore successfully.

• In the VM box, I also called that diagnostic. The result:

  perl -wle 'use DateTime; DateTime->now( time_zone => "local" )'
  Cannot determine local time zone

  So the problem was not in Dist-Zilla but in the interaction between DateTime (in this case, its dependency DateTime::TimeZone) and the operating system.

• After further study of the discussion in this ticket, I realized that the box completely lacked an /etc/localtime file. I then ran the FreeBSD tzsetup utility to establish US Eastern Time as our default time zone.

• I was then able to run cpan to install Dist::Zilla in the VM (/usr/local/lib), and then able to use cpanm to install Dist::Zilla against perl-5.27.11. This will permit me to proceed with the "test-against-dev" project using this VM.

I will submit a documentation patch which might improve matters.

Thank you very much. Jim Keenan

[autarch]

This is in fact an issue with dzil. Users of DT::TZ should not write test code using the local time zone without some extra thought first. See https://github.com/rjbs/Dist-Zilla/issues/586 for discussion.

The easy fix is to add something like local $ENV{TZ} = 'America/Chicago' to any test code that will try to determine the local tz. AFAIK every platform's local tz module checks that var first.

[jkeenan]

On 05/22/2018 01:09 PM, Dave Rolsky wrote:

This is in fact an issue with dzil. Users of DT::TZ should not write test code using the |local| time zone without some extra thought first. See rjbs/Dist-Zilla#586 https://github.com/rjbs/Dist-Zilla/issues/586 for discussion.

The easy fix is to add something like |local $ENV{TZ} = 'America/Chicago'| to any test code that will try to determine the local tz. AFAIK every platform's local tz module checks that var first.

RJBS actually mitigated one DZ test failure with that approach several years ago. And, following his lead, I (a) submitted https://github.com/rjbs/Dist-Zilla/pull/622 to fix 3 test failures I found in Dist-Zilla with the same approach; (b) began to submit patches/p.r.s to three other top-3000 distros that were also failing in this VM with the same exception.

The problem I ran into was that some CPAN authors may be reluctant to take this approach, deeming it -- not without reason -- as a kludge. (I'd name names, but not on list.) So that's when I began looking deeper, for a root cause. https://github.com/rjbs/Dist-Zilla/issues/586 documents the existence of a problem, and my p.r. works around it, but I don't think the discussion there actually pinpoints where in DZ the problem occurs (though I think I've narrowed it down to one particular module).

In any case, while I have enough to enable me to proceed on test-against-dev, I still think some mention of this somewhere in DT-TZ's docs would be helpful -- regardless of whether DZ gets fixed or not.

Thank you very much. Jim Keenan

[jkeenan]

On 05/22/2018 01:09 PM, Dave Rolsky wrote:

[snip]

Users of DT::TZ should not write test code using the |local| time zone without some extra thought first.

Perhaps that should go into the DT-TZ docs as well.

[autarch]

Perhaps that should go into the DT-TZ docs as well.

Yes, agreed.

[eserte]

After further study of the discussion in this ticket, I realized that the box completely lacked an /etc/password file. I then ran the FreeBSD tzsetup utility to establish US Eastern Time as our default time zone.

I think you mean /etc/localtime, not /etc/password. In this case this resembles #21.

[jkeenan]

On 05/22/2018 04:43 PM, Slaven Rezić wrote:

After further study of the discussion in this ticket, I realized
that the box completely lacked an /etc/password file. I then ran the
FreeBSD tzsetup utility to establish US Eastern Time as our default
time zone.

I think you mean |/etc/localtime|, not |/etc/password|.

Yes, my error.

In this case this resembles #21 https://github.com/houseabsolute/DateTime-TimeZone/issues/21.

Yes.