search

libical / vzic

A program to convert the IANA (formerly Olson) timezone database files into VTIMEZONE files compatible with the iCalendar (RFC5445) and TZDIST (RFC7808) specifications
Other
16 stars 14 forks source link

Provide a public version of the generated VTIMEZONES #4

Closed minichma closed 2 years ago

minichma commented 5 years ago

Would be great to have a publicly available master set of VTIMEZONES as produced by vzic. Should be maintained in parallel to the official IANA tzdb publications and released as regular releases of this project.

I'm aware of the set at https://github.com/libical/libical/commits/master/zoneinfo, but I think, having a clean release history only of the timezone data within this project would crtainly be beneficial.

winterz commented 5 years ago

in general I like the idea. except I don't know that I want to be on the hook to generate the VTIMEZONES every time IANA updates. the generation isn't 100% automatic either.

step 1 is to make sure the instructions are fully up-to-date step2 is to write a script that automates as much as possible then we can revisit .

do you think the VTIMEZONES should be in a new git repo or should it be part of the vzic repo?

minichma commented 5 years ago

I don't know that I want to be on the hook to generate the VTIMEZONES every time IANA updates

Sure, that's completely understandable. I just noted, when having to update the VTIMEZONES in our proect, that it's probably the same kind of work, all other users of the data have to do in a similar way. Having the data centrally published would probably safe many people quite some work and would also improve the quality of the data (because in my case, I didn't take care of stable tzids for instance). From time to time I will be able to jump in and do the update (in cases where I would do it anyways), but I probably don't want to be on the hook either. ;-)

step 1 is to make sure the instructions are fully up-to-date

What do you mean with the instructions? The README.md? Or the tzdb?

do you think the VTIMEZONES should be in a new git repo or should it be part of the vzic repo?

Hm, not sure. From a user's endpoint I don't care too much, I'm just interested in a centrally managed DB. But I think, that having it in this repo would make sense, because the code needs to be developed hand in hand together with the data (if the format changes). A flow could look as follows:

Updating to a new version of tzdata would involve:

We would probably create an initial release from the VTIMEZONE data as currently published in the libical project to have a starting point for the "previous VTIMEZONE data" setting.

I could maybe help with some of the scripting, but it will probably take some time and we would have to agree on the intended flow first.

winterz commented 5 years ago

I think your work flow is pretty much correct. I don't see any problems

the instructions can be found here

let's worry about the scripting now and decide which repo to use later. hand-editing the Makefile has always been a pain to me. perhaps we simply pass the settings on the make command line

minichma commented 5 years ago

I started a branch for building a new zoneinfo file from sources.

The new build.sh downloads the tzdata archive and the master zoneinfo, builds and runs vzic and then merges the result with the master zoneinfo.

The build is intended to be run in a docker container built from the new dockerfile. The travis.yml can be used for running automated builds on Travis.

The branch is early stage, so I didn't create a PR yet. One problem is, that the merge script doesn't seem to work as expected. It seems that all timezones are always considered as changed, even if only the tzid changed. This might be related to some trivial thing like line-endings, but it also seems that the vzic-merge.pl doesn't ignore changes in the LAST-MODIFIED field. Is this intentional? Am I missing something?

The source URLs are configured in the new settings.config file. I think, having the source files checked into the source repo helps, because it's versioned. When creating releases, the corresponding tag will cover not only the vzic source code, but also the reference to the source data that was used for building and thils will make the build more or less reproducible.

winterz commented 5 years ago

I just added your build_automation branch to the "official" vzic repo. I will also make sure you have write access.

winterz commented 5 years ago

you should have admin access on the vzic repository now. Now I will test build.sh for the tzdata2019c. If it works I will merge build_automation to master.

winterz commented 5 years ago

for the diffs, I think you are right that the LAST-MODIFIED is not being ignored by the comparison. In this instance, however, the.ics's file also had a X-PROLEPTIC-TZNAME property that is now gone.

so the comparison is seeing differences in LAST-MODIFIED and also X-PROLEPTIC-TZNAME is removed.

minichma commented 5 years ago

Allen, thanks for the permissions!

Ok, so the X-PROLEPTIC-TZNAME change correctly triggers a new TZID. Should we create a separate ticket for the LAST-MODIFIED issue or just fix it as part of this one?

There also seems to be an issue with line endings. Perl always assumes LF line (and so Perl Regex do), but at least the zoneinfos contained in my initial test release contains CRLF line endings. This is one more reason, why merging doesn't work as expected at least in the build_automation branch so far. What are the preferred line endings in libical project? I guess just LF!? So we should make sure, the zoneinfos produced by the automated build has the right line endings.

I'm not too much into Travis, so the .travis.yml will need some more work. One issue was the language, which was set to ruby, which was incorrect. Fixed this by setting the language to generic. The main issue remaining is the publishing of the build artifacts (i.e. the newly created zoneinfo).

I will try to put some more work into this when I have time.

minichma commented 5 years ago

vzic produces zoneinfo files with CRLF line endings, but perl expects and produces LF line endings. Which kind of files do we want to produce? I'd suggest LF, because the produced .tar.gz is quite Linux-specific. It contains features like linked files. Also the file type .tar.gz points towards Linux. What do you think?

winterz commented 5 years ago

LF is fine with me.

minichma commented 5 years ago

Should we also change vzic-output.c, etc. to output LF? This would improve consistency, so I'd suggest to do so, but libical and other projects might expect CRLF.

dilyanpalauzov commented 4 years ago

tzurl.org offers this.

castaway commented 3 years ago

The spec specifies CRLF - https://tools.ietf.org/html/rfc5545, so I would suggest keeping that, in case parsers throw a wobbly.

mirabilos commented 3 years ago

Maybe have a repo with just the generated files, so I can clone that even on “obscure” operating systems where porting a recent enough libglib, libical, etc. would be annoying.

I could, sure, just do the conversation myself on a Debian box and rsync the result over, but vzic is not packaged, and this is too many yaks to shave for the thing I had a crazy idea to do. Best to have “officially correct” versions of the .ics files (which, indeed, mandate CRLF).

minichma commented 2 years ago

I agree that introducing a separate repo for tracking the generated files would make sense. I put some work into preparing this repo. It holds the latest version of tzdata and the generated zoneinfo as part of the repo, which allows viewing the history very nicely. It also has a tag and a release with the archived files for each release of tzdb (starting with version 2000f). Updating to the newest version is fairly automatic and boils down to updating the version name in the settings.config file.

@winterz I'd be happy to transfer the repo to the libical organization if you feel this would be helpful. This would also make #17 obsolete.

winterz commented 2 years ago

@winterz I'd be happy to transfer the repo to the libical organization if you feel this would be helpful. This would also make #17 obsolete.

Feel free. let me know if you don't have enough permission to make this happen

minichma commented 2 years ago

Just tried to transfer. Received

You don’t have the permission to create public repositories on libical

winterz commented 2 years ago

try now I made you an owner of the libical organization

minichma commented 2 years ago

Worked, thank you. See https://github.com/libical/tzdbics. I added access for the Owners and Developers teams. Will close this ticket.

mirabilos commented 2 years ago

Markus Minichmayr dixit:

Worked, thank you. See https://github.com/libical/tzdbics. I added

Differences:

• CRLF line endings, unless they are by design (which may very well be the case)

• major differences in all files, seem to be: – no cutoff of pre-1970 data (may also be deliberate) – sub-minute timezone offsets (??)

• metadata

-PRODID:-//citadel.org//NONSGML Citadel calendar//EN +PRODID:-//github.com/libical/vzic//NONSGML ICS//EN

-TZID:/citadel.org/20220930_1/Europe/Brussels +TZID:/github.com/libical/tzdbics/20221031_2019c/Europe/Brussels

+X-PROLEPTIC-TZNAME:LMT

• symlinks instead of TZID-ALIAS-OF

That and the use of a different tzdata version for generation make an initial comparison really hard.

bye, //mirabilos -- [16:04:33] bkix: "veni vidi violini" [16:04:45] bkix: "ich kam, sah und vergeigte"...

minichma commented 2 years ago

@mirabilos The CLRF are by design, as you pointed out. For the other points, especially the 'TZID-ALIAS-OF' discussion, please create separate tickets, so we can discuss these independently. Maybe explain what you're intending to do.

minichma commented 2 years ago

@mirabilos btw, if you just wanted to get zoneinfo built in a specific way, you could just fork the tzdbics repo and modify settings.config as needed (just added some configurability). Once you push you changes, GitHub will build, commit and push the updated zoneinfo files for you. No need to set up a compatible linux box on your side.