WatchPeriodically keeps complaining about a new version

[lobo-tuerto]

Hello,

I just installed this package, and added the Tz.WatchPeriodically as a child in my supervisor.

When going into the console with iex -S mix I see these messages:

[debug] Tz is checking for IANA time zone database updates
[warning] Tz found a more recent time zone database available for download at https://data.iana.org/time-zones/releases/tzdata2022c.tar.gz

So I did a manual update with:

Tz.Updater.maybe_recompile()

All good. But when I close the console and re-open it, I'm still seeing the warning above. The output of Tz.IanaDataDir.tzdata_version() is "2022c".

Do you know what could be going on?

Thanks!

[mathieuprog]

But the new tz data has not been downloaded, right?

[lobo-tuerto]

@mathieuprog That's probably it. How do you manually update then?

[mathieuprog]

Actually I read the code and it seems it is being downloaded if you execute Tz.Updater.maybe_recompile(). Execute Application.get_env(:tz, :data_dir) || :code.priv_dir(:tz) and check inside the folder if you got the new files?

[lobo-tuerto]

@mathieuprog It looks like it's downloaded the files:

Application.get_env(:tz, :data_dir)
# nil

:code.priv_dir(:tz)
#'.../project_root/_build/dev/lib/tz/priv'

ls _build/dev/lib/tz/priv/tzdata2022c/

# contains a bunch of files like
africa  antarctica  asia  australasia  backward  etcetera  europe
iso3166.tab  northamerica  southamerica  zone1970.tab

[mathieuprog]

Can you compare output of Tz.iana_version() and Tz.IanaDataDir.tzdata_version() and Tz.Updater.fetch_iana_tz_version()

[lobo-tuerto]

Ah hah! There it is the discordance:

Tz.iana_version()
"2022a"

Tz.IanaDataDir.tzdata_version()
"2022c"

Tz.Updater.fetch_iana_tz_version()
{:ok, "2022c"}

[mathieuprog]

The docs for Module.create/3 say:

Similar to Kernel.defmodule/2, the binary will only be written to disk as a .beam file if Module.create/3 is invoked in a file that is currently being compiled.

As we recompile at runtime, no .beam file will be created, so the newly compiled module will not be persisted.

But I think your problem is how to update the tzdata when only watching?

[mathieuprog]

tz keeps all the time zone data in maps that are computed during compilation time. Recompilation is done at runtime, and as mentioned above, the compiled file is not persisted.

When you said you exit the console and restart the console, you were killing your whole application and restarting it doing so?

[lobo-tuerto]

I was just starting my console with:

iex -S mix

Then killing it with double CTRL-C, and opening it again.

Somehow I thought Tz.Updater.maybe_recompile() would download latest and save it somewhere (tmp dir, _build dir, or somewhere) so whenever the app is started again it just gets from latest cached copy.

[mathieuprog]

So you have this problem because you are testing your application, but a real app in production, it keeps running, so you won't encounter this issue normally.

[lobo-tuerto]

And yeah, I was wondering about this too:

But I think your problem is how to update the tzdata when only watching?

[mathieuprog]

That depends, can you tell me first why do you use the watcher and not the updater?

[lobo-tuerto]

I usually don't like programs downloading stuff into a computer unless I approve it. Ideally I'd like to have the watcher, be alerted if there is an update available, log that event, then update manually.

I thought that was exactly the purpose of the Tz.WatchPeriodically existence.

[mathieuprog]

The main reason for the existence of the Watcher is to support systems that can't afford to compile at runtime.

When a new version of iana time zone data is found, the Watcher will log and you may also pass a callback to the Watcher to eg send yourself an email.

Then the idea was that the developer kills the application, the developer adds the new dependency in mix.exs when tz has released a new version with the new data, and the system would be relaunched. So, no update on runtime. Think of embedded systems with a limited amount of memory, not able to compile at runtime.

Your case is for security purposes, not the memory limitation mentioned above. What would you ideally want? If you want to approve the data first, that means that you would fetch the data locally, or have a tool to fetch the data. Then once scanned and approved, you need to upload the data to the server and have a function to recompile with the newly uploaded data. What do you think? (I'm off to sleep and will answer tomorrow)

[mathieuprog]

Btw for your case that you described it wouldn't make sense to call Tz.Updater.maybe_recompile() because it downloads the data and recompiles with fetched data without your approval regarding the safety of that data.

[lobo-tuerto]

Btw for your case that you described it wouldn't make sense to call Tz.Updater.maybe_recompile() because it downloads the data and recompiles with fetched data without your approval regarding the safety of that data.

Agree.

If you want to approve the data first, that means that you would fetch the data locally, or have a tool to fetch the data. Then once scanned and approved, you need to upload the data to the server and have a function to recompile with the newly uploaded data. What do you think?

That's more like it I guess.

---

I'm not in the situation where I need this ATM, but just wondering about the different scenarios and how this tool works.
I think your exposition makes sense, and I'm out of any further questions.

Thanks for the time and patience, have a good night! :wave:

[lobo-tuerto]

Hmm, for manually updates, is it possible to do it like tzdata describes here? https://github.com/lau/tzdata#automatic-data-updates

If the autoupdate setting is set to disabled, one has to manually put updated .ets files in the release_ets sub-dir of the "data_dir" (see the "Data directory and releases" section above). When IANA releases new versions of the time zone data, this Tzdata library can be used to generate a new .ets file containing the new data.

[mathieuprog]

For manual updates, could you please refer to the readme that I updated, under

For updating manually, there are two options:

The mix task has been added in new version 0.22.0

[lobo-tuerto]

@mathieuprog Thank you, I appreciate this!