ZIM Update v2: Revisit how we update ZIMs within a single month

[benoit74]

Currently, ZIM files are named with only the year + month.

It means that whenever we update a ZIM (typically by re-running a recipe to fix something), it is deemed to be hard to know which ZIM we have and to potentially cause issues on mirror updates (mirror might keep the old file version).

Should we change our way of working?

[benoit74]

Two important information to consider:

• the ZIM already contains the full date in most cases in the "Date" metadata, allowing to differentiate ZIMs that have not been generated the same day
• I just did a small test by running dart.dev_en_all again to generate a new ZIM within the same month ; all mirrors have grabbed the updated version (I downloaded ZIM file from each mirror individually and checked the SHASUM)

[rgaudin]

all mirrors have grabbed the updated version

That's a very good news. Not all mirrors mirror this file though so if you don't mind I think we need to confirm that all mirrors do update same-filename files before releasing our constraints

[benoit74]

I forgot to check that all mirrors are in the list you've linked, indeed.

I just requested again https://farm.openzim.org/recipes/wikipedia_dag_all to rebuild a ZIM with same name and check if all mirrors are OK with https://download.kiwix.org/zim/wikipedia/wikipedia_dag_all_mini_2024-06.zim?mirrorlist

Do we agree that we should not care about the Israeli mirror for now?

[kelson42]

I think there is anyway too much of assumption that this will go right. It won't.

we should make a call IMHO to nail down this old problem.

[benoit74]

What has been discussed:

• we need to revisit how we update ZIMs within a single month
• we need to document both why it will be done and the way it will be done
• this is a significant change, and we first need to assess the impact throughout the ecosystem and properly open issues (at least for analysis) in all repos

I propose to name the "ZIM Update v2" project (I know, I suck at naming things)

I started the documentation at https://wiki.openzim.org/wiki/ZIM_Updates so that it will be easier to trace changes.

Do not hesitate to speak-up if I missed something or something is wrong (ideally simply edit the document if it is obvious).

Please give feedback if documentation seems to appropriately reflect what we've discussed, so that we can start to share the world in the openZIM community and open related issues.

[rgaudin]

I thought we decided on YY-MMll and not YY-MM.ll

Another impact on mirrors is that some (at least IL one) filter based on filename. A simple test should be conducted to check whether this post-period change breaks sync for them or not.

[benoit74]

I thought we decided on YY-MMll and not YY-MM.ll

Agreed

Another impact on mirrors is that some (at least IL one) filter based on filename. A simple test should be conducted to check whether this post-period change breaks sync for them or not.

Yep