How to demo ZIM files?

[kelson42]

We need to create instances to demo new ZIM files:

• without having to publish them publicaly on library.kiwix.org
• without mixing them with other ZIM files
• on a speedy server
• easy to deploy and destroy
• only short live instances: 1-2 months

[kelson42]

The docker image of kiwix server is able to download a ZIM file. So we just need to start it wirh right argument to have an instance. We were able to do that easily with sloopy... maybe we could achieve to do something similar with an other provider?

[rgaudin]

Looks like you've ruled out using our own infra for some reason

[benoit74]

Some questions:

• does the ZIM must be served from a kiwix.org subdomain (are random names ok?)
• does the tool must auto-destroy the instance on given date (with a possibility to postpone, but by default a given date is set where the instance will be destroyed, otherwise we will have tens of hundreds of instances)
• does the tool to demo a new ZIM must be accessible to editors without developers involvement
• does the tool must be able to serve multiple ZIMs from one single kiwix-serve instance

[rgaudin]

Following today's discussion, we'll investigate first how to deploy it on our current infra.

• single domain (demo.openzim.org?)
• single kiwix-serve
• single varnish
• kiwix-serve home page hidden (rewritten by cache) so users can only access ZIM via full URL
• wildcard entry for *.<domain>
• automatic redirection from <xxx>.<domain> to <domain>/viewer#<xxx>. Should we redirect or serve in-place?

Main objective being that partners are given a single URL that leads only to the content we want them to look at.

[benoit74]

@Popolechien would this be a suitable replacement for links you currently send to partners on dev.library.kiwix.org?

Do you have any feedback to share on the questions I asked in previous comment?

I have a proposition of enhancement to what is reported by rgaudin (could be done in a second step): do we have a simple way to showcase the ZIM logo, title, description and long description on the partner "home page"? Quite important to check this as well from my experience so far.

Maybe a small tool that generate proper HTML with Jinja2 could do the job of properly configuring things (varnish and kiwix-serve) and destroying them when not needed anymore. We could even imagine to have one custom page per partner with all their ZIMs, could be helpful.

[Popolechien]

Not opinion on the wildcard part but we could simply decide that any new zim produced would overwrite the previous one.

@benoit74 Not sure I understand what your proposal entails that we don't already have, but if it is related to sending people to a library dashboard so they see their single card and then need to click to see the actual content (kind of like this), yes I would consider this a requirement.

[kelson42]

• wildcard entry for *.<domain>
• automatic redirection from <xxx>.<domain> to <domain>/viewer#<xxx>. Should we redirect or serve in-place?

Yes, even if I don't really get the added value. Does not really seems necessary because if there is no home page, each ZIM is properly isolated from the others already.

It should be really easy to setup/remove a ZIM file from there, so it can be requested directly via chat and made within a few minutes.

[Popolechien]

Following our discussion on 3/07 we agreed on:

• One url per zim
• Automatically deleted after X days

Something that I'd like to add is that not all zims should get their unique URL, particularly if said URL is not easy to figure out (ex: zimxyz1245.kiwix.org). Ideally we should be able to flag the zims we want to showcase, and dump all others in a generic dev folder.

[benoit74]

Something that I'd like to add is that not all zims should get their unique URL, particularly if said URL is not easy to figure out (ex: zimxyz1245.kiwix.org). Ideally we should be able to flag the zims we want to showcase, and dump all others in a generic dev folder.

This is not exactly the idea for now as far as I've understood. Idea is that we continue to use the folders / Zimfarm as-is". But we do not push dev.library.kiwix.org or library.kiwix.org links to our clients. Instead whenever we want to provide a preview to a client we make a request/configuration (somewhere) to create the required "website / preview / demo" stuff (based on ZIMs published in dev or another folder, we won't mind)

[benoit74]

I've created a first infra to showcase what could be possible.

Currently everything is statically configured, but everything is in place to make it configurable at will.

In current infra, we have two demos:

• benoit at https://demo.library.kiwix.org/home/benoit/
• marc at https://demo.library.kiwix.org/home/marc/

If you don't know your demo URL, you have access to nothing. It is not really secure (one can pretty easily guess others URLs by brute-force typically) but at least it allows to not have direct interactions / confusions between our clients and we do not publicly expose our list of clients in a web page.

The /home/ part is a bit weird but it is the simplest solution to avoid me headaches and it is not ugly at all from my PoV.

For benoit we supposedly wanted to demo two ZIMs: videos/ubongo_fr_all_2024-07.zim and ted/ted_mul_all_2024-07.zim

For marc we supposedly wanted to demo two ZIMs: .hidden/dev/devhints.io_en_all_2024-08.zim and .hidden/bard/stackoverflow.com_en_bard_2021-11.zim

The idea is that we can create as many demo as necessary, each hosting as many ZIMs as necessary (so that for instance we can update the ZIM when a new version is created without having to send a new link to marc / benoit).

Everything will be configurable in a single YAML file, something like:

demos:
- url: benoit
  zims:
    - videos/ubongo_fr_all_2024-07.zim
    - ted/ted_mul_all_2024-07.zim
- url: marc 
  zims:
    - .hidden/dev/devhints.io_en_all_2024-08.zim
    - .hidden/bard/stackoverflow.com_en_bard_2021-11.zim

The file will be stored somewhere and a job will fetch it every hour to rebuild the files needed (configuration + static HTML for demo home pages).

Note that it looks like kiwix-serve had many difficulties to serve .hidden/bard/stackoverflow.com_en_bard_2021-11.zim appropriately, but now that main page is cached, it is very fast.

For curious persons, we have:

• one varnish serving as cache to speed everything up
• one caddy to host home pages with details about every ZIMs available (so that it is possible to validate the title, description, illlustration, ... all sourced from the ZIM)
• one kiwixserve serving all requested files

This infra is hosted on the same machine as library.kiwix.org and dev.library.kiwix.org and has access to all our ZIMs (prod, dev, private, ...). It is however isolated from dev.library.kiwix.org by having its own software