Selfhost zimit

[deimjons]

Hi, folks. I am trying to self-host Zimit. How to configure it with the backend? Here is my docker-compose:

version: '3.9'
services:
  zimit-ui:
    image: ghcr.io/openzim/zimit-ui:latest
    container_name: zimit-ui
    restart: unless-stopped
    labels:
      traefik.enable: true

      traefik.http.routers.zimit-http.rule: Host(`zimit.example.com`)
      traefik.http.routers.zimit-http.entrypoints: web
      traefik.http.routers.zimit-http.middlewares: https-redirect

      traefik.http.routers.zimit-https.rule: Host(`zimit.example.com`)
      traefik.http.routers.zimit-https.entrypoints: websecure
      traefik.http.routers.zimit-https.tls: true
      traefik.http.routers.zimit-https.tls.certresolver: prod
      traefik.http.routers.zimit-https.middlewares: authentik@file

      com.centurylinklabs.watchtower.enable: true
    networks:
      - traefik
      - zimit

  zimit:
    image: ghcr.io/openzim/zimit:zimit2
    container_name: zimit
    restart: unless-stopped
    command: 'zimit --url https://lurkmore.online/ --name lurkmore --workers 2 --waitUntil domcontentloaded'
    shm_size: 1gb
    volumes:
      - output:/output
    networks:
      - zimit

volumes:
  output:
    driver: local
    driver_opts:
      type: none
      o: bind
      device: $VOLUMES_PATH_PREFIX/zimit/volumes/output

networks:
  zimit:
    name: zimit
  traefik:
    external: true

[benoit74]

I'm not sure to understand what self-hosting zimit means to you.

This repo (zimit-frontend) is the web frontend you can see on youzim.it

As of today, this frontend is creating jobs a a zimfarm. One zimfarm worker will then pick-up the job, run it, and upload it somewhere.

In order to self-host it, you will hence need to host both the zimit-frontend and the zimfarm (including at least one worker).

The worker is composed of various containers which will be responsible to run the scraper in the end.

You will also need some external services: an S3 or SFTP servive to store ZIMs which will be created and a Mailgun account to send mails when the Zimit task completes.

This is a significant infrastructure to put in place. If your goal is simply to create a ZIM, you probably do not need it at all. Simply running manually the zimit container (with a docker command, no need for compose, the scraper is not long-lived, it will stop once job is done) to run the scraper is way simpler. But I agree you have no UI in such a situation.

Should you really want to self-host this whole thing because you want a UI, you can have a look at https://github.com/openzim/zimit-frontend/blob/main/dev/docker-compose.yml, this is the local stack developers use to start most components. I think the only missing part is the Zimfarm worker (plus external services of course).

Starting point to add a worker to a Zimfarm is here: https://github.com/openzim/zimfarm/wiki/Create-a-new-worker and you might have a look at how we deploy it for dev purposes at https://github.com/openzim/zimfarm/blob/main/dev/docker-compose.yml (worker is one main container, the worker_mgr, which in turn starts one container per task to run - the task mgr - which in turns start multiple containers - the scraper, a dns cache, an uploader once job is done, ...).

Another source of inspiration is probably the k8s stack we deploy for youzim.it ; everything is at https://github.com/kiwix/k8s/tree/main/zimit (I'm not saying you need to use k8s, this is probably overkill, but these files might help you identify how to properly configure it).

[deimjons]

First, I think that this frontend for zimit (because of name). If zimit can't work with this frontend I will try to setup zimfarm. Thank you for the answer.

[benoit74]

• Zimit can work without this frontend, it's a purely CLI tool
• This is not a frontend for Zimit scraper, this is a frontend to generate Zimit tasks on a Zimfarm

If your concern is that using the CLI is too complex for you, then this tool is not the right approach, it is going to be even more complex to setup. This tool provides value only to help end-users create zimit tasks to be ran on another machine than theirs.

[deimjons]

Got it. For me is ok to use CLI but for my local users needed some frontend. Thanks for advices.