Additional explanations in "Cannot add zim X to the library" error

[bwenrich]

The kiwix-manage add command is difficult to debug when failures happen. Multiple different scenarios produce the same error message, so it is unclear what is the problem.

I understand that there are multiple reasons why the add command might fail, ex:

• The Library XML does not exist
• The Library XML is not writable due to incorrect permissions
• The Library XML is not writable due to read-only filesystem
• The ZIM file does not exist
• The ZIM file is not readable due to incorrect permissions
• The ZIM file is corrupt
• (Other reasons I have not considered here)

Steps to reproduce

# Step 1: Create the ZIM file which is not readable to the kiwix user (UID=808)
/data $ ls -l /data/content/ifixit_en_all_2022-06.zim*
-rw-------    1 root     root     2717836715 Jun 18 10:24 /data/content/ifixit_en_all_2022-06.zim
-rw-------    1 root     root            92 Aug 19 03:19 /data/content/ifixit_en_all_2022-06.zim.sha256
/data $ id
uid=808 gid=808
# Step 2a: Attempt to add the unreadable ZIM
/data $ kiwix-manage /data/library/multi.zim.xml add /data/content/ifixit_en_all_2022-06.zim
Cannot add zim /data/content/ifixit_en_all_2022-06.zim to the library.
# Step 2b: Attempt to add a non-existent ZIM
/data $ kiwix-manage /data/library/multi.zim.xml add /data/content/not_existing.zim
Cannot add zim /data/content/not_existing.zim to the library.
# Step 2c: Attempt to add a corrupt or invalid format ZIM
/data $ dd if=/dev/urandom of=/data/content/corrupt.zim bs=1M count=10
/data $ kiwix-manage /data/library/multi.zim.xml add /data/content/corrupt.zim
Cannot add zim /data/content/corrupt.zim to the library.
# Step 3: Observe that the failures in 2a, 2b, and 2c produce the same error message, but have different causes.

Actual Behavior

The error message "Cannot add zim to the library." appears with no additional detail.

Expected Behavior

The error message "Cannot add zim to the library." appears with additional detail about why the failure happened. If it is not possible to detect the reasons for the failure, perhaps guidance about common issues could be displayed.

Environment

kiwix-serve:latest in Docker/k8s. /data volume is an NFS mount

kiwix-tools 3.3.0
libkiwix 11.0.0
libzim 8.0.0

[kelson42]

Sounds a good idea indeed.

[SauravChittal]

Hello, I wanted to work on this specific issue and while I understand what the issue itself is, I'm a bit more confused about the method of solving the issue. I understand that we need to check whether the ZIM file or the XML file has certain permissions.

However, understanding what is causing the specific issue requires manager.addBookFromPathAndGetId function to return the empty string, which because the class is abstracted away from this file, I'm unable to check which specific if statement has failed.

It will be possible to check for all of the pointers presented above on why the function might fail, by trying to call the file using FILE pointers, however, I was wondering if that is the recommended method, since it doesn't truly account for other potential reasons why that might occur.

Thank you so much :)

[kelson42]

@veloman-yunkan maybe you can help here? I would do a function which checks all what should be checked given a file path and call it only if the addBookFromPath() fails so the user gets the appropriate hint.

[SauravChittal]

Would it be fine if I made such a function, and the made a pull request based on how I think you would check for the issue?

[mgautierfr]

I would make addBookFromPathAndGetId throwing a exception if it cannot add it and put in the exception (as a message string or as a specific subclass) the cause of the error.

Then, the client kiwix-manage would have to catch the exception and print the information to the user.

[veloman-yunkan]

@veloman-yunkan maybe you can help here? I would do a function which checks all what should be checked given a file path and call it only if the addBookFromPath() fails so the user gets the appropriate hint.

@kelson42 I can only concur with @mgautierfr's comment above.

[kelson42]

@veloman-yunkan @mgautierfr OK!

[dounoit]

well- im setting up my raspi and have the zim files on an extfs usb stick and tried running the kiwik docker image tools- not matter what i do i get that same error that it cant add the zim and / or doesn't see it in /data when i try to run it direct via the server:

 docker run -it  ghcr.io/kiwix/kiwix-tools kiwix-manage ./zim/library.xml add ./*.zim
Cannot add zim ./wikihow_en_maxi_2023-03.zim to the library.

i was looking for a -v -vv or anything to help me troubleshoot but this is making it difficult to troubleshoot- i assume it has something to do with the filesystem- ive been noticing some issues with permissions while trying to run containers with extfs- seems to not be playing nice :-). -D

[dounoit]

ok - i was able to get it running on my laptop running:

docker run -d --name kiwix -v ./:/data -p 8282:80 ghcr.io/kiwix/kiwix-serve '*.zim'

it seems to be crashing on my raspi though- still testing! :-) -D

[dounoit]

ok good news- i think i really don't know what im doing with this :-) im more comfortable with docker-compose files tbh-

version: '3.3'
services:
  kiwix-serve:
      ports:
        - 8080:8080
      image: ghcr.io/kiwix/kiwix-serve:latest
      # uncomment next 4 lines to use it with local zim file in /tmp/zim
      volumes:
        - ./:/data
      command:
        - 'w*.zim'
      # uncomment next 2 lines to use it with remote zim file
      # environment:
      #   - 'DOWNLOAD=https://download.kiwix.org/zim/wikipedia_bm_all.zim'

i was using * which also grabbed a file i was still transferring to the same directory- i think it crashed because of that - when i excluded the partial file - its seems to have come up great on my raspi3 on an extfs usb thumbdrive - this is cool- ready for the internet to be cut now friends! i'm grateful for this solution- seriously! thank you! -D