High CPU usage when starting [Docker]

[M4RC02U1F4A4]

I use kiwix-serve with Docker and when the container starts the CPU (Ryzen 5 3600) is stuck at 100% with all 12 threads for like 5 to 10 min, is this a normal behaviour?

I found also that a lot of data are read from the SSD, so i think that the container read all of the zim files, but I didn't notice excessive RAM usage

The container is on the latest version kiwix/kiwix-serve:latest I have 3 ZIM file wikipedia_it_all_maxi.zim (22,1 GB) wikivoyage_it_all_maxi_2020-11.zim (378 MB) wiktionary_it_all_maxi_2020-11.zim (380 MB)

[veloman-yunkan]

I wonder what fraction of that start-up latency can be attributed to openzim/libzim#430 (see https://github.com/openzim/libzim/pull/430#issuecomment-701415069)

[kelson42]

Something has to be wrong. I would reduce the test case to one ZIM file.

[M4RC02U1F4A4]

Something has to be wrong. I would reduce the test case to one ZIM file.

The problem persist with only the wikipedia_it_all_maxi.zim (22,1 GB) in the directory Instead with only one of the other two smaller file thare are no problems

[kelson42]

@Marco1097 which version of WPIT maxi it is exactly? I don't see any date in the filename.

[M4RC02U1F4A4]

@kelson42 I don't know, probably the one available during 09/2020 I saw that a version dated 10/2020 is available, I try to download it and let you know if anything changes

[kelson42]

@Marco1097 We had a bug with a similar symptom earlier this year, around spring. If you could confirm with a recent ZIM file, I could try to reproduce.

[M4RC02U1F4A4]

@kelson42 Same problem with wikipedia_it_all_maxi_2020-10.zim

[xontik]

HI ! Same issue here with wikipedia_en_all_maxi_2020-10.zim, Every page is very slow, but become fast when it worked once.

Is there some kind of indexes ?

[kelson42]

This is a Docker specific problem... currently fixing it.

[kelson42]

@xontik @Marco1097 Problem should be fixed, please docker pull kiwix/kiwix-serve and confirm (or reopen the ticket).

[xontik]

I know got some kind of weird log and the server exit after 👍

kiwix_1  | /usr/local/bin/kiwix-serve --port=80 wikipedia_fr.zim wikipedia_en.zim stackoverflow_en.zim
kiwix_1  | Unable to add the ZIM file 'stackoverflow_en.zim' to the internal library.
kiwix_1  | Here is the content of /data:
kiwix_1  | /data/wikipedia_fr.zim
kiwix_1  | /data/stackoverflow_en.zim
kiwix_1  | /data/wikipedia_en.zim

it works well if i remove the stack overflow one

[M4RC02U1F4A4]

@kelson42 It all seems to work, I also tried with a larger ZIM file (93GB) and there is no problem. Thanks for all

[kelson42]

@xontik Yes, the SO ZIM has a problem, we should make a new one.

[xontik]

@kelson42 ok then thank for the quick fix !

Where can i have follow new zim posted ?

[kelson42]

@xontik We have an OPDS feed, see https://wiki.kiwix.org/wiki/OPDS