search

kiwix / operations

Kiwix Kubernetes Cluster
http://charts.k8s.kiwix.org/
5 stars 0 forks source link

Prod library generation is taking way too long #169

Closed benoit74 closed 3 months ago

benoit74 commented 3 months ago

The generation of the production library_zim.xml by zim/library-generator cronjob is taking between 3 and 4 hours while it should be much faster.

rgaudin commented 3 months ago

Looking into it it appeared to have self resolved and is back to around 3mn. I looked at the previous runs' logs and found the worst one that started at 10:00 today. It took around 3h30mn almost entirely during the READ phase which consist in listing files and reading ZIM metadata. The following runs were all shorter and shorter until it resolved completely.

This looks like an I/O resources shortage ; probably #170