Library unavailability (3.5.0-2)

[rgaudin]

This is a tracking ticket to observe the evolution of a new situation.

On Friday 2023-07-21 at 15:02 UTC, we've upgraded library.kiwix.org to kiwix-serve 3.5.0-2.

Since, Uptime robot (HEAD https://library.kiwix.org) has seen two 503 Service Unavailable events:

• Tuesday 2023-07-25 at 05:12 UTC which lasted 4mn
• Wednesday 2023-07-26 at 00:15 which lasted 1mn

This is unexpected. The last time we had a 503 was 2 years ago. This service is virtually never down.

Note: library.kiwix.org is composed of three services:

• the varnish cache as frontend
• a library-catalog kiwix-serve that serves /catalog requests
• a library-demo kiwix-serve that serves the rest.

The catalog service/endpoint was not affected. For the sake of it, I've just created an uptime robot monitor for the catalog endpoint.

I found that this 503 is not the result of a problem with the k8s nginx proxy nor the varnish cache but a restart of the library-demo container that serves the home page.

This container restarts frequently (kiwix-serve crashing) but we've never been affected because:

• homepage serves a static HTML page. Requests to homepage should not be crash-prone
• homepage is cached in varnish for 1d.

Possibilities:

• Uptime robot request happened at ttl expiry which coincided with a restart. Given uptime robots checks every minute, it's possible but seeing two occurrences of this after years of none leans towards a different cause.
• kiwix-serve now takes longer to restart, increasing chances of hitting it at ttl expiry
• kiwix-serve can crash (or exit) on request to the home page (note that uptimerobot request are simple and don't query anything but the home page)

Notes:

• The last restart's exit code was 0 which itself is problematic but can't say much without logs.
• kiwix-serve takes over a minute to start with such a library
• It doesn't seem like this kiwix-serve version restarts more than the previous one. We're at 9 restarts for 5d at the moment.

[rgaudin]

Updated:

• Tuesday 2023-07-25 at 05:12 UTC which lasted 4mn
• Wednesday 2023-07-26 at 00:15 which lasted 1mn
• Monday 2023-08-07 at 12:29 UTC which lasted 1mn

[rgaudin]

Updated:

• Tuesday 2023-07-25 at 05:12 UTC which lasted 4mn
• Wednesday 2023-07-26 at 00:15 which lasted 1mn
• Monday 2023-08-07 at 12:29 UTC which lasted 1mn
• Tuesday 2023-08-15 at 16:29 UTC which lasted 1mn

[rgaudin]

I have just configured two replicas for both the catalog backend and the demo backend. This is not a real load balancer, but requests are randomly sent to each of the two replicas randomly (0.5 each). No round-robin or anything fancy ; it's done by iptables at routing level.

@mgautierfr do you think it could make things worst in terms of performance? Having a single instance ensured that we'd use ZIM access cache correctly… but with two instances, we'd have ZIM caches on both for the same ZIM files…

Given every single page generates a lot of requests, I don't think it's gonna be much of an isssue.

[mgautierfr]

I'm not sure this has a lot of impact at zim level. Of course, it will double the cache as now we have two caches (one per process). But a lot of things are either common or totally unrelated:

• Common : css or js resources. If a user access it, it will be shared with other.
• Unrelated: different articles in the same zim or two article in two different zims. Whatever if we have one common cache or two, if things are not related, cache will not be populated with the requested resource.

But it can have a impact at kiwix level when we cache the whole Archive reader. If each requests are randomly balanced, both instance will have to open the archive. Would it be possible to load balance randomly but base on the IP (all requests of the same IP goes to the same instance) ? For common zim file (wikipedia english) both instance will probably ends by opening it, but for uncommon zim with only one reader, only one instance will handle it.

[rgaudin]

We discovered this week that varnish was not caching kiwix-serve requests anymore since the change. Our assessment was thus incorrect about what happened. Closing.

FYI, we now have a daily check to ensure we're notified should we stop returning non cached requests