Loki logger

[wneirynck]

Currenty we are using Loki for application logging, and custom implementations for build script logging. However, it may be possible to use Loki for build logging as well. It exposes an api that we can use for pushing and for pulling.

We could mark each stream with customer, repo, build and job id, and even the specific command. The advantage is that it already provides the functionality we need. Disadvantage is that it is a third party tool where we have little control over, and there seems to be no support to use OCI buckets, except for using Minio with OCI bucket backend.

[wneirynck]

Loki does not provide security of its own, but we could add it to apache, using the mod_auth_bearer module. Build scripts should get a short-lived token (also used for the api) and use that when pushing logs.

When retrieving logs, we could either pipe it through the API (inefficient), or expose Loki directly to the client. This requires of course strict security checking: customers should only be able to see their own logs, for instance. This would also mean that the client token should be valid for the log server (which it currently is).

Loki has a concept of tenants, which may be useful to separate logs on a client basis. We should investigate how we could use this to enable Apache to block any requests going to another tenant (= customer). Possibly with header checks, or somehow use the customer id from the request as the X-Scope-OrigID token for Loki.

[wneirynck]

Tenant implementation ok, as well as using OCI buckets for storage. But security is not complete yet: it is possible to access Loki directly from a container, and the token check does not verify if the user has access to that specific customer (= tenant id).

For the first issue we should add network rules restricting direct access. For the second we either need to add a module to Apache (if one exists that is capable of those checks), or use another proxy (Nginx, HaProxy), should one exist that supports our requirements. If not, we will have to implement something ourselves.