App Suggestion: Loki

[clouedoc]

Loki is a log database that is well integrated with Grafana. It would make sense to have Loki as a Caprover one-click app to help deploy a complete telemetry stack.

Do you know if there is any official Docker image for the app?

This image is the official image for Loki: https://hub.docker.com/r/grafana/loki

[clouedoc]

To deploy Loki, I used the following method:

1. Create an app named "loki"
2. Deploy Docker image named grafana/loki:main-6205b96
3. Add a persistent volume. Directory: /loki, volume name: loki-data
4. Set container HTTP port to 3100
5. Enable HTTPS
6. Enable basic authentication.

[JannikArndt]

Thank you for the helpful instructions 👍

Some more clues:

• there is no default web interface, so loki.mysite.com will return 404 page not found — you can check /metrics or /ready, more helpful endpoints are /config, /services and /loki/api/v1/labels (see https://grafana.com/docs/loki/latest/api/ for full api)

• you might want to install the Log Driver as a Docker plugin, see https://grafana.com/docs/loki/latest/clients/docker-driver/ ssh into your machine and run

  docker plugin install grafana/loki-docker-driver:latest --alias loki --grant-all-permissions
  docker service update --log-driver=loki --log-opt loki-url="http://srv-captain--loki:3100/loki/api/v1/push" --log-opt loki-retries=5 --log-opt loki-batch-size=400 srv-captain--the-service-to-follow

  • Alternatively you can define the log driver in the Pre-Deploy Script (see here)

    function preDeployFunction(capRoverAppObj, dockerUpdateObject) {
    dockerUpdateObject.TaskTemplate.LogDriver = {
    Name: "loki",
    Options: {
      "tag": "captain-srv",
      "loki-url": "http://srv-captain--loki:3100/loki/api/v1/push",
      "loki-retries": 2,
      "loki-batch-size": 400
    }
    };
    return dockerUpdateObject;
    }

  • or set it globally as the default driver for the docker daemon in /etc/docker/daemon.json (source):

    {
    "debug" : true,
    "log-driver": "loki",
    "log-opts": {
      "loki-url": "http://host.docker.internal:3100/loki/api/v1/push"
    }
    }

    Note that this will only apply to new containers, not already created ones (see docs) For debugging, follow the docker daemon logs at journalctl -xu docker.service -f

  Disclaimer: I haven't figured out how to use the service's url (srv-captain--loki) yet and resorted to the public url + https + basic auth

• if your services and your Grafana run inside CapRover, you don't need to expose Loki as a web app

[kinsomicrote]

Hello @clouedoc and @JannikArndt thanks for this. I have a question though; do you make use of promtail too? If so, how are you making it work?

[kinsomicrote]

Never mind, I've figured it out - I don't need promtail. However, I'm faced with an issue where I get an error like this for my Rails app container;

Sep 21 10:08:33 server dockerd[70205]: time="2023-09-21T10:08:33.505909778Z" level=error msg="Not continuing with pull after error: errors:\ndenied: requested access to the resource is denied\nunauthorized: authentication required\n"
Sep 21 10:08:33 server dockerd[70205]: time="2023-09-21T10:08:33.505979898Z" level=info msg="Ignoring extra error returned from registry" error="unauthorized: authentication required"
Sep 21 10:08:33 server dockerd[70205]: time="2023-09-21T10:08:33.509659128Z" level=error msg="pulling image failed" error="pull access denied for img-captain-railsapp, repository does not exist or may require 'docker login': denied: requested access to the resource is denied" module=node/agent/taskmanager node.id= service.id=
Sep 21 10:08:47 server dockerd[70205]: time="2023-09-21T10:08:47.137726268Z" level=info msg="Container failed to exit within 10s of signal 15 - using the force" container=

Looks like a docker thing, not sure what the solution is yet.

[clouedoc]

@kinsomicrote this is not related, so you should ask your question somewhere else. I'll give you a hint: pull access denied for img-captain-railsapp <-- are you sure this is the Docker image you want to be pulling?

[pcace]

To deploy Loki, I used the following method:

1. Create an app named "loki"
2. Deploy Docker image named grafana/loki:main-6205b96
3. Add a persistent volume. Directory: /loki, volume name: loki-data
4. Set container HTTP port to 3100
5. Enable HTTPS
6. Enable basic authentication.

Hi there, i have just followed these instructions, but run into this problem:

2024-01-30T13:51:19.528971112Z level=warn ts=2024-01-30T13:51:19.48553466Z caller=scheduler_processor.go:98 msg="error contacting scheduler" err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.0.0.62:9095: i/o timeout\"" addr=10.0.0.62:9095

do you have any idea what that is or where that comes from?

[mebaysan]

I published an article on Medium. Hopefully, this can help you for containerized setup.

https://mebaysan.medium.com/do-not-fly-blind-setup-grafana-loki-for-caprover-docker-swarm-f60abd75c203