[SPIKE] - promtail/loki setup

[tarnas14]

extracted from #11

give it a day to try and setup promtail/loki pipeline for log collection and querying

• dismissing docker driver, because it's not yet supported on windows: https://grafana.com/docs/loki/latest/clients/docker-driver/#docker-driver-client

[tarnas14]

promtail/loki: (not recommended)

it works, but promtail accepts only static configurations - it won't discover new containers on a local environment we cannot just listen for /var/lib/docker/containers/*/*log because there can possibly be other containers than those with tari

to counteract this we would probably have to generate promtail scrape configs dynamically from backend in the file when containers list changes and restart promtail container to pick up configuration changes this means that every time we create a new container on backend we would add to promtail config something like:

- job_name: container_<service>_<id>
  static_configs:
  - targets:
      - localhost
    labels:
      job: container_<service>_<id>_logs
      __path__: /var/lib/docker/containers/<container-id>/<container-id>-json.log

this might work, but is very clunky

logspout/logstash/loki: (looks promising)

logspout is a docker log router, which allows us to listen for specific containers based on docker labels (and we control the label) logspout is dumb, it only routes logs, so we need something to process them, the best integration is with logstash logstash is needed to parse specific services logs, because they are not standardized - this way we can assure all logs have fields: datetime, localisation, level, message so we add logstash, set input from logspout, configure filters, set output to our loki instance, loki will give us query logic through https://grafana.com/docs/loki/latest/api/#get-lokiapiv1query_range we can then add a command to tari backend which will take a LogQL query from frontend, query loki and return results

LIMITATION: logstash/logspout/loki containers need to be up when we start new containers, so it should be part of the startup LIMITATION: for loki to be available for querying (either HTTP or LogCLI), we need to expose port to host machine

setup: #209

[tarnas14]

missclick...

[CjS77]

Slight clarification. The logs are not stored with docker metadata, but in a path of the user's choosing.

Although it's not actually configurable from the front-end at the moment and defaults to ~/Library/Caches/tari/tmp/dibbler/{service}/log/ on MacOS.

The issue you identified remains though, because the root path (~/Library/Caches/tari/tmp/) should be configurable in the long term

[CjS77]

We don't worry about the docker stdout logs so much. It's the Tari logs that are important

[CjS77]

The log folders themselves are highly configurable, but by default, they log at DEBUG level, and are rolling 10MB log files with a 5 file history.

Core messages are stored in core.log, networking messages in network.log and any other messages in other.log

 {contrib_guidelines} ~/tari/tari$ ll ~/Library/Caches/tari/tmp/dibbler/base_node/log/
drwxr-xr-x cayle staff   192 B  Tue May 10 13:24:11 2022   ./         
drwxrwxrwx cayle staff   256 B  Tue May  3 09:02:18 2022   ../        
.rw-r--r-- cayle staff    10 MB Tue May 10 13:24:05 2022   core.1.log 
.rw-r--r-- cayle staff   2.5 MB Wed May 11 21:39:03 2022   core.log   
.rw-r--r-- cayle staff   1.5 MB Wed May 11 21:38:41 2022   network.log
.rw-r--r-- cayle staff 251.4 KB Sat May  7 12:36:45 2022   other.log  

[CjS77]

Sample from core.log:

2022-05-10 12:50:12.159659700 [base_node::app::status] INFO  12:50 v0.31.1, dibbler, State: Listening, Tip: 46665 (Sat, 07 May 2022 11:03:38 +0000), Mempool: 0tx (0g, +/- 0blks), Connections: 0, Banned: 0, Messages (last 60s): 0, Rpc: 0/100, RandomX: #1 with flags FLAG_HARD_AES | FLAG_JIT | FLAG_ARGON2_SSSE3 | FLAG_ARGON2_AVX2 | FLAG_ARGON2
2022-05-10 12:50:17.185169800 [base_node::app::status] INFO  12:50 v0.31.1, dibbler, State: Listening, Tip: 46665 (Sat, 07 May 2022 11:03:38 +0000), Mempool: 0tx (0g, +/- 0blks), Connections: 0, Banned: 0, Messages (last 60s): 0, Rpc: 0/100
2022-05-10 12:50:22.208164300 [base_node::app::status] INFO  12:50 v0.31.1, dibbler, State: Listening, Tip: 46665 (Sat, 07 May 2022 11:03:38 +0000), Mempool: 0tx (0g, +/- 0blks), Connections: 0, Banned: 0, Messages (last 60s): 0, Rpc: 0/100
2022-05-10 12:50:27.235026500 [base_node::app::status] INFO  12:50 v0.31.1, dibbler, State: Listening, Tip: 46665 (Sat, 07 May 2022 11:03:38 +0000), Mempool: 0tx (0g, +/- 0blks), Connections: 0, Banned: 0, Messages (last 60s): 0, Rpc: 0/100

[tarnas14]

we will be putting promtail/loki/grafana for log querying, this issue has served its purpose