facebookincubator / dynolog

Dynolog is a telemetry daemon for performance monitoring and tracing. It exports metrics from different components in the system like the linux kernel, CPU, disks, Intel PT, GPUs etc. Dynolog also integrates with pytorch and can trigger traces for distributed training applications.
MIT License
260 stars 38 forks source link

basic prototype for prometheus logging #146

Closed briancoutinho closed 1 year ago

briancoutinho commented 1 year ago

Prometheus support

148

Enable logging to prometheus- this will be used in environments with their prometheus logging set up already. We leverage the library https://github.com/jupp0r/prometheus-cpp/ for this work.

Parts

  1. Add a Metrics list registry in the code.
  2. Use Prometheus-cpp to set up an HTTP endpoint for prometheus.

Output

Ran on my laptop using a docker container.

How to Run

Please install Docker desktop.

  1. Download this docker file in a directory https://gist.github.com/briancoutinho/d90e1c5d3fca59274fcd14f8863d6409
  2. Build using docker build . -t prometheus:v1
  3. Run docker container forwarding port and mounting dynolog open source repo docker run -p 9090:9090 -it -v ~/Work/dynolog_oss/dynolog:/workspace/dynolog prometheus:v1 /bin/bash
  4. Build dynolog ./scripts/build.sh

To get the logging setup add the following in prometheus.yml

  - job_name: "dynolog"
    static_configs:
      - targets: ["localhost:8080"]

Then run prometheus and dynolog.

cd /workspace/prometheus/prometheus-2.44.0.linux-amd64;
./prometheus --config.file prometheus.yml &
cd -
./build/bin/dynolog -kernel_monitor_reporting_interval_s 10 -use_JSON -use_prometheus &

Open https://localhost:9090/

Screenshot 2023-06-09 at 4 26 52 PM Screenshot 2023-06-09 at 4 26 59 PM

facebook-github-bot commented 1 year ago

@briancoutinho has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

facebook-github-bot commented 1 year ago

This pull request was exported from Phabricator. Differential Revision: D46710501

facebook-github-bot commented 1 year ago

This pull request was exported from Phabricator. Differential Revision: D46710501

facebook-github-bot commented 1 year ago

@briancoutinho merged this pull request in facebookincubator/dynolog@1fa5af75c52aab0ff685b85b6fb32626164b8aab.