jgosmann / dmarc-metrics-exporter

Export Prometheus metrics from DMARC reports.
MIT License
48 stars 7 forks source link

.. image:: https://github.com/jgosmann/dmarc-metrics-exporter/actions/workflows/ci.yml/badge.svg :target: https://github.com/jgosmann/dmarc-metrics-exporter/actions/workflows/ci.yml :alt: CI and release pipeline .. image:: https://codecov.io/gh/jgosmann/dmarc-metrics-exporter/branch/main/graph/badge.svg?token=O4M05YWNQK :target: https://codecov.io/gh/jgosmann/dmarc-metrics-exporter :alt: Codecov coverage .. image:: https://img.shields.io/pypi/v/dmarc-metrics-exporter :target: https://pypi.org/project/dmarc-metrics-exporter/ :alt: PyPI .. image:: https://img.shields.io/pypi/pyversions/dmarc-metrics-exporter :target: https://pypi.org/project/dmarc-metrics-exporter/ :alt: PyPI - Python Version .. image:: https://img.shields.io/pypi/l/dmarc-metrics-exporter :target: https://pypi.org/project/dmarc-metrics-exporter/ :alt: PyPI - License

dmarcs-metrics-exporter

Export metrics derived from DMARC aggregate reports to Prometheus. This exporter regularly polls for new aggregate report emails via IMAP. The following metrics will be collected and exposed at an HTTP endpoint for Prometheus:

Each of these metrics is subdivided by the following labels:

In addition, there is a dmarc_invalid_reports_total metric with a count of DMARC report emails from which no report could be parsed. It is subdivided by a single from_email label.

Installation

This describes the manual setup fo dmarc-metrics-exporter. An Ansible role for automated deployment is provided in roles. Further instructions for Ansible are given in the readme file provided in that directory.

It is best to run dmarc-metrics-exporter under a separate system user account. Create one for example with

.. code-block:: bash

adduser --system --group dmarc-metrics

Then you can install dmarc-metrics-exporter with pip from PyPI for that user:

.. code-block:: bash

sudo -u dmarc-metrics pip3 install dmarc-metrics-exporter

You will need a location to store the metrics.db that is writable by that user, for example:

.. code-block:: bash

mkdir /var/lib/dmarc-metrics-exporter
chown dmarc-metrics:dmarc-metrics /var/lib/dmarc-metrics-exporter

Configuration

To run dmarc-metrics-exporter a configuration file in JSON format is required. The default location is /etc/dmarc-metrics-exporter.json.

Because the configuration file will contain the IMAP password, make sure to ensure proper permissions on it, for example:

.. code-block:: bash

chown root:dmarc-metrics /etc/dmarc-metrics-exporter.json
chmod 640 /etc/dmarc-metrics-exporter.json

An example configuration file is provided in this repository in config/dmarc-metrics-exporter.sample.json.

The following configuration options are available:

Logging configuration ^^^^^^^^^^^^^^^^^^^^^

When providing a custom logging configuration, it must follow the dictionary schema (version 1) described in the logging.config documentation <https://docs.python.org/3/library/logging.config.html#configuration-dictionary-schema>_. In general, a provided top-level key will replace the default configuration, but there are some exceptions. The following keys are always fixed:

In addition, the root key has some special handling. If it is overridden, but not handlers key is provided, handlers: ['default'] will be inserted automatically. Also, the level key will be set to 'DEBUG' if the application is started with the --debug flag.

Configuring log level """""""""""""""""""""

To change the log level globally:

.. code-block:: json

{
    "logging": {
        "root": {
            "level": "WARNING"
        }
    }
}

Configuring logging format """"""""""""""""""""""""""

To change the logging format:

.. code-block:: json

{
    "logging": {
        "handlers": {
            "default": {
                "class": "logging.StreamHandler",
                "formatter": "json"
            }
        }
    }
}

Valid formats are plain, colored, and json.

Disabling Uvicorn access logs """""""""""""""""""""""""""""

To disable the Uvicorn access logs:

.. code-block:: json

{
    "logging": {
        "loggers": {
            "uvicorn.access": {
                "propagate": false
            }
        }
    }
}

Usage

To run dmarc-metrics-exporter with the default configuration in /etc/dmarc-metrics-exporter.json:

.. code-block:: bash

sudo -u dmarc-metrics python3 -m dmarc_metrics_exporter

To use a different configuration file:

.. code-block:: bash

sudo -u dmarc-metrics python3 -m dmarc_metrics_exporter --configuration <path>

You can enable debug logging with the --debug if you do not want to provide your own logging configuration:

.. code-block:: bash

sudo -u dmarc-metrics python3 -m dmarc_metrics_exporter --debug

systemd ^^^^^^^

Instead of manually starting the dmarc-metrics-exporter, you likely want to have it run as a system service. An example systemd service file is provided in this repository in config/dmarc-metrics-exporter.service. Make sure that the paths and user/group names match your configuration and copy it to /etc/systemd/system to use it. To have systemd pick it up a systemctl daemon-reload might be necessary.

You can than start/stop dmarc-metrics-exorter with:

.. code-block:: bash

systemctl start dmarc-metrics-exporter
systemctl stop dmarc-metrics-exporter

To have dmarc-metrics-exporter start on system boot:

.. code-block:: bash

systemctl enable dmarc-metrics-exporter

Docker ^^^^^^

A new docker image is build for each release with GitHub Actions as described in this yaml-file: .github/workflows/docker-publish.yml.

Note that you should configure the listen_addr to 0.0.0.0 to be able to access the metrics exporter from outside the container.

Example docker-compose file:

.. code-block:: yml

version: "3"

services:

  dmarc-metrics-exporter:
    # source: https://github.com/jamborjan/dmarc-metrics-exporter/pkgs/container/dmarc-metrics-exporter
    container_name: dmarc-metrics-exporter
    hostname: dmarc-metrics-exporter
    image: jgosmann/dmarc-metrics-exporter:1.2.0
    restart: unless-stopped
    user: 1000:1000 #PUID=1000:PGID=1000
    expose:
      - 9797
    volumes:
      - '/host/folder/dmarc-metrics-exporter.json:/etc/dmarc-metrics-exporter.json'
      - '/host/folder/dmarc-metrics-exporter/metrics:/var/lib/dmarc-metrics-exporter:rw'
    logging:
      driver: "json-file"
      options:
        tag: "{{.ImageName}}|{{.Name}}|{{.ImageFullID}}|{{.FullID}}"
    networks:
      - YourDockerLan

# $ docker network create -d bridge --attachable YourDockerLan
networks:
  YourDockerLan:
    external:
      name: YourDockerLan

Prometheus ^^^^^^^^^^

Example prometheus config file:

.. code-block:: yml

global:
  scrape_interval: 15s
  evaluation_interval: 15s

rule_files:

scrape_configs:

  - job_name: 'dmarc-metrics-exporter'
    static_configs:
      - targets: ['dmarc-metrics-exporter:9797']

Grafana ^^^^^^^

An example configuration file is provided in this repository in config/dmarc-metrics-exporter.grafana.sample.json. This example dashboard displays the collected metrics as shown in the screenshot below.

.. figure:: config/dmarc-metrics-exporter.grafana.sample.png

Example grafana dashboard

Hints ^^^^^

You should not use your normal email and password credentials for the dmarc-metrics-exporter. If you are not able to create a dedicated service account email account, you should use an app password.

Microsoft Exchange Online """""""""""""""""""""""""

Development

Prerequisites ^^^^^^^^^^^^^

Setup development environment ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

.. code-block:: bash

pre-commit install
poetry install

Run tests ^^^^^^^^^

.. code-block:: bash

docker-compose up -d
poetry run pytest