search

TritonDataCenter / triton-prometheus

Prometheus for TritonDC
Mozilla Public License 2.0
5 stars 2 forks source link

readme

triton-prometheus

This repository is part of the Joyent Triton and Manta projects. For contribution guidelines, issues, and general documentation, visit the main Triton and Manta project pages.

The Triton and Manta core Prometheus service. Triton and Manta use Prometheus and Grafana to track their own metrics and monitor themselves. All metrics are gathered via CMON.

This repo builds one zone image which can be deployed as an instance of a Triton or Manta service. The targets Prometheus scrapes will depend on whether the instance is deployed as part of Triton or Manta.

Active Branches

There are currently two active branches of this repository, for the two active major versions of Manta. See the mantav2 overview document for details on major Manta versions.

Status

Joyent is actively developing Prometheus and Grafana services for use in Triton and Manta. RFD 150 describes the current plan and status.

Setup for Triton

First ensure that CMON and CNS are set up in your TritonDC, typically via:

sdcadm post-setup cns [OPTIONS]
sdcadm post-setup cmon [OPTIONS]

Then run the following from your TritonDC's headnode global zone:

sdcadm post-setup prometheus [OPTIONS]

Setup for Manta

As with Triton above, ensure that CMON and CNS are deployed. Then, create a new Manta config file with your desired number of Prometheus instances and update using manta-adm, as described in the Manta Operator's Guide

Configuration

Primarily this VM runs a Prometheus server (as the "prometheus" SMF service). The config files for that service are as follows. Note that "/data/..." is a delegate dataset to persist through reprovisions -- the Prometheus time-series database is stored here.

/opt/triton/prometheus/etc/config.json        # SAPI config
/opt/triton/prometheus/etc/prometheus.yml     # Prometheus config; generated
                                              # from SAPI config
/opt/triton/prometheus/keys/*                 # Key and cert for CMON auth

/data/prometheus/data/*                       # Prometheus database

Like most Triton and Manta core services, a config-agent is used to gather some config data. Unlike many core services, this VM uses an additional config processing step to create the final prometheus config. At the time of writing, this is to allow post-processing of the SAPI config variables to produce the Prometheus config file. This code is pulled out into /opt/triton/prometheus/bin/prometheus-configure, which runs from boot/setup.sh, boot/configure.sh, and as the config-agent post_cmd.

SAPI Configuration

There are some Prometheus service configuration options that can be set in SAPI. Note that the default values listed here are the values that the instance's setup scripts will use if the SAPI variables are unset. They may be overridden using application- and size-specific defaults if a Prometheus instance is deployed using sdcadm or manta-adm.

Key Type Description
cmon_domain String Optional. The domain at which Prometheus should talk to this DC's CMON, e.g. "cmon.us-east-1.triton.zone". The actual endpoint is assumed to be https and port 9163. See notes below.
cmon_enforce_certificate Bool Optional. This can be set to true to have Prometheus fail on TLS cert errors from a self-signed cert. This is false by default.
scrape_interval Integer Optional. The interval, in seconds, at which Prometheus should scrape its targets. Defaults to 10.
scrape_timeout Integer Optional. The amount of time, in seconds, that is allotted for each scrape to complete. Defaults to 10.
evaluation_interval Integer Optional. The interval, in seconds, at which Prometheus should evaluate its alerting rules. Defaults to 10.

Prometheus gets its metrics from the DC's local CMON, typically over the external network. To auth with CMON properly in a production environment, Prometheus needs to know the appropriate CMON URL advertised to public DNS for which it has a signed TLS certificate. This is what cmon_domain is for. If this variable is not set, then this image will attempt to infer an appropriate URL via querying the DC's local CNS. See bin/prometheus-configure for details.

An example setting some of these values:

promSvc=$(sdc-sapi /services?name=prometheus | json -Ha uuid)
sdc-sapi /services/$promSvc -X PUT \
    -d '{"metadata": {"cmon_domain": "mycmon.example.com", "cmon_enforce_certificate": true}}'

CMON Auth

Prometheus needs to authenticate with the local CMON. To do this, the setup script generates a certificate using the bin/certgen tool. This certificate can be regenerated by running the tool manually.

Name resolution

The Prometheus zone resolves names using CNS. To allow queries from an arbitrary number of CNS servers and offload some traffic from CNS, the Prometheus zone runs a BIND server that resolves the CNS names using zone transfer. The BIND server forwards all requests outside of the CNS zone to external and binder resolvers. BIND runs on localhost, and localhost is the only entry in /etc/resolv.conf.

Security

Prometheus listens on the admin and external networks. The firewall with the standard Triton rules is enabled to disallow incoming requests on the external network.

Prometheus is on the external network so it can access CMON and work with CNS -- at least until CNS supports split horizon DNS to provide separate records on the admin network. This is because CMON's Triton service discovery returns the CNS domain names for Triton's core VMs.

Prometheus is on the admin network because Triton Grafana instances access Triton and Manta Prometheus instances on the admin network.

Troubleshooting

Prometheus doesn't have Triton/Manta data

Prometheus gets its Triton (or Manta) data from CMON. Here are some things to check if this appears to be failing:

LX script

This repo also contains a script to set up an ad-hoc LX-branded zone running Prometheus. One might find this script useful for setting up a Prometheus instance for experimentation -- the user will be able to safely edit configuration files manually without config-agent overwriting them. This script can also be used in environments where the relevant Triton components (sdcadm, CMON, manta-deployment) haven't been updated to versions that support the native Prometheus service. Here's how to run the script:

Run ./setup-prometheus-adhoc.sh from a Triton headnode. All CLI flags are optional; here's an explanation of what each flag does:

An appropriate invocation for a development setup would be:

./setup-prometheus-adhoc.sh \
-i \
-r <CNS IP>,8.8.8.8 \
-k /root/.ssh/sdc.id_rsa.pub

An appropriate invocation for a production environment would be:

./setup-prometheus-adhoc.sh \
-f \
-r <CNS IP>,8.8.8.8 \
-s <server UUID> \
-k /root/.ssh/sdc.id_rsa.pub