Sending Metrics to Grafana Cloud Graphite

[josephcy95]

Feature/improvement

I've finally able to send metrics to the free Grafana Cloud Graphite after a few attempt, and this is a short walkthrough of the how to do it.

1. Install carbon-relay-ng based on your distribution, this doesn't work with ARM arch, unfortunately I couldn't find ARM compatible nor able to build it.

$ wget --content-disposition https://packagecloud.io/raintank/raintank/packages/ubuntu/xenial/carbon-relay-ng_1.2-1_amd64.deb/download.deb
$ sudo apt install ./carbon-relay-ng_1.2-1_amd64.deb

2. Login your grafana cloud and click "send metrics" on graphite, in that page you should find the config for your carbon-relay-ng, generate and replace the user and api key in below config, save the config in /etc/carbon-relay-ng/carbon-relay-ng.conf

## Global settings ##
# instance id's distinguish stats of multiple relays.
# do not run multiple relays with the same instance id.
# supported variables:
#  ${HOST} : hostname
instance = "${HOST}"

## System ##
# this setting can be used to override the default GOMAXPROCS logic
# it is ignored if the GOMAXPROCS environment variable is set
max_procs = 2
pid_file = "carbon-relay-ng.pid"
# directory for spool files
spool_dir = "spool"

## Logging ##
# one of trace debug info warn error fatal panic
# see docs/logging.md for level descriptions
# note: if you used to use "notice", you should now use "info".
log_level = "info"

## Inputs ##
### plaintext Carbon ###
listen_addr = "0.0.0.0:2003"
# close inbound plaintext connections if they've been idle for this long ("0s" to disable)
plain_read_timeout = "2m"
### Pickle Carbon ###
pickle_addr = "0.0.0.0:2013"
# close inbound pickle connections if they've been idle for this long ("0s" to disable)
pickle_read_timeout = "2m"

## Validation of inputs ##
# you can also validate that each series has increasing timestamps
validate_order = false

# How long to keep track of invalid metrics seen
# Useful time units are "s", "m", "h"
bad_metrics_max_age = "24h"

[[route]]
key = 'grafanaNet'
type = 'grafanaNet'
addr = 'https://graphite-prod-10-prod-us-central-0.grafana.net/graphite/metrics'
apikey = '662532:<Your Grafana.com API Key>'
schemasFile = '/etc/carbon-relay-ng/storage-schemas.conf'
aggregationFile = '/etc/carbon-relay-ng/storage-aggregation.conf' # optional

## Instrumentation ##
[instrumentation]
# in addition to serving internal metrics via expvar, you can send them to graphite/carbon
# IMPORTANT: setting this to "" will disable flushing, and metrics will pile up and lead to OOM
# see https://github.com/grafana/carbon-relay-ng/issues/50
# so for now you MUST send them somewhere. sorry.
# (Also, the interval here must correspond to your setting in storage-schemas.conf if you use Grafana Cloud)
graphite_addr = "localhost:2003"
graphite_interval = 10000  # in ms

3. You also need to save below config as /etc/carbon-relay-ng/storage-schemas.conf which define your metrics retention period, make changes based on your needs as well.

[default]
  pattern = .*
  retentions = 10s:1d

[sitespeed]
pattern = ^sitespeed_io\.
retentions = 10m:40d

4. Now you can start your carbon-relay-ng

$ /usr/bin/carbon-relay-ng /etc/carbon-relay-ng/carbon-relay-ng.conf

5. I also use systemd to make sure my carbon-relay-ng start when server reboot

[Unit]
Description=carbon-relay-ng
After=network.target

[Service]
Type=simple
WorkingDirectory=/etc/carbon-relay-ng
PIDFile=/etc/carbon-relay-ng/carbon-relay-ng.pid
ExecStart=/usr/bin/carbon-relay-ng /etc/carbon-relay-ng/carbon-relay-ng.conf
ExecReload=/bin/kill -USR2 $MAINPID
Restart=always
SyslogIdentifier=carbon-relay-ng

[Install]
WantedBy=multi-user.target

6. Now just run sitespeed to send data to carbon-relay-ng, make sure to set the --graphite.host to your host IP, which is usually 172.17.0.1 in linux, you can find it with ifconfig, not localhost which will point to localhost inside the container.

docker run --rm -v "$(pwd):/sitespeed.io" sitespeedio/sitespeed.io:25.5.1 --graphite.host=172.17.0.1 https://www.sitespeed.io/ --slug yourTestName --graphite.addSlugToKey true

Note

By default the free cloud graphite db only alllow to store up to 10k metrics, so you would probably need to reduce the metrics, I think you can run it once with -vvv and see what metric is being sent, then use --metrics.filter to remove metrics you doesn't need. @soulgalore also told me to use —graphite.skipSummary to reduce metrics by not sending any metrics grouped by domain.

Btw I did not end up using the free grafana cloud graphite because I haven't spend time on reducing the metrics, instead I just run graphite on the same server I run sitespeed, I'm using the oracle cloud free ARM vm, and for monitoring only a few pages it works fine for me. If you want to do the same I also rebuild an image of graphite that works on ARM

docker pull josephcy95/graphite:1.1.7-9

[soulgalore]

Thank you @josephcy95 !

[tprocter1984]

Thanks @josephcy95 - This could be the way forward instead of setting up local alerting. I am going to try this today

[tprocter1984]

Hey @josephcy95

Tripped on the first hurdle, the following instruction: Login your grafana cloud and click "send metrics" on graphite

I'm in my cloud account, but unsure where the "sure metrics" option is, can you clarify?

[soulgalore]

Have anyone been able to try this out? I haven't done this yet.