Monitor mlab-ns rate limit decisions

[stephen-soltesz]

Stackdriver custom metrics can extract values from a log line:

• for log lines, refer to the "field name": protoPayload.line.logMessage
• create a regex that will extract those values on match.

Examples: https://stackoverflow.com/questions/47355723/google-stackdriver-log-based-metrics-how-to-extract-values-using-a-regular-expr https://groups.google.com/forum/#!topic/google-appengine/DB4aZG0Wg8k

[stephen-soltesz]

The REST API supports all parameters needed to specify and extract label values based on regex: https://cloud.google.com/logging/docs/reference/v2/rest/v2/projects.metrics

However, neither the python nor Go package libraries support the advanced custom metric parameters needed to specify a regex.

https://godoc.org/cloud.google.com/go/logging/logadmin#Metric https://googlecloudplatform.github.io/google-cloud-python/latest/logging/usage.html

[stephen-soltesz]

Once a custom metric is defined for a project, the stackdriver_exporter can export the values easily:

~/bin/stackdriver_exporter --monitoring.metrics-offset=5m \
    --monitoring.metrics-interval=1m --google.project-id=mlab-ns \
    --monitoring.metrics-type-prefixes=logging.googleapis.com/user/<custom-metric-name>

[stephen-soltesz]

An auto-generated Go package does support the advanced options:

logpb "google.golang.org/genproto/googleapis/logging/v2"

https://godoc.org/google.golang.org/genproto/googleapis/logging/v2#LogMetric

[stephen-soltesz]

Or simpler, the gcloud beta includes support for creating a metric from a spec (json or yaml):

$ cat metric.yaml 
name: soltesz-test-cli2
description: Test creating rule with regex from cli
filter: >
  resource.type="gae_app"
  resource.labels.module_id="default"
  logName="projects/mlab-ns/logs/appengine.googleapis.com%2Frequest_log"
  protoPayload.line.logMessage:"[lookup]None"

metricDescriptor:
  name: projects/mlab-ns/metricDescriptors/logging.googleapis.com/user/soltesz-test-cli2
  labels:
   - key: site
  metricKind: "DELTA"
  valueType: "INT64"
  unit: "1"
  description: "use regex"
  type: logging.googleapis.com/user/soltesz-test-cli2

labelExtractors:
  site: 'REGEXP_EXTRACT(protoPayload.line.logMessage, ".*([a-z]{3}[0-9ct]{2}).*")'

$ gcloud beta logging metrics create soltesz-test-cli2 --config-from-file=metric.yaml
INFO: Refreshing access_token
Created [soltesz-test-cli2].
INFO: Display format: "none"