search

BonnyCI / projman

A project management repository -- meta
0 stars 2 forks source link

Bonny monitors should be better tested before merging #223

Closed rattboi closed 7 years ago

rattboi commented 7 years ago

Currently, datadog monitors are run against a schema check that exists within datadog-builder, which finds obvious issues, but can't find issues such as illegal references in the query string, as this is dependent on the configuration of the datadog instance, such as which integrations we are using (like zuul, nodepool, gearman, etc).

rattboi commented 7 years ago

I propose we can fix this by extending datadog-builder to individually push monitors one at a time, making sure they pass datadog's blackbox validation, then immediately removing them. Since we don't have an accurate model of what is tested by datadog, this seems the most straightforward test.

Datadog, on POST of a bad monitor, will send a 400 Bad Request Client Error, so we can capture those and report on them.

Note that this has the side effect of finding older monitors that are broken because of changes to our datadog instance.

For now, we'd have to validate these against our production datadog instance. Assuming we get a QA environment running, we'd then set up a QA datadog instance and point testing at that environment instead.