Pitometer is part of keptn. This Pitometer keptn UI repository is the first version of on out-of-the-box solution to enforce automated quality gates in your CI/CD Pipeline. It will work either with keptn as your deployment orchestrator or any other CI/CD tool such as Jenkins, Bamboo, Azure DevOps, Tekton, Argo CD, AWS CodePipeline or others.
Keptn Pitometer Standalone UI runs as a service, stores result in a mongodb, provides an API and a simple WebUI to trigger the following actions
We are planning to integrate this Pitometer UI as part of the keptn's bridge UI. Once available you can leverage Pitometer by simply installing keptn!
docker run -d -p 27017:27017 mongo
npm install --only=prod
npm start
When you run Pitometer evaluations you always need to specify the following input data:
Now - lets go on with the use cases
If you want to run a single test you can specify "Starttime", "Endtime" and "Test Run Name" and click on "Single Run". This will evaluate the perfspec file against the specified timeframe and will store the results in your test context with the name you specified as test run name, e.g: "TestRun_1"!
If everything goes according to plan you should see the output in the bottom half of the page saying: "Finished Run!".
If you want to look at the result you can generate a report for just that test run:
If you want to fill up your Pitometer database with a couple of results do this in the section for "Multi Run". Here are the fields you have to fill out and the meaning behind them
After clicking on "Multi Run" Pitometer will start running all your evaluations sequentially. It will always re-evaluate with which test run it should compare the current evaluation with depending on what you have specified in 'Compare With'
If everything goes according to plan you should see the output in the bottom half of the page saying: "Finished Run!".
If you want to see the results you can execute a report that shows you, e.g: the last 6 runs. More details on how to generate reports for several runs follow in the next section!
If you want to see a report of the Pitometer data in your database simply click on 'Generate Report'. In my example I get a report across the last 7 test runs (one for my initial single run + 6 for my multi runs) and it looks like this. You have the following options for the 'Query for Report' field:
This report starts with the "Score" that pitometer calculated for each test run. The color indicates whether the score exceeded the pass (=green), warning (=orange) or failed (=red) objective! Following the score are the individual results for each Indicator that is specified in the perfspec.json file!
If you want to clear the database simply click on 'Clear Database'! This will remove the data stored for the 'Test context' specified in the top input field.
This standalone Pitometer UI not only provides the Web UI when browsing to /. It also provides individual API endpoints you can call directly if you wish. Here are the individual endpoints that the Web UI is calling when clicking on the 3 offered buttons:
Clearing the Database:
HTTP GET /api/cleardb?context=mytestcontext
Running Single Pitometer Evaluation:
HTTP GET /api/singlerun?context=mytestcontext&perfspec=./samples/perfspecfile.json&start=2019-07-01T08:00:00&end=2019-07-01T08:01:00&testRunName=TestRun_1&tags=[%22environment:prod-keptnsample%22,%22service:simplenodeservice%22]&comparecontext=
Running Multiple Pitometer Evaluations:
HTTP GET /api/multirun?context=mytestcontext&perfspec=./samples/perfspecfile.json&start=2019-07-01T08:00:00&length=60000&count=6&testRunPrefix=TestRun_&testRunIx=1&tags=[%22environment:prod-keptnsample%22,%22service:simplenodeservice%22]&comparecontext=
Getting Pitometer Report:
HTTP GET /api/report?context=mytestcontext&count=7
You need to create the following config files
secrects.json:
{
"DynatraceToken" : "YOUR_TOKEN",
"DynatraceUrl" : "https://XXXXX.live.dynatrace.com",
"PrometheusQueryUrl : "<prometheus query url>,
"NeoloadToken" : "YOUR_NEOLOAD_TOKEN",
"NeoloadAPIURL" : "https://neoload-api.saas.neotys.com",
"NeoloadWebUploadURL" : "https://neoload-files.saas.neotys.com"
}
config.json:
{
"port" : 8000,
"mongodb" : "mongodb://MONGOSERVER:MONGOPORT/"
}
secrets.json: allows you to configure the Pitometer Data Sources and you need to have at least one configured, e.g: Prometheus. For Neoload the only mandatory field is NeoloadToken - the other two are optional and shows values above are defaults For Dynatrace you need both Url and Token!
config.json: tells Pitometer how to connect to mongodb and the port the Web UI will be exposed. MongoDB is NOT MANDATORY. If you leave it out Pitometer will still be able to execute validations with static thresholds
When specifying the "Compare With" field you have the following options
This is a very simplified query language allowing you to specify a JSON object like the following example which returns 5 test results that have a result of 'pass' with a descending timestamp:
{
"find": { "result" : "pass"},
"sort": { "timestamp" : -1},
"limit" : 5
}
As you have probably guessed. 'find' is like a WHERE clause, 'sort' is like an ORDER BY and 'limit' defines the number of results you want to have. Here are the possible fields for both 'find' and 'sort':
In case you don't get a positive response when running a single or multiple tests then please validate the following