search

postmanlabs / newman

Newman is a command-line collection runner for Postman
https://www.postman.com
Apache License 2.0
6.88k stars 1.17k forks source link

Did anyone export metrics from newman docker to prometheus? #2557

Open alasgarov83 opened 3 years ago

alasgarov83 commented 3 years ago

I have created a custom docker image for newman and packed into it required collections and env. files. I'm running it as a openshift pod. It runs correctly and brings an output inside container (e.g. with junit runner xml output). Afterwards container gets into completed mode.
I would like to push this metrics into the prometheus-pushgateway and let prometheus scrap it. But the format of existing runners isn't really suitable for prometheus. Did anyone already do anything like that by any chance? I can use a bit of advice if anyone has one. Many thanks in advance!

alasgarov83 commented 3 years ago

I've asked a wider community also here: https://stackoverflow.com/questions/64856126/newman-postman-results-to-prometheus-for-automated-tests

dimikirschner commented 3 years ago

@alasgarov83 , could you please provide an example of the newman-report.xml. What exactly the issue with the parsing?

alasgarov83 commented 3 years ago

@dimikirschner thanks for replying

I was trying to generate reports with junit 'reporter' see sample output below: (the test itself is faulty, due to an expired id I was testing with, thus tests=0, but it's not important, this is just an example)

cat newman-run-report-2020-11-16-14-22-32-860-0.xml

<?xml version="1.0" encoding="UTF-8"?>
<testsuites name="xxxxxxxxxxxxxxxxxxxxxxxx" tests="2" time="2.973">
  <testsuite name="api/v1 / query the the xxxxxxxxxxxx for given GUID and register for notification" id="xxxxxxxxxxxxxx
3c0ab382bc" timestamp="2020-11-16T14:22:29.726Z" tests="0" failures="0" errors="0" time="2.848"/>
  <testsuite name="api/v1 / create  results" id="xxxxxxxxxxxxxxxxxxx" timestamp="2020-11-16T14:22:29.726Z" tes
ts="0" failures="0" errors="0" time="0.125"/>

Not sure what would be the smart way of passing labels and metrics to prometheus pushgateway in this case. Or shall I perhaps try it out with the json reporter and rely on response codes i.e. 200/400 etc which I can get into the pushgateway somehow by means of 'curl' for example?

I was hoping there is someone who has done something similar already.

splitt3r commented 3 years ago

What about writing a small reporter https://github.com/postmanlabs/newman#creating-your-own-reporter to push your results to the Prometheus Push Gateway? I think that would be a relatively easy job and a great addition to Newman.

Btw. there is already:

influxdb - This reporter sends the test results information to InfluxDB which can be used from Grafana to build dashboards

thim81 commented 2 years ago

Ben Colemont did some excellent work on this topic: https://github.com/benc-uk/postman-prometheus