Closed lordofthejars closed 7 years ago
uglyog
commented
8 years ago That would definitely be useful, and not just for the JUnitRunner. I mainly use the gradle plugin, and having a report is something missing because the only way to diagnose a CI build failure is to look through the console logs.
The Ruby version generates output in various formats: Markdown, HTML. We could do something similar, allow different report generators to be defined.
lordofthejars
commented
8 years ago I have been reading this: https://github.com/realestate-com-au/pact/wiki/Autogenerated-documentation The autogenerated documentation is from a pact file (generated by consumer) and not from a producer run. In this case I think it would be great to have both reports.
uglyog
commented
8 years ago Here is the result of the first stab at a JSON report
{
"metaData": {
"date": "2016-04-23T06:14:45+0000",
"pactJvmVersion": "3.2.7",
"reportFormat": "0.0.0"
},
"provider": {
"name": "Activity Service"
},
"execution": [
{
"consumer": {
"name": "sampleconsumer",
"source": {
"file": "/home/ronald/Development/Projects/Pact/pact-gradle-test/src/test/resources/sample-pact.json"
}
},
"interactions": [
{
"interaction": {
"providerState": "many activities exist",
"description": "a request for activities",
"request": {
"method": "GET",
"path": "/activities",
"headers": {
"Accept": "application/json"
},
"body": null
},
"response": {
"status": 201,
"headers": {
"Content-Type": "application/json"
},
"body": {
"activities": [
{
"description": "f_UXcxIXYhgqtxjiPumRiCo9C5JNDX",
"name": "hx55sbvMPk1kF-9"
},
{
"description": "f_UXcxIXYhgqtxjiPumRiCo9C5JNDX",
"name": "hx55sbvMPk1kF-9"
}
]
},
"matchingRules": {
"$.body.activities": {
"min": 2
},
"$.body.activities[*].*": {
"match": "type"
},
"$.body.activities[*].description": {
"match": "type"
},
"$.body.activities[*].name": {
"match": "type"
}
}
}
},
"verification": {
"result": "Failed",
"status": [
"assert expectedStatus == actualStatus",
" | | |",
" 201 | 200",
" false"
],
"body": {
"comparison": {
"$.body.activities": "Expected List(Map(description -> 100, name -> Fred)) to have minimum 2",
"$.body.activities.0.description": "Expected 100 to be the same type as 'f_UXcxIXYhgqtxjiPumRiCo9C5JNDX'"
},
"diff": [
"@3",
" {",
"- \"description\": \"f_UXcxIXYhgqtxjiPumRiCo9C5JNDX\",",
"- \"name\": \"hx55sbvMPk1kF-9\"",
"- },",
"- {",
"- \"description\": \"f_UXcxIXYhgqtxjiPumRiCo9C5JNDX\",",
"- \"name\": \"hx55sbvMPk1kF-9\"",
"+ \"description\": 100,",
"+ \"name\": \"Fred\"",
" }",
""
]
}
}
}
]
},
{
"consumer": {
"name": "sampleconsumer2",
"source": {
"file": "/home/ronald/Development/Projects/Pact/pact-gradle-test/src/test/resources/sample-pact2.json"
}
},
"interactions": [
{
"interaction": {
"providerState": "many activities exist",
"description": "a request for activities",
"request": {
"method": "GET",
"path": "/activities",
"headers": {
"Accept": "application/json"
},
"body": null
},
"response": {
"status": 200,
"headers": {
"Content-Type": "application/json"
},
"body": {
"activities": [
{
"description": "COUxFaKRDRKaBnjg-LuOASNn_Xxy6H",
"name": "V1Zfa-LCfYPEE-o"
},
{
"description": "COUxFaKRDRKaBnjg-LuOASNn_Xxy6H",
"name": "V1Zfa-LCfYPEE-o"
}
]
},
"matchingRules": {
"$.body.activities": {
"min": 2
},
"$.body.activities[*].*": {
"match": "type"
},
"$.body.activities[*].description": {
"match": "regex",
"regex": "(\\w|\\s)+"
},
"$.body.activities[*].name": {
"match": "regex",
"regex": "(\\w|\\s)+"
}
}
}
},
"verification": {
"result": "Failed",
"body": {
"comparison": {
"$.body.activities": "Expected List(Map(description -> 100, name -> Fred)) to have minimum 2"
},
"diff": [
"@3",
" {",
"- \"description\": \"COUxFaKRDRKaBnjg-LuOASNn_Xxy6H\",",
"- \"name\": \"V1Zfa-LCfYPEE-o\"",
"- },",
"- {",
"- \"description\": \"COUxFaKRDRKaBnjg-LuOASNn_Xxy6H\",",
"- \"name\": \"V1Zfa-LCfYPEE-o\"",
"+ \"description\": 100,",
"+ \"name\": \"Fred\"",
" }",
""
]
}
}
}
]
},
{
"consumer": {
"name": "sampleconsumer3",
"source": {
"file": "/home/ronald/Development/Projects/Pact/pact-gradle-test/src/test/resources/sample-pact3.json"
}
},
"interactions": [
{
"interaction": {
"providerState": null,
"description": "add a broker",
"request": {
"method": "GET",
"path": "/api/broker/add",
"query": {
"options": [
"delete.topic.enable=true"
],
"broker": [
"1"
]
},
"body": null
},
"response": {
"status": 200,
"headers": {
"Content-Type": "text/html;charset=UTF-8"
},
"body": "Ok"
}
},
"verification": {
"result": "Failed",
"header": {
"Content-Type": "Expected header 'Content-Type' to have value 'text/html;charset=UTF-8' but was 'text/plain;charset=UTF-8'"
},
"body": {
"comparison": "Expected a response type of 'text/html' but the actual type was 'text/plain'"
}
}
}
]
}
]
}System properties pact.verification.reports and pact.verification.reportDir now control this behaviour.
lordofthejars
commented
8 years ago Looks really really good. Hope that it is merge as soon as possible so I can write the asciidoc backend :)
BTW have you thought about how users could create its own backends? Basically an SPI for hooking and implement your own format.
uglyog
commented
8 years ago Currently, it is a bit messy because the reporters are based on the console output code extracted from the verification process.
But they just represent events that are occurring as the verification runs, so I have been thinking about a simple interface that receives the event and event parameters. This should make it easier for people to add their own implementations. Maybe need another property to add new event handlers at runtime.
uglyog
commented
8 years ago Initial implementation has been released with versions 3.2.7 and 2.4.9
lordofthejars
commented
8 years ago Great!!! now I can fork and implement support for AsciiDoc
When you run pact producer in JUnit, the results are stored in JUnit format. This is good for compatibility with JUnit reporters of the IDE and so on, but in terms of generating a report for putting on your CI environment I think it is not enough ( or we could provide more information). WhatI suggest is to modify the JUnitRunner (since there is where you have all the information) to generate some kind of JSON file that includes the fragment executed and the result, in case of FAIL adding the stacktrace/message. In this way you will have all the information in one place, you will be able to know for example that "system did a request to
http://myhost:8080/mysite/rest/servicewith method GET" and the response was the expected one which is "blabla" or the answer was not the expected because "blablabla".I see something like:
So everything is packed at the same place. After that we will be able in Jenkins/Gradle side to create a fancy HTML file with all the information of the execution.