Open agross opened 1 year ago
The expected behavior is that the provider will receive the passed example value of 10:11:12:
No, it isn't.
Matchers are for loosening the need for the provider to expect the exact same data used in the consumer tests. Have a look at the documentation on matchers
I'd say the purpose of MatchersV3
is twofold:
generator
elements in the Pact JSON when given no example value. When using such Matchers in the request
part of the Pact the Pact Verifier will then use the generator to craft a value while building the request to verify the provider.This issue is about the second case. MatchersV3.datetime
will not include the generator
bits when being supplied with an example value, but MatchersV3.time
and MatchersV3.date
will.
I've created an example to illustrate the use-case. The important interaction is "A sample request with Matcher on request query string": it uses Matcher.time
for the t
query string, such that the stub server used by the frontend team accepts any time, not just 10:11:12
:
$ http GET 'localhost:8080/test-query?t=10:11:12'
HTTP/1.1 200 OK
access-control-allow-origin: *
content-length: 33
content-type: application/json
date: Sun, 19 Mar 2023 08:11:10 GMT
{
"answer": "42",
"time": "10:11:12"
}
$ http GET 'localhost:8080/test-query?t=11:11:12'
HTTP/1.1 200 OK
access-control-allow-origin: *
content-length: 33
content-type: application/json
date: Sun, 19 Mar 2023 08:11:13 GMT
{
"answer": "42",
"time": "10:11:12"
}
The provider will be verified with a value of t
that is generated by the Pact Verifier on the fly, because there is this generator
piece in the Pact JSON:
{
"consumer": { ... },
"interactions": [
{
"description": "A request where Matcher is used on request",
"request": {
"generators": {
"query": {
"$.t[0]": {
"format": "HH:mm:ss",
"type": "Time"
}
}
},
"method": "GET",
"path": "/test-query",
"query": {
"t": [
"10:11:12"
]
}
},
...
By default that is the value of "now". While running the provider verification with debug logging:
verify_interaction{interaction="A request where Matcher is used on request"}: pact_matching: Applying query generators...
verify_interaction{interaction="A request where Matcher is used on request"}: pact_models::generators: Generator = Time(Some("HH:mm:ss"), None), Generated value = Ok("09:17:41")
See that the generated value is now (09:17:41
) instead of the provided example value of 10:11:12
. This is where MatchersV3.datetime
differs: The provided example value would be passed instead of a generated one.
Example Pact:
Interesting, this bit in the the JSON will compute now + 2 hours for provider verification:
"generators": {
"query": {
"$.t[0]": {
"format": "HH:mm:ss",
"expression": "+ 2 hours",
"type": "Time"
}
}
}
I think there’s a misunderstanding here- either yours or mine.
As I understand it, one way you can think of matchers is “this test covers every case that passes this matcher”. It’s not a description of the API- if you need an exact match, you just use the exact value.
I’m not completely across generators, as they’re after my time as a pact maintainer - but my understanding is that they’re just a way of building default values for matchers.
Following this, it would be valid for a pact implementation to use the generator OR the example during verification. If you need an exact value during verification, it should be written into the pact as an exact value.
The problem especially with date and time values is that sometimes the domain requires them to be in the future, but maybe also not too much in the distant future. In these cases, static values won't cut it.
Consider a hotel reservation system where reservations might be acceptable from today until today + 1 year, but not beyond. You could use generators for this, as the docs for date/time expressions state:
These expressions work relative to a base date + time, normally the current system clock. They provide a reliable way to work with relative dates in tests.
The consumer could specify a reservation request for from tomorrow until tomorrow + 2 days
should be acceptable. The provider can then be verified against values generated from the current (real) system date and time + 1 day
and +3 days
without having to fiddle with mocking the system clock. This is how I understand generators. Very valuable for provider verification.
Perhaps the naming of the API is a bit misleading. Matchers
imply what you and I understand, stated as bullet point 1 above. Matching is useful if returned data (e.g. UUIDs) cannot be known at the time when writing consumer tests because the data is dynamic.
But... It appears that the Matchers
API can also be used to define generators
in the Pact JSON (bullet point 2). Perhaps another API would be useful to differentiate between matching values for verification and generating values to issue requests.
On the other hand, I can see why both concepts are included in a single API. As far as I understand it generating values is also useful when you run the stub server, where maybe you want to return stubbed responses including timestamps that always reflect a future date.
I think normally you would handle that case with provider state- ie, “the time is {some specific time}”
Docs on generators do exist:
[Generating contents] will happen in consumer tests when the mock server needs to generate a response or the contents of a message, or during verification of the provider when the verifier needs to send a request to the provider.
Despite all of the above, this issue exists because MatchersV3.datetime
and MatchersV3.date
/MatchersV3.time
handle example values differently.
No matchers appear to handle expressions for date and time generation currently.
Looks like @agross has the right point here.
If other Matchers
do not use generators when an example value is provided, then MatchersV3.datetime
and MatchersV3.date
/MatchersV3.time
should behave in the same way for constancy.
Otherwise, while they do behave differently at the moment, it is confusing and causes issues.
while they do behave differently at the moment, it is confusing and causes issues.
Yes, I agree that it would be better to be consistent - even if the behaviour is an implementation detail rather than an expectation, it would be an improvement to keep that implementation detail consistent across all matchers.
I also agree that there's a naming issue with matchers in general - they're more like EquivalentTo
or something, meaning "this test covers all cases that pass this matcher", not "here's a description of the API".
My point was that I don't think it's expected behaviour that the provider verification will use the example (although it might). I think appropriate uses of matchers would mean that the test would be valid with any example that passes the matcher. If you need a specific, exact value, then that specific value should be used instead of a matcher.
If it is the case that there needs to be some constraints on the value (like the example of "within $X
time from now") or it depends on some other field in the request, then I think that is best addressed with provider state variables.
Despite all of the above, this issue exists because MatchersV3.datetime and MatchersV3.date/MatchersV3.time handle example values differently.
You're right, the definitions are slightly different:
export function datetime(format: string, example: string): DateTimeMatcher {
if (!example) {
throw new Error(`you must provide an example datetime`);
}
return pickBy((v) => !isNil(v), {
'pact:generator:type': example ? undefined : 'DateTime',
'pact:matcher:type': 'timestamp',
format,
value: example,
});
}
vs
/**
* String value that must match the provided date format string.
* @param format Date format string. See [Java SimpleDateFormat](https://docs.oracle.com/javase/8/docs/api/java/text/SimpleDateFormat.html)
* @param example Example value to use. If omitted a value using the current system date will be generated.
*/
export function date(format: string, example: string): DateTimeMatcher {
if (!example) {
throw new Error(`you must provide an example date`);
}
return {
format,
'pact:generator:type': 'Date',
'pact:matcher:type': 'date',
value: example,
};
}
The datetime
will never produce a generator, because example
can't be falsey.
My thinking here is that you should be opting in to generators explicitly, and that by having generators mixed into a matcher makes the behaviour confusing and unexpected.
I could see a few ways to do it:
example
optional, and if not provided, it will generate a value on the fly (this is slightly more intuitive at least)XYZGenerated
version (more explicit, but pollutes the API a bit)@uglyog is the expression language linked above designed for use outside of plugins?
These predate plugins, and are supported with V3 generators
Software versions
v16.19.0
Issue Checklist
Please confirm the following:
Expected behaviour
We use
MatchersV3.date
andMatchersV3.time
. The docs state:The expected behavior is that the provider will receive the passed example value of
10:11:12
:Actual behaviour
The provider receives a generated date/time value, despite examples being given.
The Pact JSON file also contains
generators
for such date or time matchers.Seems to be related to #620 which fixed it for
MatchersV3.datetime
.While researching this I came across this document which seems to indicate that date/time values on the provider side could be dynamically generated. How can such expressions be included when generated values are used (i.e. no concrete examples are given, but a recipe describing the values to be generated). It would be very useful to use because "the current date/time" is not enough for most of our use-cases.
Steps to reproduce