Allow example data for stub server to be different than template for verification

[agross]

Feature description

We use the stub server extensively to provide frontend developers with good examples. A common use case is that APIs return a list of objects:

[
  { "foo": 42 },
  { "foo": 23 },
  { "foo": 1 }
]

We were unable to find a way to specify the following for the interaction that would return this set of data:

• The stub server should return the document above to provide >1 examples for frontend devs
• When the API that generates the document above is verified the array returned should at least contain a single item that matches the { "foo": <integer> } template

The interaction for the example above includes this:

.willRespondWith({
  status: StatusCodes.OK,
  body: Matchers.like([
    { foo: Matchers.integer(42) },
    { foo: Matchers.integer(23) },
    { foo: Matchers.integer(1) },
  ]),
});

This provides all 3 examples to the fontend developer as intended. But since the like matcher does not validate the actual array length, the API would be verified even if it returned [].

The main reason we use Pact is to make sure the schema between frontend and API is compatible. Since the API developers can "work around" being really verified by returning an empty array this is rather unfortunate.

Something we tried is using atLeastOneLike, but that will return the array encapsulated in another array when using the stub server:

.willRespondWith({
  status: StatusCodes.OK,
  body: Matchers.atLeastOneLike([
    { foo: Matchers.integer(42) },
    { foo: Matchers.integer(23) },
    { foo: Matchers.integer(1) },
  ]),
});

Another variant ensures that the API contains an array with length >=1, but the fontend developer just gets 1 example to work with:

.willRespondWith({
  status: StatusCodes.OK,
  body: Matchers.atLeastOneLike(
    { foo: Matchers.integer(42) }
  ),
});

Maybe we're doing it wrong?

[mefellows]

Unfortunately, the way the matchers work today, is that you provide a single representative sample that can be used in the list object.

But I do think there is merit in allowing the scenario you are talking about, you're not the first to ask, and it makes sense when used with the stub server.

@uglyog what do you think about supporting this? I'm wondering if there are other similar matchers that could allow a similar form (e.g. matchKeys and the min/max variants of the array matcher)

[mefellows]

The main reason we use Pact is to make sure the schema between frontend and API is compatible. Since the API developers can "work around" being really verified by returning an empty array this is rather unfortunate.

This is a separate problem but also needs to be handled. It sounds like a bug on the surface of it.

[rholshausen]

With Pact-JVM, the DSL has a second form for all the like matching functions that allows you to specify the number of examples. I think having this option would help here?

I.e.:

.willRespondWith({
  status: StatusCodes.OK,
  body: Matchers.atLeastOneLike(
    { foo: Matchers.integer(42) },
    3
  ]),
});

However, they would all have the same example value (42).

.willRespondWith({
  status: StatusCodes.OK,
  body: Matchers.atLeastOneLike(
    { foo: Matchers.integer() }, // Use a random integer generator
    3 // have 3 example values
  ]),
});

You could also have a form where you don't provide the example value, and the Pact framework could use a generator to replace the values at runtime with random ones:

[mefellows]

With Pact-JVM, the DSL has a second form for all the like matching functions that allows you to specify the number of examples. I think having this option would help here?

We have that option already also (you can optionally set the min, which defaults to 1). Is that an option @agross ?

I think the ask is that the user can specify heterogenous examples that are more useful in a stub context (e.g. different rows of data). The contract could store these examples, but each example given must also match the matcher.

e.g.

.willRespondWith({
  status: StatusCodes.OK,
  body: Matchers.atLeastOneLike(
    { "name": "Moe" },      // All examples must have the key `name` that is a string
    [                                    // Additional examples to use in the current test / serialised pact
      { "name": "Marge" },
      { "name": "Apu" },
      { "name": "Homer" }, 
    ]
  ]),
});

This way, when the contract is used in a stub scenario, the data will be more realistic. What do you think?

[agross]

specify heterogenous examples that are more useful in a stub context (e.g. different rows of data)

Yes, absolutely! Random data would not be beneficial, and probably hard to generate if it's not just a number but a complex object that also needs to make sense in the problem domain.

This way, when the contract is used in a stub scenario, the data will be more realistic. What do you think?

It would be great if such an API would be supported! I pasted your code, looks like it's currently not.

[seyfer]

That is a pretty actual issue. I work from FE side, defining Pacts for BE API. The problem I have at the moment is very similar. Consider this code:

    const someMatcher = Matchers.atLeastLike(
      {
        address: Matchers.string(entry1.address),
        buildingIds: Matchers.atLeastLike(
          Matchers.uuid(entry1.buildingIds[0]),
          0,
          2
        ),
      },
      0,
      2
    );

I have min 0 as we want to allow empty array and count 2 as I want 2 examples. But both examples will be equal to entry1.address value, making it not useful on FE, as I need 2 different examples and I have fixtures for these examples that are entry1 and entry2 objects. And if I leave it Matchers.string() to get a random string, that is not useful for the state representation on FE.

Another option is to code a matcher explicitly with my fixtures data:

const someMatcher = [
      {
        address: Matchers.string(entry1.address),
        buildingIds: [
          fromProviderState(
            "${buildingId1}",
            Matchers.uuid(entry1.buildingIds[0])
          ),
          fromProviderState(
            "${buildingId2}",
            Matchers.uuid(entry1.buildingIds[1])
          ),
        ],
      },
      {
        address: Matchers.string(entry2.address),
        buildingIds: [
          fromProviderState(
            "${buildingId3}",
            Matchers.uuid(entry2.buildingIds[0])
          ),
        ],
      },
    ];

That will use my fixtures, but it will implicitly match that the expected array length is 2 elements, and this is not what we want. We want constraints like in the first example - allow empty array and generate 2 examples, and do not have an upper array length bound - but with my fixtures values for these 2 generated examples.

Would be great to have 4th parameter to provide example values:

const someMatcher = Matchers.atLeastLike(
      {
        address: Matchers.string(),
        buildingIds: Matchers.atLeastLike(
          Matchers.uuid(entry1.buildingIds[0]),
          0,
          2
        ),
      },
      0,
      2,
      [entry1, entry2]
    );

given entry1 and entry2 have the same object structure and the property value will be taken only if no value was specified for a matcher. In this case, the address would be taken from entry1 and entry2, but buildingIds will use a specified matcher, as it has a value set in the matcher. or, if object merging is too complex, then a complete overwrite for both properties from provided example objects for generated examples, but the matcher JSON types definition will be the same, as it is now.

Also, imagine I need a meaningful entry to be matched, such as a rectangle coordinates, which is 4 integers. Now I do something like coordinates: Matchers.constrainedArrayLike(Matchers.number(135), 4, 4, 4), that gives me 4 times 134 in an example. Random numbers won't work as well, as I need a proper rectangle width and height, also x,y bound to some range, let's say 0 to 1000. But I have fixtures for rectangles and would like to use these fixture values in the stub server-generated example, which is not possible.

Would be great to have 5th parameter: coordinates: Matchers.constrainedArrayLike(Matchers.number(), 4, 4, 4, [11, 12, 326, 247]), and this code would use each provided example value accordingly.

[ipfefferpax8]

Hello! This would be incredibly useful in allowing our Frontend's be able to test more accurately and provide better isolated testing. Are there any updates on this?

[mefellows]

No update. If you are interested, it would require a change in the core. I'll move the issue there as a feature request.

[rholshausen]

By stub server, I assume you mean the mock server provided by Pact?

[agross]

In the OP I was referring to the stub server that provides examples for the frontend people.

[rholshausen]

Ok, do you mean this one? https://github.com/pact-foundation/pact-stub-server

[agross]

Yes, I think so. Answering this while boarding an airplane. Cannot check my project's source code right now.