search

stoplightio / prism

Turn any OpenAPI2/3 and Postman Collection file into an API server with mocking, transformations and validations.
https://stoplight.io/open-source/prism
Apache License 2.0
4.28k stars 343 forks source link

Add a flag to allow running Prism as a validation proxy without mocking 501 responses #1963

Open johnboyes opened 2 years ago

johnboyes commented 2 years ago

User story.

As someone doing integration testing, I can use a flag to run Prism in validation proxy mode without 501 responses being mocked, so that Prism truly identifies all discrepancies between the OpenAPI spec and the server.

Is your feature request related to a problem?

I want to use Prism in validation proxy mode to catch any discrepancies between the server and the OpenAPI spec. Currently if the server responds with a 501 then Prism automatically mocks the response instead of raising an error (that the server does not conform to the OpenAPI spec). This current behaviour is definitely useful in many situations, but is definitely not desired when wanting to use Prism's validation proxy as an authoritative source of truth as to whether the server conforms to the OpenAPI spec.

Example workflow to illustrate the problem:

  1. spec-first approach using OpenAPI
  2. default the server to 501 responses to begin with
  3. progressively implement the server responses
  4. but how can you verify if you have forgotten to implement some of the server responses?

I guess that the only way currently to verify if you have forgotten to implement a server response would be to run the tests directly against the server too (i.e. without Prism). It would be much simpler though if Prism was able to catch this.

Describe the solution you'd like

I propose that the Prism CLI has a flag (something like mock-unimplemented, maybe) to set the 501 mocking on or off. It would default to on, which would provide complete backwards compatibility with how Prism works today.

Additional context

My context is using Prism as a key part of a consumer-driven contract testing approach in a large organisation. We want to use Prism to catch any discrepancies (including 501s) when running our contract tests.

johnboyes commented 2 years ago

Hi, really grateful if anyone is able to let me know if this will be considered, please? Thanks v much.

chohmann commented 2 years ago

@johnboyes we think this feature would be useful! We don't have a whole lot of time to devote to building this feature right now, would you be interested in putting up a PR for this?