Replace Gavel Validation With Custom Validation

[ttahmouch]

Gavel currently uses a simple text diff to validate HTML response entity bodies. I would like Gavel to support X/HTML validation in the future using XML Schema. However, here is how I am circumventing Gavel validation, and validating a response myself.

var hooks = require('hooks');
var stash = {};
...
/**
 * Validate HTML generated from an API Blueprint. Gavel validates X/HTML using
 * Simple Text Diff. Simple Text Diff is primitive. We can do better.
 * If the HTML passes our validation, then force Gavel to pass by masking the
 * real response body with the expected body, else fail before Gavel validation.
 *
 * @see {@link https://www.relishapp.com/apiary/gavel/docs/expectations}
 */
function validateBlueprintHtml(transaction) {
  var protagonist = require('protagonist');
  var cheerio = require('cheerio');
  var $ = cheerio.load(transaction.real.body);
  var apibName = protagonist.parseSync(transaction.request.body).ast.name;
  var htmlName = $('#apiHeadline').text();
  var valid = apibName === htmlName;

  if(valid) {
    transaction.real.body = transaction.expected.body;
  } else {
    transaction.fail = "Generated HTML Does Not Include An API Name From Its Blueprint.";
  }
}
...
hooks.beforeValidation('Blueprint > Generate Blueprint HTML > Generate Blueprint HTML', validateBlueprintHtml);

As a user, I want to replace Gavel validation in cases where testing a transaction could not feasibly pass because a real response body will never equal an expected response body.

Perhaps there could be a hooks.validation() or hooks.duringValidation() that allows for an alternative to Gavel. Perhaps there could also be transaction.pass, with similar intention as transaction.fail, to pass before or during Gavel validation.

[netmilk]

@honzajavorek Workaround solution for this could be one afterEachValidation hook and adjusting the transaction object:

• remove the Gavel results and the test.valid property
• do the custom validation based on expected data from the transaction object
• set the test.status
• set the test.isValid
• use the same results data structure for the custom validator output

[honzajavorek]

test.valid or test.isValid? (you mention both)

Otherwise, your proposal make sense to me. However, this would mean Gavel would always run and user could just overwrite it's results, but I guess that's not an issue.

[pksunkara]

I would love to have the option to say that dredd needs to validate the exact values of the responses even if it's application/json.

[netmilk]

You can do that already! Just look to the documentarion for custom chai expectstions in hooks.

[honzajavorek]

Dredd's life cycle, as it was designed and documented, was broken until now - you could not change the transaction.test object in hooks and in some cases, Dredd would emit test fail events to reporters right after Gavel validation. This is now fixed in https://github.com/apiaryio/dredd/pull/648.

I'm pretty confident we have now, as @netmilk wrote, everything in place. Moreover, everything should also work as expected and documented.

Hence implementing this in hooks should be entirely possible now and not only with afterEachValidation hooks, but also with after and afterEach hooks. This issue can be closed if the use case gets a nice, working example in documentation (and preferably also test).

[artem-zakharchenko]

As I understood the original intention is to replace Gavel validation by arbitrary validation. In that case the usage of Gavel as a validator should be opted-out, meaning Gavel must not be called in the first place.

Perhaps, a hooks.validate() hook to perform the validation by yourself? It would be required to meet the Gavel's unified validation result structure in that case. It's quite generic at this point, so it's not coupled with any kind of underlying validator. This way the input to Dredd would be reliable, as there can be multiple things related on the validation result (argument against manual transaction mutation to imitate custom validation).