metanorma / ci

Build scripts for Metanorma, works with continuous integration
3 stars 2 forks source link

Disable documents test on processors metanorma-* #130

Closed CAMOBAP closed 7 months ago

CAMOBAP commented 7 months ago

This PR to discuss our metanorma test approach

Recently @opoudjis asked me:

are the samples tests in metanorma-* now redundant? ... Just want to make sure documents aren't being tested twice

For now we do testing documents in 3 places:

  1. on metanorma-* gem changes, we test only specific mn-samples-* and mn-templates-*
  2. on metanorma-cli gem changes, we test subset of all our mn-samples-* and mn-templates-*, king of smoke test
  3. on mn-samples-* and mn-templates-*changes

@opoudjis proposal is to remove processor tests (first row from the list above)

Pros:

Cons:

@ronaldtse any thoughts?

P.S. If we can make development more comfortable for @opoudjis I have no objections to go with this approach

opoudjis commented 7 months ago

The way I do development, the long wait on flavour testing is more disruptive than an issue that turns up in documents or templates and not identified in the flavour rspec. The former happens constantly, the latter only happens every few months. I don't think that there is a real problem in identifying flavour-specific issues only when I get to metanorma-cli: the release takes long enough that stopping to debug the flavour (after a very very long wait) is going to end up happening the next day anyway.

CAMOBAP commented 7 months ago

@ronaldtse if no objections, I propose to merge this PR

ronaldtse commented 7 months ago

@CAMOBAP @opoudjis I can live with the new proposal. Feel free to merge when ready. Thanks!