Documentation-Driven Development

brettz9 commented 5 years ago

Problem (enhancement)

I have need for producing a usage guide. Besides providing these results for end users, it may be useful during the design phase to ensure non-technical collaborators can provide their own input. This is very important in commercial environments but also potentially in non-commercial/open source projects.

I would like for my documentation to inform the tests, allowing them to be written before the tests as well as along with the tests--Documentation-Driven Development.

Current behavior:

Currently, I must either write documentation separately from my tests or extract comment blocks added above my tests, either of which can lead to synchronization problems and redundancy.

Desired behavior:

I would instead prefer to be able to provide human-readable directions and labels in the context of my test code (which I write with my application code), allowing this information to be used to enhance human-readable, non-technical documentation which may be built automatically out of cypress commands.

Steps to reproduce: (app code and test code)

Your docs have the following code sample:

it('changes the URL when "awesome" is clicked', function() {
  cy.visit('/my/resource/path')

  cy.get('.awesome-selector')
    .click()

  cy.url()
    .should('include', '/my/resource/path#awesomeness')
})

I would like to be able to provide the following human-documentation-enhanced version instead

it('changes the URL when "awesome" is clicked', function() {
  cy.doc('my resource page').visit('/my/resource/path')

  cy.doc('the "awesome" button').get('.awesome-selector')
    .click()

  cy.doc('the awesomeness page').url()
    .should('include', '/my/resource/path#awesomeness')
})

Which can be used to auto-generate the following documentation:


## Changes the URL when "awesome" is clicked

1. Visit my resource page.

2. Click the "awesome" button.

3. See that you are brought to the awesomeness page.

If a doc() command were omitted, other information could be checked, e.g., the page title for a cy.url() call, the title (or aria-*) attribute found on the targeted cy.get element, etc. Ideally, the testing process could flag items missing a label, so that the developer could ensure a human-readable label were added within the code.

In some ways such an approach would be superior to explicit doc() calls, as it may better promote accessibility (and indeed, it could in turn also encourage the tests themselves to be made more human-readable in that the enforced presence of title and aria labels would allow for the tests to incorporate more readable selectors such as cy.get('[title~="the \\"awesome\\" button"]')) (or with an escaping utility, cy.getAttrContent('title', 'the "awesome" button')), but no doubt there would be times when an explicit doc() would still be called for.

Other documentation formats could be generated as well:
1. Non-English locales
2. Each step of the above documentation could optionally have a screenshot inserted. (Programmatic code could be added along with doc() to conditionally suppress or enable these screenshots on a per-line basis.)
3. A distinct documentation version could also be created which allows the actual test code to appear as a tooltip upon hovering over the documentation line items.
4. HTML (as opposed to Markdown), PDFs, etc.
The test results within the cypress Test Runner could themselves show these documentation details, e.g., with the human-readable information showing as a tooltip on hover (or if desired, in place of the default text), providing better human-readability to the tester, and easier confirmation by the tester that their docs will indeed remain understandable and accurate in the course of carrying out their testing/development process.

(The above approach could even be used for to-dos. To-dos could be written as bare tests, and could be tied to a tracker issue number such that an issue is automatically filed and/or updated (along with human-readable info) for each test marked as a to-do, upon commit, thus reducing redundancy between to-do lists and tracker to-dos, as well as minimizing time toward beginning test-writing/coding, as the skeleton would already be written.)

Versions

This is an enhancement, but FWIW, I'm currently using Cypress 3.4.1, on Mac 10.14.6, with Chrome.

kuceb commented 5 years ago

This is a cool idea, and can be built as a cypress plugin. I would also check out https://github.com/TheBrainFamily/cypress-cucumber-preprocessor, which tackles a similar problem using Cucumber.

I'm going to close this for now only because there is no work to be done in core for this feature unless a plugin is made and there's a request to bake it into core

bahmutov commented 5 years ago

@brettz9 I love the idea, and I have written https://github.com/cypress-io/cypress-fiddle to kind of have self-documented code. I have even extended our documentation with custom {% fiddle path/to/spec/file "name of the test" %} to include test examples in the doc pages, see https://github.com/cypress-io/cypress-documentation/pull/2063

I am thinking now after discussing this that I should do the inverse for the documentation - since you don't want to decouple examples from the rest of the documentation page, but instead just run it. So I am thinking cypress-fiddle should implement Cypress preprocessor http://on.cypress.io/preprocessors-api and be able to run tests from a Markdown file. I have opened an issue https://github.com/cypress-io/cypress-fiddle/issues/14 to not forget it

brettz9 commented 4 years ago

Firstly, let me say, @bahmutov and all, how much I admire your work. You all have done a really impressive and thorough job, and it seems Cypress has anticipated almost everything! You really seem to have put a lot of thought into making things work in an ideal way for users.

And your documentation is awesome is well.

As far as cypress-fiddle... I think that running tests from a Markdown file would be an intriguing option, particularly when you need more surrounding documentation and background than just a step-by-step. Besides avoiding redundancy of multiple instances of test code, it indeed also nicely avoids the need for adding separate inclusion statements (though those who like to keep their test files separate could still presumably embed individual tests through a Markdown inclusion processor?).

However, after your proposal may be implemented, there would still be some redundancy remaining (though at least within the same file).

To avoid redundancy completely in having to add step-by-step documentation manually in addition to the embedded code, I'd think your approach could be enhanced--even within the proposed preprocessor of cypress-fiddle--by auto-generating the more step-by-step documentation out of the embedded test code alone (through my proposed doc() and auto-detection of element labels, etc.), and embedding this auto-generated documentation in the final Markdown output above or below the test code (or even optionally without the test code) along with optional embedding of the test-generated screenshots.

Off topic slightly... I was thinking that with cypress' exciting code coverage, cypress-fiddle might also auto-generate a badge (or tabular report) indicating the level of coverage, with an ability to auto-reference it within the docs, so that projects could advertise their use of cypress and the degree of coverage as well as current test status.

cypress-io / cypress