Questions regarding setup for code examples

[schultyy]

Hello all,

for the next big task (the setup/providing context for code examples) I collected questions I would like to discuss with you. Short explanation of the problem: Given a markdown file having code examples which use code defined elsewhere. Currently ataru cannot handle this case. To support this kind of documentation, the user of ataru should be able to provide a setup which provides the necessary context needed by the code examples.

Questions:

1. Which types of documentation should ataru support?
2. How should a setup look like?
3. How to test a code sample which has a dependency to e.g. a third party service or database?
4. How to include a code sample in the generated test cases?
5. How to provide setup for each code sample?

To 1 There are types of documentation which are hard to test, for example executing shell commands. Can these be tested in a good way or are they not supported?

To 2 How should a setup look like technically? One option could be that the setup file contains a Ruby module or a class which must provide methods that are called by ataru (like rspec's before and after).

To 3 How does ataru handle code examples which for example depend on a third party web service? Could libraries like VCR be an option? And if yes, how does the user include them in the setup?

To 4 When the user provides a setup file, how is this file made available in the generated test cases?

To 5 Given the case each code example in a file needs its own setup. How do we structure the setup file?

[schultyy]

Regarding the setup file structure: My thought is that this could be a module so it can be mixed into the generated test class later on. The code samples are in a class which inherits from MiniTest::Test. When the module provides a setup method then it is automatically called by MiniTest before each test case.

[emig]

Ataru should be able to run using the dependencies from the target code

(1) Which types of documentation should ataru support?

We should be incrementally supporting types of documentation, starting small. A good starting point could be just ruby code with dependencies that are fulfilled by the project's dependencies alone. (no extra setup). In order have a standard target project, we can require the project to have a gem structure itself. This will garantee the project has a Gemfile

(2) How should a setup look like technically? One option could be that the setup file contains a Ruby module or a class which must provide methods that are called by ataru (like rspec's before and after).

As the intention is to load the dependencies from the target project, the easiest would be to use Bundler to load them for us. The setup file can provide the target project dependencies to Bundler at runtime

(3) How does ataru handle code examples which for example depend on a third party web service? Could libraries like VCR be an option? And if yes, how does the user include them in the setup?

We won't include dependencies on external services for now

(4) When the user provides a setup file, how is this file made available in the generated test cases?

We let Bundler manage loading gems.

(5) Given the case each code example in a file needs its own setup. How do we structure the setup file?

Are we talking about having a setup per code example? Wouldn't that make documentation task too complicated? Having general setup capacity is desirable, and could be loaded in a before hook, pretty much like rspec does.

[kazjote]

I agree with emig about loading gem dependencies. If Ataru will be used as a gem inside of the other project, it will have all the other projects dependencies loaded automatically.

Here is my idea on how to implement test setup.

Let's assume we are testing Sinatra using Ataru and Rake::Test. We want to do it similarly to what is here http://www.sinatrarb.com/testing.html

Let's assume our markdown with testable documentation looks like this:

Given application:

`
# setup module

require 'rack/test'

include Rack::Test::Methods

class BasicRequestApp < Sinatra::Base
  get '/' do
    "Hello World #{params[:name]}".strip
  end
end
`

Sinatra sends String returned by action body as HTTP body of the response:

`
get '/'
assert last_response.ok?
assert_equal 'Hello World', last_response.body
`

Ataru would check the first line of each loaded snippet and look for # setup module string. Snippet starting from such line would be evaluated in the context of MiniTest::Test class. It wouldn't be used to create new testcase.

Let's say we have the first snippet in the setup variable. We need to evaluate it as follows:

MiniTest::Test.class_eval(setup)

After processing both snippets by Ataru we would get something equal to:

class MiniTest::Test
  require 'rack/test'

  include Rack::Test::Methods

  class BasicRequestApp < Sinatra::Base
    get '/' do
      "Hello World #{params[:name]}".strip
    end
  end
end

class Klass < MiniTest::Test
  def test_my_file_test1
    get '/'
    assert last_response.ok?
    assert_equal 'Hello World', last_response.body
  end
end

I think this solution is good because

1. It is simple
2. It allows us to keep all setup in markdown (documentation) files
3. It is very powerful. We can do practically everything.

When developer would like to use this setup module to setup his own test he could do it with 2 snippets:

Setup snippet:

# setup module

module SimpleApp
  def setup_fast_car
    Database.create(:car, speed: 300)
  end
end

extend SimpleApp

Test snippet:

setup_fast_cars

assert_equal 300, Database.first.speed

The problem of this solution is that it doesn't support teardown. However, teardown is much less used than setup.

[bitboxer]

:+1: for @kazjote proposal. Let's start small and see where you go from there.

[moonglum]

The @kazjote proposal sounds very reasonable.

[nanoparsec]

I am afraid that I need some more explanation. I am totally lost atm. Sorry. :(

[bitboxer]

Don't be sorry for this. I think the best way to go forward is to do the explanation in person. Using a whiteboard. That should make things easier to understand for you two.

[emig]

and today you have @kazjote around! whiteboard force!

[moonglum]

Ok, I tried to make a simple example for the way I would imagine this to work. Does that help you, @NerdBabe and @madziaf?

Gist

[nanoparsec]

We are not so happy with the idea to put requirements into the markdown code samples because it makes it harder for the normal reader (the person that wants to read only the documentation) to find out what is relevant.

So we decided to put all requirements into a setup.rb file that is required by every generated test class. This setup.rb file will contain only requirements at the moment but it could be expanded with helper methods later.

Workflow:

1. User adds ataru to her gemfile -> you can run bundler (loading all dependencies)
2. run bundle
3. bundle exec ataru setup (setup is, next to gogo, a Thor method, it will create a ataru_setup.rb file)
4. from target application folder user runs: bundle exec ataru gogo md-files [TO DO: expand gogo method with: require ataru_setup.rb if the file is in the target project directory require ataru_setup.rb if File.exist?('ataru_setup.rb')

-> Ataru runs and gives a nice output in the end.

How ataru_setup.rb will look like at the moment:

comment: Please write all the requirements needed to run the code snippets from your Markdown files require Application Code the user wants to test require Gems ... require xxxx require xyz

How to connect our Test Classes generated in CodeSamples with the setup requirements

Why: Because we want to put all requirements from the ataru_setup.rb into every automaticly generated test class.

How:

Add to gogo: require ataru_setup.rb with absolut path -> require Dir.pwd + '/ataru_setup.rb'

setup-file has to be in target application root directory!

Workflow for generating a ataru_setup.rb file with Thor

1. run ataru setup (setup is another Thor method that starts the setup file generator - still has to be written)
2. The generator asks the user to give all requirements that are needed.
3. When finisehd, the generator asks where the file should be stored.

[madziaf]

One of the proposed solutions was implemented with #55