Public API for generating and executing test cases

[GoogleCodeExporter]

We should have an API to generate test cases on the fly programmatically. This 
would allow, for example, reading test data from external sources such as data 
bases.

The API could work somehow like this:

    from robot.running import TestSuite, TestCase, TestStep

    suite = TestSuite('Example suite', doc='...')
    tc = TestCase('Example test', tags=['t1', 't2'])
    tc.add_step(TestStep('Log', args=['Hello, world!'])
    suite.add_test(tc)

Original issue reported on code.google.com by pekka.klarck on 27 Apr 2011 at 1:55

• Blocking: #1463

[GoogleCodeExporter]

Issue 794 has been merged into this issue.

Original comment by pekka.klarck on 27 Apr 2011 at 1:56

[GoogleCodeExporter]

See also issue 826 about better API for running these tests.

Original comment by pekka.klarck on 29 Apr 2011 at 7:17

[GoogleCodeExporter]

This API could enlarge the way RF can be used. I would like the API to be even 
more high level than what Peke is suggesting.

Mainly: API that allows the use of robot libraries and log generation without 
having to use RF test data syntax.

Some situations where this could allow new ways to use RF:
- Model based testing
- Executable Sequence diagrams
- Pure Python tests with all the things you can do in Python
- Testing with Python REPL
- Other exiting SUTs that require very exotic DSL:s for efficient testing
- Executable Robot Logs (if randomness is used in tests than it is very good 
idea that the logs with the concrete values are executable)

It would be very cool if I could do something like this in a nose test:

from robot import RF

class SomeTestSuite(RF.TestSuite):

   def test_something(self):
          selenium = RF.import_library('SeleniumLibrary')
          selenium.open_browser()
          selenium.input_text("foo", "bar")
          selenium.click_button("button")

and the execution of it would produce RF style logs...

Original comment by mikko.ko...@gmail.com on 14 May 2011 at 11:29

[GoogleCodeExporter]

I totally agree with Mikko's ideas. Probably all this cannot be done already in 
2.6, though.

Original comment by pekka.klarck on 15 May 2011 at 10:57

[GoogleCodeExporter]

RF 2.6 deadline is getting close and unfortunately this issue cannot make it.

Original comment by pekka.klarck on 15 Jun 2011 at 12:39

• Added labels: Target-2.7

[GoogleCodeExporter]

It's very useful feature. You can write programming language or txt test cases. 
They can transform to each other.

Original comment by xuedipia...@gmail.com on 27 Sep 2011 at 8:20

[GoogleCodeExporter]

Unfortunately we don't have time to implement this enhancement in RF 2.7. The 
required changes are simply too big and we are afraid that the changes would 
escalate.

The main focus of RF 2.8 will be enhancing the actual test execution and 
implementing better APIs for that suites the scope very well.

Original comment by pekka.klarck on 7 Feb 2012 at 10:37

• Added labels: Target-2.8

[GoogleCodeExporter]

Issue 1209 has been merged into this issue.

Original comment by pekka.klarck on 27 Aug 2012 at 8:50

[GoogleCodeExporter]

This API should be somehow usable also via listener interface to modify tests 
dynamically. That would basically require implementing issue 1208.

Original comment by pekka.klarck on 10 Jan 2013 at 8:49

[GoogleCodeExporter]

Incidentally, I am now having a requirement to add [just] tags to existing test 
cases.
I saw this issue just now, and now I wish if its done things will be much 
easier for me.
My requirement originated from the requirement to write just the testcase(s) 
first upload it to a web application,  and then later add related tags , from 
the web application itself. 
The ideas is that managerial guys can organize test cases any which way they 
like.

Requesting to consider this facility as well (adding [just] tags to existing 
test cases)

Original comment by binithb@gmail.com on 2 May 2013 at 3:08

[GoogleCodeExporter]

Issue 826 has been merged into this issue.

Original comment by pekka.klarck on 31 May 2013 at 11:33

[GoogleCodeExporter]

This issue has been the biggest task in the forthcoming 2.8 release. Finally it 
is pretty much done, the main task is documenting the new APIs and cleaning up 
internal code.

Here is an example of what works now:

from robot.running import TestSuite

suite = TestSuite(name='Example')
tc = suite.tests.create(name='Test', tags=['t1', 't2'])
tc.keywords.create(name='Log', args=['Hello world!'])

Original comment by pekka.klarck on 31 May 2013 at 11:40

• Changed title: Public API for generating and executing test cases
• Changed state: Started
• Added labels: Priority-Critical
• Removed labels: Priority-High

[GoogleCodeExporter]

One important line missing from the example in the previous comment:

from robot.running import TestSuite

suite = TestSuite(name='Example')
tc = suite.tests.create(name='Test', tags=['t1', 't2'])
tc.keywords.create(name='Log', args=['Hello world!'])
result = suite.run()    # This was missing.

Running the above will create output.xml file and also return a result object.

Original comment by pekka.klarck on 3 Jun 2013 at 8:48

[GoogleCodeExporter]

This issue was updated by revision 4b645083af18.

Implemented shortcuts for importing, e.g. following is now possible:

suite = TestSuite(name='Suite')
suite.imports.library('OperatingSystem')
suite.imports.resource('resource.txt')
suite.imports.variables('variables.py')

instead of saying, e.g.:

suite.imports.create('Library', 'OperatingSystem')

Moved class 'Imports' from 'Running' to 'Model' as a new module.
Created separate unit test module for import related tests.

Original comment by anssi.sy...@eficode.com on 6 Jun 2013 at 10:14

• Changed state: Review

[GoogleCodeExporter]

This issue was updated by revision b0d9b4f396dd.

It is now possible to set criticality like this:

suite = TestSuite(name='My suite')
test_case = suite.tests.create(name='My test case', tags='regression')
test_case.keywords.create('Log', args=['hello'])
test_case2 = suite.tests.create(name='My second test case', tags='tag')
test_case2.keywords.create('Log', args=['world'])
result = suite.run(critical='regression')

assert result.suite.statistics.total == 1
assert result.suite.statistics.all.total == 2

Also moved code relating to test criticality from model's test suite to 
result's test suite. Also moved criticality from model's test suite configurer 
to result's test suite configurer. Finally, moved unit tests relating to 
criticality under test_resultmodel.py.

Unfortunately could not remove criticality property from model's test suite as 
model's test case depends on it. Need someone else to look on how to go about 
achieving that.

Original comment by tatu.ka...@eficode.com on 6 Jun 2013 at 4:24

[GoogleCodeExporter]

Original comment by pekka.klarck on 6 Jun 2013 at 4:42

• Now blocking: #1463

[GoogleCodeExporter]

There are still some TODOs in the code related to changes and parts of the API 
still need to be documented better. These are covered by issue 1460 and issue 
1468, respectively, so this issue can be considered done.

Original comment by pekka.klarck on 8 Jun 2013 at 7:45

• Changed state: Done

[GoogleCodeExporter]

Please share/document examples for variable assignment in the area of using API 
test_case.keywords.create. 

*** guess ***
from robot.variables import *
from robot.running import EXECUTION_CONTEXTS as EC

suite = TestSuite(name='my test')
tc = suite.tests.create('my test')
result_handler = tc.keywords.create('Callable returning something')
local_context = EC.test()
local_context.variables()["named_variable"] = result_handler

*** end guess ***

Original comment by msud...@gmail.com on 4 May 2014 at 10:52