Construct test-cases

[GoogleCodeExporter]

We need a comprehensive set of test cases to test the compiler, along with
an automated process to run them and check for passes.

Original issue reported on code.google.com by arachnid...@gmail.com on 3 Jun 2007 at 12:16

[GoogleCodeExporter]

Original comment by arachnid...@gmail.com on 3 Jun 2007 at 12:23

[GoogleCodeExporter]

I'll get to work on adding a nunit test assembly. 

Original comment by philpr...@gmail.com on 3 Jun 2007 at 7:36

[GoogleCodeExporter]

Excellent, thanks for picking this up.

Reading this forum post: http://forum.lolcode.com/viewtopic.php?id=44 it seems 
like
it would be a good idea if we make the LOLCode portion of our unit tests serve 
a dual
purpose - introducing language features and testing them - so we can post them 
to the
wiki as a comprehensive set of examples.

Original comment by arachnid...@gmail.com on 3 Jun 2007 at 9:23

[GoogleCodeExporter]

I've started the ground work for this, and have some working examples. 
Everything is 
in the LOLCode.net.tests.dll assembly using NUnit as the testing framework. I 
suggest users grab a copy of Nunit 2.4.1 (NUnit) and if you are using VS get 
TestDriven.net personal edition (free). Right click -> Run test(s) is a god 
send. 

Overview:

 - Currently sample LOLCode is embedded into the assembly (Under the sample name 
space), this means tests can be run without having a bunch of external 
depedancies, 
ergo the assembly is self contained.

 - Fetching samples is done with the SampleHelper class (it even has its own unit 
tests) 

 - Samples can either be complete, or in blocks. Files should contain a "code" 
and "baseline" block (see haiworld.lol for an example), or no blocks at all 
(fulltest.lol). Blocks are specified by '-->begin <name>\r\n'. Baselise blocks 
are 
used for run-time tests to ensure the expected output is given. 

 - Runtime tests really test two things. 1) The file compiles (errors are asserted 
if they dont) and 2) they run as expected 

 - There are 3 runtime tests at the moment, haiworld, simple1 and visible. The 
current implimentations spawns a process to run the test, I've stubbed out the 
ability to execute tests in memory, i just need to figure out how to do this 
(and 
collect whatever ends out on STDOUT) 

 - A bunch of unit tests have been created for stdlol, however they are by no means 
complete.

TODO:

 - More Runtime tests! Need a comprehensive set of lolcode to test.
 - Parser tests, no runtime check the tree is in order
 - Negative tests for parser and runtime (we expect something to go wrong)
 - Enable interation with run time tests (give input to GIMMEH statements) 
 - Regression tests for current bug (when found ;))

Original comment by philpr...@gmail.com on 3 Jun 2007 at 11:03

Attachments:

• test-assembly.patch

[GoogleCodeExporter]

Revision 13 contains the above patch.

Original comment by philpr...@gmail.com on 4 Jun 2007 at 12:50

[GoogleCodeExporter]

Excellent work! Once we're more organised, this should probably go on the wiki 
or in
docs. :)

Changed status to 'started'.

Original comment by arachnid...@gmail.com on 4 Jun 2007 at 2:06

• Changed state: Started