Compiler Tests

[GoogleCodeExporter]

The test suite for the compiler needs to be committed.

Original issue reported on code.google.com by ryoohki@gmail.com on 13 Feb 2012 at 8:20

[GoogleCodeExporter]

Actually, it plain old needs to be written :-)

Right now I spot check the compiler by running various programs from the 
Examples/Compiler and VM Tests folder 
(http://code.google.com/p/epoch-language/source/browse/?repo=examples#hg%2FCompi
ler%20and%20VM%20Tests). There really needs to be a way to run those in a 
single batch and check the compiled bytecode output for sanity.

I'm waffling over whether to have the compiler test be strictly on compiled 
files or if the test should be executed as well, although at that point we're 
honestly blurring the lines between testing the compiler and testing the VM.

Part of the reason I haven't automated all of this yet is that regression 
testing is nigh on worthless with the rate that syntax and semantic changes are 
going into the language. Even the bytecode structure is fluid enough still that 
just doing a naive comparison of compiler outputs will generate a lot of false 
failures.

I'm totally open to input on how to proceed with this. As Release 12 gets 
closer and the new compiler is more or less locked down, I really want to see a 
way to make sure it doesn't get subtly broken in the future.

It'd also be nice to have a real validation suite for when I start working on 
the self-hosting compiler.

Original comment by don.ap...@gmail.com on 15 Feb 2012 at 5:23

[GoogleCodeExporter]

Well, the first thing is that I don't see any sort of operator precedence. 
Which means that a simple expression like:

3+5*4

will produce the wrong results, it ends up grouped as: (3+5)*4, instead of the 
expected 3+(5*4). This has to do with the dynamic operator syntax, and the fact 
that the library registers the operators.

It is my opinion that the base operators for types (*,/,+,-,++,--,<<,>>,!,~) 
should all be built in with a defined precedence order. Baring that, the 
operator overloading syntax needs to be redesigned to utilize a couple of 
precedence levels (perhaps 3 levels... */ at the highest, then +-++--, then << 
>> (or however you want the shift operators to look)).

In EBNF this looks something like...
expr = expr + term | expr - term | term
term = term * factor | term / factor | factor
factor = identifier | literalNumber | '(' expr ')'

for a basic grammar. Also need to toss in prefix and postfix operators, etc.

Original comment by ryoohki@gmail.com on 15 Feb 2012 at 6:12

[GoogleCodeExporter]

AFter that, and once we have a basic grammar in place, the next thing to do 
should be to compose a complete set of tests for each major language component. 
We can create a Visual Studio project (an empty project) that has all the epoch 
files in it, with a compiler event that executes a batch file which copies the 
tests to a bin folder (or doesn't) and then runs the various EpochTools 
/execute commands...

On that note, I don't like having to run /execute. It should just be 
"EpochTools epochfile" dammit.

ALso on that note, we should get away from MessageBox calls, they're horrible 
for running tests as they will stop all execution till you address them. 
Instead perhaps redirect to std::cerr, then you can use piping to produce a log 
file... epochtools test.epoch 2>> errors.txt

Original comment by ryoohki@gmail.com on 15 Feb 2012 at 6:23

[GoogleCodeExporter]

Operator precedence is a known issue. There's a TODO lurking somewhere deep in 
the semantic checking code for reimplementing operation reordering to obey 
precedence rules. It's kept dynamic to allow for implementing user-defined 
operators easily, and to allow new operators to be added/modified by the 
library implementation (at any precedence) without having to shuffle the 
grammar around.

That said, I might be able to be convinced to embed the precedence ordering 
directly in the grammar, but I might not ;-)

I'll see about making the other changes as well; can't promise anything as my 
task list is already a mile long but they're good points.

I'll definitely also be thinking about ways to get a good automated testing 
setup like you describe.

Original comment by don.ap...@gmail.com on 15 Feb 2012 at 7:06

[GoogleCodeExporter]

Posting the task list would help :P especially if you use the issues list. An 
issue per task :P

Then as they get knocked out we can close them. It also means we can use issues 
as discussion points around the language to find solutions or alternatives.

Original comment by ryoohki@gmail.com on 15 Feb 2012 at 7:17

[GoogleCodeExporter]

Yeah, now that I'm not flying this thing solo I really need to get more 
organized. I'll start posting up issues for everything I know needs work.

Original comment by don.ap...@gmail.com on 15 Feb 2012 at 7:21

[GoogleCodeExporter]

How do we want to report non-fatal errors. I.e. syntax errors?

Original comment by ryoohki@gmail.com on 15 Feb 2012 at 7:46

[GoogleCodeExporter]

Syntax errors or semantic errors?

Syntax errors will require parser work as I need to get the 
filename/line/column tracking stuff hooked back into Spirit so I can emit 
sensible errors in the first place. It's mainly a matter of writing in rules to 
the grammar that catch malformed syntax and enable parsing to continue as much 
as possible, which is a slow and tedious blob of work. It's definitely 
something that needs to be addressed, though.

Semantic errors are different in that they can happen during the semantic 
validation/IR generation/code generation passes and need some kind of context 
from the parser as well.

I'm thinking of writing a basic "ErrorReporting" sort of thing that 
encapsulates recording error messages, their causes/locations, and so on. This 
will probably start by spamming cerr and then later on get plumbed through to 
display error details in the IDE and so on.

Original comment by don.ap...@gmail.com on 15 Feb 2012 at 8:45

[GoogleCodeExporter]

Well, syntax errors would be the "on_error" stuff with spirit, which is messy 
as hell, but. yeah.

For Semantic errors we need a way to fail out of the semantic validator when we 
do encounter an error though.

My initial thought would be to store some sort of std::vector<Error> and test 
that its non-empty before continuing on to the compilation/bytecode generation

Original comment by ryoohki@gmail.com on 15 Feb 2012 at 8:51

[GoogleCodeExporter]

Now that we have a proper test suite in the Examples repo I'm going to go ahead 
and close this; we certainly could use more test cases in the repo but I'll be 
adding them over time as I get features fixed up in the Release 12 compiler.

So far this has already caught a good half-dozen issues and helped fix all of 
them without causing collateral breakage, so I'm happy.

Original comment by don.ap...@gmail.com on 31 Mar 2012 at 4:45

• Changed state: Fixed