search

fir-lang / fir

MIT License
23 stars 3 forks source link

Start testing the interpreter #8

Open osa1 opened 2 months ago

osa1 commented 2 months ago

We should start testing the interpreter.

The tests should be written in a way that is possible to run in the bootstrapped compiler as well.

My current thinking is that we should have everything needed to evaluate output of a program in the same source as the program. The test runner should parse the comments in the file for: exit code, stderr and stdout outputs.

In the interpreter, we can probably use golden-tests or something similar.

In the bootstrapped compiler we can keep using the same library, or port it to Fir.

osa1 commented 2 months ago

I've started adding some tests in https://github.com/fir-lang/fir/tree/main/tests, using golden-tests.

We should add more.

osa1 commented 2 weeks ago

Two things that I think we should add to golden-tests:

  1. We should make it possible to somehow process stdout/stderr before comparing, e.g. maybe with a line of shell.

    This allows checking stderr in case of type error without comparing the stack trace printed by the error.

    Of course type errors shouldn't be panics, but (1) preprocessing the program output before comparing would still be useful filter out unimportant details (2) panicking may be acceptable in the interpreter, which we want to keep simple.

  2. We should make it possible to run a program multiple times, passing different inputs. For example:

    fn main(input: Str) = ...

# input:
# ...
# expected stdout:
# ...

# input:
# ...
# expected stdout:
# ...

    Here the program should be run twice, with the given inputs and expected outputs.

    This is useful in cases like:

    1. We can test example programs this way. Example programs will have a main and print stuff.

      We could add a test mode and another entry for testing, but the program would have to be refactored to return values and we would have to check those values, instead of checking stdout.

    2. When testing crashes/errors, we can't use a test function with assertions as the program will crash after an error.

      Also, we don't have error/exception handling right now and it may take some time before we have the features and standard library functions in place.