crashappsec / libcon4m

Base Compiler and Runtime Support for con4m
Apache License 2.0
0 stars 0 forks source link

Initial test runner #40

Closed viega closed 4 weeks ago

viega commented 4 weeks ago

When you run dev test, if you do not provide an argument, it assumes that it should run over the entire tests directory.

If you do provide arguments, you can provide a list of files or directories.

When you specify a directory, it will be selective about what it runs:

  1. It only looks at files with a '.c4m' extension.
  2. It requires there to be a test spec for the file (see below).
  3. It does recurse provided paths.

As part of supporting this, I added:

c4m_file_kind c4m_get_file_kind(c4m_utf8_t *);

// The keyword args require the syntax of my kw arg implementation, 
// just wanted to express more naturally. e.g., kw("recurse", c4m_ka(true),) 

c4m_xlist_t *
c4m_path_walk(c4m_utf8_t *, bool recurse = true, bool yield_links = false,
              bool yield_dirs = false, bool ignore_special = true,
              bool follow_links = false);

The former returns one of these constants:

typedef enum {
    C4M_FK_NOT_FOUND       = 0,
    C4M_FK_IS_REG_FILE     = S_IFREG,
    C4M_FK_IS_DIR          = S_IFDIR,
    C4M_FK_IS_FLINK        = S_IFLNK,
    C4M_FK_IS_DLINK        = S_IFLNK | S_IFDIR,
    C4M_FK_IS_SOCK         = S_IFSOCK,
    C4M_FK_IS_CHR_DEVICE   = S_IFCHR,
    C4M_FK_IS_BLOCK_DEVICE = S_IFBLK,
    C4M_FK_IS_FIFO         = S_IFIFO,
    C4M_FK_OTHER           = ~0,
} c4m_file_kind;

Also, the harness looks for two environment variables:

  1. The tremendous amount of debug info now only shows up if you have CON4M_DEV set (ignores value).
  2. The default search location is pulled from CON4M_TEST_DIR if provided.

The test specification goes in the SECOND of the two doc strings in a con4m file. If you provide an explicit file name and it does not have a second doc string, it is compiled and run, but not treated as a test case.

The test runner currently allows two possible items in the test spec (both may exist):

  1. $output: Matches the basic output from the builtin print. The builtin print is meant to be temporary while developing though; once I have kw args done in the language, I'll re-implement it. And I also will, at that point, change the runner to use my subprocess library.
  2. $errors: Matches against an ordered list of expected error codes.

$errors: is always checked, and if you leave it out, the runner assumes it should see no errors/warnings. $output: is also optional; if you leave it out, the output will not be checked. Eventually I may add some pattern matching here instead. But it's good enough for now.

For tests that don't output, but that should be checked (i.e., they output error info if there's an error), you can specify $output: with nothing after it.

The output and the contents of each of the two spec items get stripped before comparison.

Fixes: https://github.com/crashappsec/libcon4m/issues/37