How does look like the test runner?

amirouche commented 5 years ago

What I like in test runner:

Automatic discovery of tests, easy with the lib-name/test.scm convention.
Run tests for a single library, something like $ ./test-run lib-name/test.scm
Run a single test procedure, something like $ ./test-run lib-name/test.scm test-0000
Do not capture output OR provide a way to print the captured output
Return an exit code bigger than 0 when one or more tests fail
Very simple assert primitives like gambit _test procedures check-equal? and check-tail-exn (unlike python's unittest or junit).
Flag to "fail fast" or "until the end". In development mode, one might prefer "fail fast" where the test runner will stop at the first failure. Unlike in CI, where we would prefer to have the full report.

amirouche commented 5 years ago

What I was using in my previous work, is a macro that defines a thunk:

    (define-syntax-rule (test expected actual)
      (lambda ()
        (let ((expected* expected)
              (actual* actual))
          (if (equal? expected* actual*)
              (list #t)
              (list #f expected* actual*)))))

Then in the test library it would be used as follow:

(library (tests-tests)

  (export test-0)

  (import (chezscheme)
          (tests))

  (define test-0
    (test #t #t)))

The test runner will look up all files that ends with tests.scm and run all the thunk of the interface of those modules. This requires to be able to lookup the public interface of a library.

amirouche commented 5 years ago

As far as I know, SRFI-64 is not used much in SRFI for R7RS-large. Most (if not all) rely on chibi's test macro which has two forms:

(test "description of the test" expected actual)

Or without a description:

(test expected actual)

A given test suite for an SRFI is wrapped inside a procedure run-tests that is imported in a run-tests.scm that is executed with make. I assume they hardcode the test suite because there is no library introspection.

amirouche commented 5 years ago

Here is the test runner in gambit: https://github.com/udem-dlteam/gambit/blob/master/tests/run-unit-tests.scm

amirouche commented 5 years ago

My proposal is to rely on the library system to allow all the features I listed in the issue. This is a proposal that will change how the tests look, how the original tests are reworked and also must stand the test of time.

It will require to create a replacement for check-equal? and check-tail-exn. For the latter, I used to rely on R7RS guard, so a test that wants to check that a given exn condition is raised will look like the following:

(define test-0
    (test #t (guard (ex ((condition? ex) #t) (else #f)) 
      (something)))

There is a problem if (something) returns #t, so we might need to add some sugar test-exception to replace test in that case.

Where test is the following macro:

    (define-syntax-rule (test expected actual)
      (lambda ()
        (let ((expected* expected)
              (actual* actual))
          (if (equal? expected* actual*)
              (list #t)
              (list #f expected* actual*)))))

@lassik @feeley Do you see any interest in my proposal to use the library system for creating the unit tests? Or do you prefer to keep tests as much as the original SRFI provide? Or something else?

amirouche commented 5 years ago

By the way, I could not find the definition of check-equal? and check-tail-exn in gambit source. Any help?

amirouche commented 5 years ago

Also, how can I query the public interface of a library? And retrieve the exported forms?

feeley commented 5 years ago

By the way, I could not find the definition of check-equal? and check-tail-exn in gambit source. Any help?

They are in the _test module (the source code of the macros is in lib/_test/_test#.scm). You get these macros:

check-equal?
check-not-equal?
check-eqv?
check-not-eqv?
check-eq?
check-not-eq?

check-=

check-true
check-false
check-not-false

check-exn
check-tail-exn

I'd like to keep tests simple and uniform across all modules. I previously had a -test switch to the interpreter to trigger unit tests, but I simplified by adopting the convention that the unit tests for module foo are in its test submodule. For example, to run the unit tests of the _hamt module you can do:

gsi _hamt/test

Which silently runs all the tests and exits with status 0. If the test submodule did not exist an error would be signalled (so you know the tests were run if you have no error). This is rather simple, but other options could be added to the _test module to add a summary of the traces, stop at the first failure, etc. This could be done by adding appropriate submodules for these options to the _test module. For example, to stop at the first failure there could ge a fail-fast submodule:

gsi _test/fail-fast _hamt/test

The names of these options should be carefully chosen, and also the default behaviour. For example, I think the default should stop at the first failure and otherwise print the number of tests run if there are no failures, and there should be options for quiet and all.

For your other question, to check if a module exists you could use this code (I'm considering adding it to lib/_module.scm):

(define (module-reference string-or-source)
  (let ((modref (##parse-module-ref string-or-source)))
    (if modref
        (##string->symbol (##modref->string modref))
        (error "invalid module reference" string-or-source))))

(define (module-exists? string-or-source)
  (let ((module-ref (module-reference string-or-source)))
    (with-exception-catcher
     (lambda (exc)
       (if (module-not-found-exception? exc)
           #f
           (raise exc)))
     (lambda ()
       (##get-module module-ref)
       #t))))

(pp (module-exists? '(_test))) ;;=> #t

(pp (module-exists? '(scheme fu))) ;;=> #f

(pp (module-exists? '(_define-library define-library-expand))) ;;=> #t

(pp (module-exists? '(github.com/gambit/hello test @2.1))) ;;=> #t

(pp (module-exists? '(github.com/gambit/hello test @2.0))) ;;=> #f

(pp (module-exists? "github.com/gambit/hello/test/@2.1")) ;;=> #t

That could be used to scan for modules that contain a test submodule.

Because the module system is implemented as a macro expanding to namespace and include forms, there is no easy way to determine what is exported by a module. If you limit yourself to R7RS libraries, then you could probably reuse the define-library parser in the _define-library/define-library-expand module.

feeley commented 5 years ago

I have pushed some changes to the _test module to

1) by default give a final report of the tests run and stop at the first failure 2) if _test/all is loaded then keep going after failures 3) if _test/quiet is loaded then don't show final report 3) if _test/verbose is loaded then show pass/fail message for each test

For example:

% gsi _test/quiet github.com/gambit/hello/test@2.1
% gsi github.com/gambit/hello/test@2.1
*** all tests passed out of a total of 4 tests
% gsi _test/verbose github.com/gambit/hello/test@2.1
"../../.gambit_userlib/github.com/gambit/hello/@2.1/test.scm"@10.5: PASSED (check-equal? (with-output-to-string (lambda () (hi "you"))) "hello you!\n")
"../../.gambit_userlib/github.com/gambit/hello/@2.1/test.scm"@13.5: PASSED (check-equal? (with-output-to-string (lambda () (salut "hi"))) "bonjour hi!\n")
"../../.gambit_userlib/github.com/gambit/hello/@2.1/test.scm"@16.5: PASSED (check-tail-exn wrong-number-of-arguments-exception? (lambda () (hi)))
"../../.gambit_userlib/github.com/gambit/hello/@2.1/test.scm"@19.5: PASSED (check-tail-exn wrong-number-of-arguments-exception? (lambda () (hi "foo" "bar")))
*** all tests passed out of a total of 4 tests

amirouche commented 4 years ago

The tests should rely on gambit's (_test) module

I will close the issue when I have implemented some tests.

amirouche commented 4 years ago

I think srfi 175 is a good example of how things look like (once https://github.com/udem-dlteam/libs/pull/11/ is merged)

feeley commented 4 years ago

I advise against using want as in srfi 175 for unit tests because:

1) The name want does not convey how the check is performed (numerical equality, eq?, equal?, ...)

2) The second parameter of want (i.e. check-equal?) should be the expected value. The order is important because the second parameter is shown in the error message.

lassik commented 4 years ago

The test is a direct copy of the sample implementation, just with a few things added at the top of the file for Gambit. I can consider changing want in the sample implementation. But first there's a more fundamental problem that prevents the current test from working at all in Gambit. I don't understand why it doesn't. PR #11 should adapt the tests for Gambit but no luck yet.

feeley commented 4 years ago

The tests work for me when I do this:

gsi/gsi -:=. . srfi/175/test

Also, you should write the unit tests as a module (e.g. test.sld) with a relative reference to srfi 175, like this:

(define-library (test)
  (import (..)) ;; import parent = srfi 175
  (import (scheme base)) ;; import define, lambda, etc
  (import (_test)) ;; import check-equal?, etc
  (begin

(define-syntax want (syntax-rules () ((_ x y) (check-equal? x y))))

(want #f (ascii-codepoint? -1))
(want #t (ascii-codepoint? 0))
(want #t (ascii-codepoint? #x7f))
(want #f (ascii-codepoint? #x80))
...

That way, if you ever start using versions for SRFI 175 the test module will import the 175 module with the same version:

gsi srfi/175/test@2.0

lassik commented 4 years ago

Works for me too, I just had a stale build of Gambit for some reason.

Should the define-library go into a separate file srfi/175/test.sld which includes srfi/175/test.scm, i.e. the usual R7RS convention?

I've been thinking it could be really useful to have a big repo with up-to-date tests for all (or many) SRFIs, shared by all implementations. That way we can all leverage each other's work. The tests in each SRFI's own sample implementation are not always as full-featured as the ones that some implementations have.

The main problem here would be agreeing on a test runner API that all implementations use for SRFI tests (of course, it can be implemented as a macro that expands to call the implementation's native framework). As for the tests themselves, we'd just have to scavenge implementations' repos and merge it all into a common repo. All of that code should be pretty liberally licensed.

lassik commented 4 years ago

With a common test framework there is also the concern of different R5RS/R6RS/R7RS library filename and Scheme declaration conventions, but I've found it's surprisingly easy to write an auto-converter to do that.

amirouche commented 4 years ago

For what it is worth, that is somewhat what started working on in the repository at https://github.com/amirouche/nomunofu/issues/3.

I have guile and chez on board with a handful of missing SRFI for R7RS large. The test runner does not do autodiscovery, one must pass the filenames of the library containing the tests to see https://github.com/amirouche/nomunofu/blob/wip-portable/Makefile#L170.

Autodiscovery is no brainer once the following is resolved:

I tried to reproduce a similar behavior with gambit, but I fail to express the following Chez code:

    (define (library-exports* filepath library-name)
      ;; return the procedure defined in library LIBRARY-NAME at FILEPATH
      ;; XXX: hackish at best, there might be a better solution
      (let ((env (interaction-environment)))
        (let ((program `(begin
                          (import ,library-name)
                          (let ((exports (library-exports ',library-name)))
                            exports))))
          (let ((exports (eval program env)))
            (let ((program `(begin
                              (import ,library-name)
                              (map cons ',exports (list ,@exports)))))
              (reverse (eval program env)))))))

Similar Guile code:

(define (run filename)
  (let ((module-name (filename->module-name filename)))
    (display "* ")
    (display module-name)
    (display " @ ")
    (display filename)
    (newline)
    (let ((module (resolve-interface module-name))
          (exports '()))
      (module-for-each
       (lambda (name variable)
         (set! exports (cons (cons name variable) exports)))
       module)
      (set! exports
            (sort exports (lambda (a b) (string<? (symbol->string (car a))
                                                  (symbol->string (car b))))))
      (for-each (lambda (x) (run-one (car x) (cdr x))) exports))))

The important procedure are respectively (library-exports library-name) and (resolve-interface module-name) when passed the '(path to library)` will return the exported symbols (or an object referencing those exported symbols).

Using gambit I got to this point:

$ cat local/bin/run-gambit 
#!/bin/sh

gsi -:r7rs $(pwd)/src/ $@

$ run-gambit 
Gambit v4.9.3

> (import (gambit))                                                                                                          
> (define-values (ld import-name) (_define-library/define-library-expand#get-libdef 'check '(arew check test))) 
> ld
#(#(source1)
  (#(#(source1) import "/home/amirouche/src/scheme/nomunofu/src/check.scm" 65536)
   #(#(source1)
     (#(#(source1) arew "/home/amirouche/src/scheme/nomunofu/src/check.scm" 589824) #(#(source1) check "/home/amirouche/src/scheme/nomunofu/src/check.scm" 917504))
     "/home/amirouche/src/scheme/nomunofu/src/check.scm"
     524288))
  "/home/amirouche/src/scheme/nomunofu/src/check.scm"
  0)
>

What I want is the module test inside (arew check):

$ cat src/arew/check/test.sld 
(define-library (arew check test)

  (export test-0)

  (import (gambit)
          (scheme base)
          (arew check))

  (include "test.body.scm"))

When I do the following I have a segfault:

> (define-values (ld import-name) (_define-library/define-library-expand#get-libdef '(arew check test) ""))  
Segmentation fault (core dumped)

amirouche commented 4 years ago

I am using

commit 1b373b07814669e6051034d58ce24f3da5541fcd (HEAD -> master, origin/master, origin/HEAD)

lassik commented 4 years ago

For what it is worth, that is somewhat what started working on in the repository at amirouche/nomunofu#3.I have guile and chez on board with a handful of missing SRFI for R7RS large.

Very good progress mapping out the library support!

There should probably be a portable "all SRFI tests" repo separate from the SRFI implementations, since each Scheme implementation may want to have its own implementation of a SRFI, whereas the tests only test the public interface which ought to be the same in all Scheme implementations. So the tests are much more portable.

Of course, there's a portable implementation of many SRFIs as well, but those are not necessarily optimized for a particular implementation.

lassik commented 4 years ago

I created a test repo to try how it would work: https://github.com/srfi-explorations/srfi-test We can scrap it if we figure out that it's a bad idea.

amirouche commented 4 years ago

I tried to reproduce a similar behavior with gambit, but I fail to express the following Chez code:

Nevermind, parsing the `define-library is trivial.

udem-dlteam / libs

How does look like the test runner? #5