running tests with `runt` generates test failures

[CrepeGoat]

system

• M1 macbook
• macOS 14.5
• cargo 1.80.1 (376290515 2024-07-16)
• rustc 1.80.1 (3f5fd8dd4 2024-08-06)
• Filament repo commit bb771f1b470b31178b68d238844995de159b7f30
• runt v0.4.1 (commit 2448532b6f4501e5fc55f2762a9e7d3a7c529318)

problem

running runt in the main filament directory, as recommended for running Filament tests, produces the following error report:

$ cargo build
...
$ runt
- errors:tests/errors/sharing/conflicting-use.fil
- errors:tests/errors/sharing/dynamic-share.fil
✗ check:tests/check/branched-implication.fil
✗ check:tests/check/borrow.fil
✗ check:tests/check/sig-bind-inst.fil
✗ check:tests/check/sig-bind-port.fil
✗ check:tests/check/exist-rec.fil
✗ check:tests/check/exist-sig-assume.fil
✗ check:tests/check/if-expr.fil
✗ check:tests/check/merge.fil
✗ check:tests/check/resolve-expr-first.fil
✗ check:tests/check/scope-invoke-check.fil
✗ check:tests/check/when.fil
✗ check:primitives/comb.fil
✗ check:primitives/core.fil
✗ check:primitives/reshape.fil
✗ check:primitives/signed.fil
✗ errors:tests/errors/assign-check/missing-large.fil
✗ errors:tests/errors/assign-check/multiple.fil
✗ errors:tests/errors/bundle/bundle-delay.fil
✗ errors:tests/errors/bundle/bundle-mismatch.fil
✗ errors:tests/errors/bundle/multi-len.fil
✗ errors:tests/errors/bundle/multi-liveness.fil
✗ errors:tests/errors/existential/body.fil
✗ errors:tests/errors/existential/sig.fil
✗ errors:tests/errors/existential/some-param-trans.fil
✗ errors:tests/errors/sharing/share-range.fil
✗ errors:tests/errors/typecheck/borrow.fil
✗ errors:tests/errors/typecheck/invalid-interface-input.fil
- errors:tests/errors/well-formed/user-level-constraint.fil
✗ errors:tests/errors/typecheck/invoke-trig-too-often.fil
✗ errors:tests/errors/typecheck/invalid-interface-output.fil
✗ check:primitives/state.fil
✗ errors:tests/errors/assign-check/missing.fil
✗ errors:tests/errors/existential/multi-assign.fil
✗ errors:tests/errors/bundle/bundle.fil
✗ errors:tests/errors/typecheck/param-width.fil
✗ errors:tests/errors/typecheck/poly-mismatch.fil
✗ errors:tests/errors/typecheck/port-mismatch.fil
✗ errors:tests/errors/typecheck/sig-errors.fil
✗ errors:tests/errors/typecheck/sig-constraints.fil
✗ errors:tests/errors/typecheck/unprovable-cons.fil
✗ errors:tests/errors/typecheck/unsatisfied-guarantee.fil
✗ errors:tests/errors/typecheck/unsatisfied-requirement.fil
✗ run:tests/run/add.fil
✗ run:tests/run/assume-log2.fil
✗ run:tests/run/assume-pow2.fil
✗ run:tests/run/bundle-mono.fil
✗ run:tests/run/comb.fil
✗ run:tests/run/concat-bundle.fil
✗ run:tests/run/deserialize.fil
✗ run:tests/run/even-odd.fil
✗ run:tests/run/event-multiple.fil
✗ run:tests/run/exist.fil
✗ run:tests/run/fsm-guard.fil
✗ run:tests/run/misaligned-reuse.fil
✗ run:tests/run/mult-chain.fil
✗ run:tests/run/multi-arg-fn.fil
✗ run:tests/run/multi-assign.fil
✗ run:tests/run/param-default.fil
✗ run:tests/run/param-if.fil
✗ run:tests/run/param-let-loop.fil
✗ run:tests/run/param-let.fil
✗ run:tests/run/param-mac.fil
✗ run:tests/run/param-moving-sum.fil
✗ run:tests/run/pass-through-reg.fil
✗ run:tests/run/phantom-pipeline.fil
✗ run:tests/run/reduction-tree.fil
✗ run:tests/run/pipeline-with-ii-1.fil
✗ run:tests/run/serialize.fil
✗ run:tests/run/split-wire.fil
✗ run:tests/run/window-sum.fil
✗ floating point-edge:tests/floating-point/add/harness.fil
✗ floating point-edge:tests/floating-point/mul/harness.fil
✗ iterative divider-edge:tests/iterative-divider/harness.fil
✗ systolic array:tests/systolic/fast-mult.fil
✗ iterative divider-random:tests/iterative-divider/harness.fil
✗ bindings:tests/gen/fp-add/add_800.toml
✗ systolic array:tests/systolic/simple.fil
✗ bindings:tests/gen/fp-add/add_double.toml
✗ aetherling parameteric:apps/aetherling/conv2d_1/shim.fil
✗ bindings:tests/gen/shift/5.toml
✗ bindings:tests/gen/fp-add/add_200.toml
✗ bindings:tests/gen/shift/6.toml
✗ aetherling parameteric:apps/aetherling/conv2d_144/shim.fil
✗ aetherling parameteric:apps/aetherling/conv2d_16/shim.fil
✗ aetherling parameteric:apps/aetherling/conv2d_48/shim.fil
✗ aetherling parameteric:apps/aetherling/conv2d_8/shim.fil
✗ floating point-random:tests/floating-point/mul/harness.fil
✗ floating point-random:tests/floating-point/add/harness.fil
✗ frisc tests:apps/frisc/test/alu.fil
✗ frisc tests:apps/frisc/test/cpu.fil
✗ frisc tests:apps/frisc/test/decode.fil
✗ frisc tests:apps/frisc/test/pc.fil
✗ frisc tests:apps/frisc/test/regfile.fil
✗ blas:apps/blas/dot/test.fil
✗ blas:apps/blas/axpy/test.fil
? blas:apps/blas/rot/test.fil
✗ blas:apps/blas/scal/test.fil
? blas:apps/blas/syr/test.fil
✗ aetherling parameteric:apps/aetherling/conv2d_2/shim.fil
✗ aetherling parameteric:apps/aetherling/conv2d_4/shim.fil
✗ run examples:examples/tut-pipe.fil
✗ run examples:examples/tut-seq.fil
✗ fft-random:apps/fft/versions/filament/fft.fil
✗ fft-edge:apps/fft/versions/filament/fft.fil
✗ fft-random:apps/fft/versions/flopoco/fft.fil
✗ examples:examples/sequential.fil
✗ examples:examples/tut-seq.fil
✗ examples:examples/tut-pipe-wrong-2.fil
✗ examples:examples/tut-wrong-2.fil
✗ examples:examples/tut-pipe-wrong-1.fil
✗ examples:examples/tut-pipe.fil
✗ fft-edge:apps/fft/versions/flopoco/fft.fil
✗ examples:examples/tut-wrong-1.fil
✗ examples:examples/tut-pipe-wrong-3.fil
 20 passing / 111 failing / 2 missing / 3 skipped / 0 remaining

[rachitnigam]

Can you run it with runt -d to see what the errors are? I suspect you don't have your environment configured correctly.

[CrepeGoat]

reran with -d, lots of output but here's one excerpt that looks pretty similar to the rest:

✗ errors:tests/errors/assign-check/missing-large.fil
         ~
    1   1│  ---CODE---
    2    │- 1
        2│+ 101
    3   3│  ---STDERR---
    4    │- error: bundle `a' has 99 unassigned locations
    5    │-   ┌─ tests/errors/assign-check/missing-large.fil:2:12
    6    │-   │
    7    │- 2 │     bundle a[100]: for<k> ['G, 'G+1] 32;
    8    │-   │            ^ bundle indices are unassigned: 1, 2, 3, 4, 5, ... and 94 others
        4│+ thread 'main' panicked at crates/filament/src/ir_passes/discharge.rs:131:14:
        5│+ called `Result::unwrap()` on an `Err` value: Os { code: 2, kind: NotFound, message: "No such file or directory" }
        6│+ note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
    9   7│  
   10    │- Compilation failed with 1 errors.
   11    │- Run with --show-models to generate assignments for failing constraints.
   12    │- 
         ~

lmk if you want the full output. full output here: https://gist.github.com/CrepeGoat/4ce12c51e1ee5440056aeed48f908f6b

[rachitnigam]

You probably don't have z3 installed. For any test, you can do:

runt -i tests/errors/assign-check/missing-large.fil -n

to see the command being run.

Run the command yourself and see if you can get more information that way?

[CrepeGoat]

that fixed some, but I'm still getting errors like this:

✗ run:tests/run/add.fil
         ~
    1    │- {"out": {"0": [10], "1": [257], "2": [31], "3": [43]}, "cycles": 4}
    2    │- 
    3    │- 
        1│+ ---CODE---
        2│+ 127
        3│+ ---STDERR---
        4│+ sh: line 1: fud: command not found
        5│+ 
         ~

I updated the previous gist with the most recent errors.

[CrepeGoat]

(thanks for the dependency install fix though, I'm a little embarrassed I made that mistake 😅)

[rachitnigam]

You have to install fud as described in the full build installation instructions.