Motoko Candid implementation: Run the candid test suite

nomeata commented 4 years ago

There is now a test suite in candid at https://github.com/dfinity/candid/tree/master/test, and we should run it as part of our Motoko test suite.

I think the most plausible way is to write a stand-alone tool that converts these files into files that are understood by the motoko test suite, e.g. turning

assert blob "DIDL\00\01\7e\00" == "(false)" : (bool) "bool: false";
assert blob "DIDL\00\01\7e\00" != blob "DIDL\00\01\7e\01" : (bool) "bool: false";

into a file like

//MOC-ENV MOC_UNLOCK_PRIM=yesplease
import Prim "mo:prim";
ignore((prim "deserialize" : Blob -> Bool) "DIDL\00\01\7e\00");
assert((prim "deserialize" : Blob -> Bool) "DIDL\00\01\7e\00" != (prim "deserialize" : Blob -> Bool) "DIDL\00\01\7e\01");

This way they can run in wasmtime tests.

For don't-parse-tests like

assert blob "DIDL\00\01\7e" !: (bool) "bool: too short";

we have the option of

restructuring Candid parsing in the RTS to not simply trap on invalid data, but exit more gracefully, so that we can have a (prim "deserialize" : Blob -> ?Bool) function. Then we can do these assertions easily in wasm time as well.
putting each trapping test into a separate file (like in test/fail/*.mo).

run them as a drun test and write

actor a {
 public func go() : async () {
    try {await async {
      ignore((prim "deserialize" : Blob -> Bool) "DIDL\00\01\7e");
      Prim.debugPrint("[fail] bool: too short");
    }} catch e {
      Prim.debugPrint("[pass] bool: too short");
    }
    …
  };
}
a.go(); //OR-CALL ingress go "DIDL\x00\x00"

following the idiom in test/run-drun/overflow.mo. This will be kinda slow, with all the extra round-trips, but it’s the best we have to test traps, I fear.

Generate a method for each does-not-parse test, and call them like this:
```
actor a {
  public query func test1(Bool) : async () {
      Prim.debugPrint("[fail] bool: too short");
  };
  …
 }
 //CALL query test1 "DIDL\00\01\7e"
```
This is a bit closer to how we actually use Candid. A downside is that the expected output will be full of drun reject messages, and there is a higher risk that a failing test will go unnoticed.
Instead of producing tests that run with out test/run.sh, produce one test file per test, compile individually, run with wasmtime and programmatically check the output for expected trap or no-trap. This would be closer to how our quick-check based tests work. Probably the most flexible and comfortable, although it might be slow. Maybe we need to compile it once and select tests via a command line argument to the test. This would require extending our WASI support a bit. Can happen later.

Because this tool needs to be able to turn Candid types into Motoko types as per i(…) in https://github.com/dfinity-lab/motoko/blob/master/design/IDL-Motoko.md, and the only place that is currently implemented in is Ocaml, the tool is probably best written in Ocaml.

nomeata commented 4 years ago

Right now I think the last option (i.e. the tool that generates the tests also runs moc and wastime and checks the output) is the best way to go. This would make this more similar to the quickcheck-tests, and less similar to the unit tests. Which is reasonable.

chenyan-dfinity commented 4 years ago

Because this tool needs to be able to turn Candid types into Motoko types as per i(…) in https://github.com/dfinity-lab/motoko/blob/master/design/IDL-Motoko.md, and the only place that is currently implemented in is Ocaml, the tool is probably best written in Ocaml.

i(...) is only for types and easy to implement, so we are not constraint by that. didc (both the OCaml version and Rust version) supports converting Candid types into the host language, seems we need a conversion tool from Candid value to the host language value now.

nomeata commented 4 years ago

Because this tool needs to be able to turn Candid types into Motoko types as per i(…) in https://github.com/dfinity-lab/motoko/blob/master/design/IDL-Motoko.md, and the only place that is currently implemented in is Ocaml, the tool is probably best written in Ocaml.

i(...) is only for types and easy to implement, so we are not constraint by that.

Fair enough. Also needs a pretty-printer for Motoko types, though.

didc (both the OCaml version and Rust version) supports converting Candid types into the host language, seems we need a conversion tool from Candid value to the host language value now.

With “Candid values”, do you mean textual values? If so, then no, we don’t strictly need that (did you see the discussion in this thread at https://github.com/dfinity/candid/pull/71#discussion_r468365078)?

nomeata commented 4 years ago

Preview of some early experiments:

~/dfinity/motoko/src $ candid-tests  -i ../../candid/test/
Parsing overshoot.test.did...
overshoot:13 type table length... ok!
overshoot:14 argument sequence length... ok!
overshoot:15 text length... ok!
overshoot:16 principal length... ok!
overshoot:18 record field number... ok!
overshoot:19 variant field number... ok!
overshoot:25 vector length... ok!
Parsing prim.test.did...
prim:6 empty... ok!
prim:7 no magic bytes... ok!
prim:8 wrong magic bytes... ok!
prim:9 wrong magic bytes... ok!
prim:10 overlong typ table length... ok!
prim:11 overlong arg length... ok!
prim:14...prim:15 nullary: too long... ok!
prim:16 Additional parameters are ignored... ok!
prim:19...prim:20 null: too long... ok!
prim:21 null: missing... ok!
prim:23 bool: false... ok!
prim:24 bool: true... ok!
prim:25 bool: missing... ok!
prim:26 bool: out of range... ok!
prim:27 bool: out of range... ok!
prim:29 nat: 0... ok!
prim:30 nat: 1... ok!
prim:31 nat: 0x7f... ok!
prim:32 nat: leb (two bytes)... ok!
prim:33 nat: leb (two bytes, all bits)... ok!
prim:34 nat: leb too short... ok!
prim:35 nat: leb overlong... ok!
prim:36 nat: leb overlong... ok!
prim:38 int: 0... ok!
prim:39 int: 1... ok!
prim:40 int: -1... ok!
prim:41 int: -64... ok!
prim:42 int: leb (two bytes)... ok!
prim:43 int: leb too short... ok!
prim:44 int: leb overlong (0s)... ok!
prim:45 int: leb overlong (1s)... fail (unexpected pass)!
prim:46 int: leb not overlong when signed... fail (unexpected trap)!
IDL error: not shortest encoding
Error: failed to run main module `tmp.wasm`

Caused by:
    0: failed to invoke `_start`
    1: wasm trap: unreachable, source location: @030f
       wasm backtrace:
         0: <unknown>!rts_trap
         1: <unknown>!trap_with_prefix
         2: <unknown>!idl_trap_with
         3: <unknown>!bigint_sleb128_decode
         4: <unknown>!@deserialize_go<I>
         5: <unknown>!@deserialize<I>
         6: <unknown>
prim:47 int: leb not overlong when signed... ok!
prim:49 nat8: 0... ok!
prim:50 nat8: 1... ok!
prim:51 nat8: 255... fail (unexpected trap)!
IDL error: left-over bytes n8
Error: failed to run main module `tmp.wasm`

Caused by:
    0: failed to invoke `_start`
    1: wasm trap: unreachable, source location: @0442
       wasm backtrace:
         0: <unknown>!@deserialize<n8>
         1: <unknown>!init
         2: <unknown>!_start

prim:52 nat8: too short... ok!
prim:54 nat16: 0... ok!
prim:55 nat16: 1... ok!
prim:56 nat16: 255... fail (unexpected trap)!
IDL error: left-over bytes n16
Error: failed to run main module `tmp.wasm`
prim:57 nat16: 256... ok!
prim:58 nat16: 65535... fail (unexpected trap)!
IDL error: left-over bytes n16
Error: failed to run main module `tmp.wasm`
prim:59 nat16: too short... ok!
prim:60 nat16: too short... ok!
prim:62 nat32: 0... ok!
prim:63 nat32: 1... ok!
prim:64 nat32: 255... fail (unexpected trap)!
IDL error: left-over bytes n32
Error: failed to run main module `tmp.wasm`

Caused by:
    0: failed to invoke `_start`
    1: wasm trap: unreachable, source location: @0443
       wasm backtrace:
         0: <unknown>!@deserialize<n32>
         1: <unknown>!init
         2: <unknown>!_start

prim:65 nat32: 256... ok!
prim:66 nat32: 65535... fail (unexpected trap)!
IDL error: left-over bytes n32
Error: failed to run main module `tmp.wasm`

Caused by:
    0: failed to invoke `_start`
    1: wasm trap: unreachable, source location: @0443
       wasm backtrace:
         0: <unknown>!@deserialize<n32>
         1: <unknown>!init
         2: <unknown>!_start

prim:67 nat32: 4294967295... fail (unexpected trap)!
IDL error: left-over bytes n32
Error: failed to run main module `tmp.wasm`

Caused by:
    0: failed to invoke `_start`
    1: wasm trap: unreachable, source location: @0443
       wasm backtrace:
         0: <unknown>!@deserialize<n32>
         1: <unknown>!init
         2: <unknown>!_start

prim:68 nat32: too short... ok!
prim:69 nat32: too short... ok!
prim:70 nat32: too short... ok!
prim:71 nat32: too short... ok!
prim:73 nat64: 0... ok!
prim:74 nat64: 1... ok!
prim:75 nat64: 255... fail (unexpected trap)!
IDL error: left-over bytes n64
Error: failed to run main module `tmp.wasm`

Caused by:

prim:76 nat64: 256... ok!
prim:77 nat64: 65535... fail (unexpected trap)!
IDL error: left-over bytes n64
Error: failed to run main module `tmp.wasm`
prim:78 nat64: 4294967295... fail (unexpected trap)!
IDL error: left-over bytes n64
Error: failed to run main module `tmp.wasm`
prim:79 nat64: 18446744073709551615... fail (unexpected trap)!
IDL error: left-over bytes n64
Error: failed to run main module `tmp.wasm`

Caused by:
    0: failed to invoke `_start`
    1: wasm trap: unreachable, source location: @0444
       wasm backtrace:
         0: <unknown>!@deserialize<n64>
         1: <unknown>!init
         2: <unknown>!_start

prim:80 nat64: too short... ok!
prim:81 nat64: too short... ok!
prim:82 nat64: too short... ok!
prim:83 nat64: too short... ok!
prim:84 nat64: too short... ok!
prim:85 nat64: too short... ok!
prim:86 nat64: too short... ok!
prim:87 nat64: too short... ok!
prim:90 int8: 0... ok!
prim:91 int8: 1... ok!
prim:92 int8: -1... fail (unexpected trap)!
IDL error: left-over bytes i8
Error: failed to run main module `tmp.wasm`

Caused by:
    0: failed to invoke `_start`
    1: wasm trap: unreachable, source location: @0442
       wasm backtrace:
         0: <unknown>!@deserialize<i8>
         1: <unknown>!init
         2: <unknown>!_start

prim:93 int8: too short... ok!
prim:95 int16: 0... ok!
prim:96 int16: 1... ok!
prim:97 int16: 255... fail (unexpected trap)!
IDL error: left-over bytes i16
Error: failed to run main module `tmp.wasm`

Caused by:
    0: failed to invoke `_start`
    1: wasm trap: unreachable, source location: @0442
       wasm backtrace:
         0: <unknown>!@deserialize<i16>
         1: <unknown>!init
         2: <unknown>!_start

prim:98 int16: 256... ok!
prim:99 int16: -1... fail (unexpected trap)!
IDL error: left-over bytes i16
Error: failed to run main module `tmp.wasm`

Caused by:
    0: failed to invoke `_start`
    1: wasm trap: unreachable, source location: @0442
       wasm backtrace:
         0: <unknown>!@deserialize<i16>
         1: <unknown>!init
         2: <unknown>!_start

prim:100 int16: too short... ok!
prim:101 int16: too short... ok!
prim:103 int32: 0... ok!
prim:104 int32: 1... ok!
prim:105 int32: 255... fail (unexpected trap)!
IDL error: left-over bytes i32
Error: failed to run main module `tmp.wasm`

Caused by:
    0: failed to invoke `_start`
    1: wasm trap: unreachable, source location: @0443
       wasm backtrace:
         0: <unknown>!@deserialize<i32>
         1: <unknown>!init
         2: <unknown>!_start

prim:106 int32: 256... ok!
prim:107 int32: 65535... fail (unexpected trap)!
IDL error: left-over bytes i32
Error: failed to run main module `tmp.wasm`

Caused by:
    0: failed to invoke `_start`
    1: wasm trap: unreachable, source location: @0443
       wasm backtrace:
         0: <unknown>!@deserialize<i32>
         1: <unknown>!init
         2: <unknown>!_start

prim:108 int32: -1... fail (unexpected trap)!
IDL error: left-over bytes i32
Error: failed to run main module `tmp.wasm`

Caused by:
    0: failed to invoke `_start`
    1: wasm trap: unreachable, source location: @0443
       wasm backtrace:
         0: <unknown>!@deserialize<i32>
         1: <unknown>!init
         2: <unknown>!_start

prim:109 int32: too short... ok!
prim:110 int32: too short... ok!
prim:111 int32: too short... ok!
prim:112 int32: too short... ok!
prim:114 int64: 0... ok!
prim:115 int64: 1... ok!
prim:116 int64: 255... fail (unexpected trap)!
IDL error: left-over bytes i64
Error: failed to run main module `tmp.wasm`
prim:117 int64: 256... ok!
prim:118 int64: 65535... fail (unexpected trap)!
IDL error: left-over bytes i64Error: failed to run main module `tmp.wasm`
prim:119 int64: 4294967295... fail (unexpected trap)!
IDL error: left-over bytes i64
Error: failed to run main module `tmp.wasm`
prim:120 int64: -1... fail (unexpected trap)!
IDL error: left-over bytes i64
Error: failed to run main module `tmp.wasm`

Caused by:
    0: failed to invoke `_start`
    1: wasm trap: unreachable, source location: @0444
       wasm backtrace:
         0: <unknown>!
prim:121 int64: too short... ok!
prim:122 int64: too short... ok!
prim:123 int64: too short... ok!
prim:124 int64: too short... ok!
prim:125 int64: too short... ok!
prim:126 int64: too short... ok!
prim:127 int64: too short... ok!
prim:128 int64: too short... ok!
prim:133 text: empty string... ok!
prim:134 text: Motoko... ok!
prim:135 text: too long... ok!
prim:136 text: too short... ok!
prim:137 text: too overlong length leb... ok!
prim:138 text: Unicode... fail (unexpected trap)!
IDL error: UTF-8 validation failure
Error: failed to run main module `tmp.wasm`

Caused by:
    0: failed to invoke `_start`
    1: wasm trap: unreachable, source location: @030f
       wasm backtrace:
         0: <unknown>!rts_trap
         1: <unknown>!trap_with_prefix
         2: <unknown>!idl_trap_with
         3: <unknown>!utf8_validate
         4: <unknown>!@deserialize_go<t>
         5: <unknown>!@deserialize<t>
prim:139 text: Unicode escape... ignored.
prim:140 text: Unicode escape (unclosed)... ignored.
prim:141 text: Invalid utf8... ok!
prim:142 text: Unicode overshoots... ok!
prim:143 text: Escape sequences... ok!
prim:146 reserved from null... compilation failed:
tmp.mo:1.9-1.129: type error, operator not defined for operand types
  Any
and
  Any

prim:147 reserved from bool... compilation failed:
tmp.mo:1.9-1.132: type error, operator not defined for operand types
  Any
and
  Any

prim:148 reserved from nat... compilation failed:
tmp.mo:1.9-1.138: type error, operator not defined for operand types
  Any
and
  Any

prim:149 reserved from text... compilation failed:
tmp.mo:1.9-1.150: type error, operator not defined for operand types
  Any
and
  Any

prim:150 reserved from too short text... ok!
prim:151 reserved from invalid utf8 text... ok!
prim:153 cannot decode empty type... ok!

chenyan-dfinity commented 4 years ago

Preliminary result from Rust

---- decode_test_test_prim_test_did stdout ----
Checking 0:Some("empty")...[pass]
Checking 1:Some("no magic bytes")...[pass]
Checking 2:Some("wrong magic bytes")...[pass]
Checking 3:Some("wrong magic bytes")...[pass]
Checking 4:Some("overlong typ table length")...[Expect fail, but passed!]
Checking 5:Some("overlong arg length")...[Expect fail, but passed!]
Checking 6:None...[pass]
Checking 7:Some("nullary: too long")...[pass]
Checking 8:Some("Additional parameters are ignored")...[pass]
Checking 9:None...[pass]
Checking 10:Some("null: too long")...[pass]
Checking 11:Some("null: missing")...[pass]
Checking 12:Some("bool: false")...[pass]
Checking 13:Some("bool: true")...[pass]
Checking 14:Some("bool: missing")...[pass]
Checking 15:Some("bool: out of range")...[pass]
Checking 16:Some("bool: out of range")...[pass]
Checking 17:Some("nat: 0")...[pass]
Checking 18:Some("nat: 1")...[pass]
Checking 19:Some("nat: 0x7f")...[pass]
Checking 20:Some("nat: leb (two bytes)")...[pass]
Checking 21:Some("nat: leb (two bytes, all bits)")...[pass]
Checking 22:Some("nat: leb too short")...[pass]
Checking 23:Some("nat: leb overlong")...[Expect fail, but passed!]
Checking 24:Some("nat: leb overlong")...[Expect fail, but passed!]
Checking 25:Some("int: 0")...[pass]
Checking 26:Some("int: 1")...[pass]
Checking 27:Some("int: -1")...[pass]
Checking 28:Some("int: -64")...[pass]
Checking 29:Some("int: leb (two bytes)")...[pass]
Checking 30:Some("int: leb too short")...[pass]
Checking 31:Some("int: leb overlong (0s)")...[Expect fail, but passed!]
Checking 32:Some("int: leb overlong (1s)")...[Expect fail, but passed!]
Checking 33:Some("int: leb not overlong when signed")...[pass]
Checking 34:Some("int: leb not overlong when signed")...[pass]
Checking 35:Some("nat8: 0")...[pass]
Checking 36:Some("nat8: 1")...[pass]
Checking 37:Some("nat8: 255")...[pass]
Checking 38:Some("nat8: too short")...[pass]
Checking 39:Some("nat16: 0")...[pass]
Checking 40:Some("nat16: 1")...[pass]
Checking 41:Some("nat16: 255")...[pass]
Checking 42:Some("nat16: 256")...[pass]
Checking 43:Some("nat16: 65535")...[pass]
Checking 44:Some("nat16: too short")...[pass]
Checking 45:Some("nat16: too short")...[pass]
Checking 46:Some("nat32: 0")...[pass]
Checking 47:Some("nat32: 1")...[pass]
Checking 48:Some("nat32: 255")...[pass]
Checking 49:Some("nat32: 256")...[pass]
Checking 50:Some("nat32: 65535")...[pass]
Checking 51:Some("nat32: 4294967295")...[pass]
Checking 52:Some("nat32: too short")...[pass]
Checking 53:Some("nat32: too short")...[pass]
Checking 54:Some("nat32: too short")...[pass]
Checking 55:Some("nat32: too short")...[pass]
Checking 56:Some("nat64: 0")...[pass]
Checking 57:Some("nat64: 1")...[pass]
Checking 58:Some("nat64: 255")...[pass]
Checking 59:Some("nat64: 256")...[pass]
Checking 60:Some("nat64: 65535")...[pass]
Checking 61:Some("nat64: 4294967295")...[pass]
Checking 62:Some("nat64: 18446744073709551615")...[pass]
Checking 63:Some("nat64: too short")...[pass]
Checking 64:Some("nat64: too short")...[pass]
Checking 65:Some("nat64: too short")...[pass]
Checking 66:Some("nat64: too short")...[pass]
Checking 67:Some("nat64: too short")...[pass]
Checking 68:Some("nat64: too short")...[pass]
Checking 69:Some("nat64: too short")...[pass]
Checking 70:Some("nat64: too short")...[pass]
Checking 71:Some("int8: 0")...[pass]
Checking 72:Some("int8: 1")...[pass]
Checking 73:Some("int8: -1")...[pass]
Checking 74:Some("int8: too short")...[pass]
Checking 75:Some("int16: 0")...[pass]
Checking 76:Some("int16: 1")...[pass]
Checking 77:Some("int16: 255")...[pass]
Checking 78:Some("int16: 256")...[pass]
Checking 79:Some("int16: -1")...[pass]
Checking 80:Some("int16: too short")...[pass]
Checking 81:Some("int16: too short")...[pass]
Checking 82:Some("int32: 0")...[pass]
Checking 83:Some("int32: 1")...[pass]
Checking 84:Some("int32: 255")...[pass]
Checking 85:Some("int32: 256")...[pass]
Checking 86:Some("int32: 65535")...[pass]
Checking 87:Some("int32: -1")...[pass]
Checking 88:Some("int32: too short")...[pass]
Checking 89:Some("int32: too short")...[pass]
Checking 90:Some("int32: too short")...[pass]
Checking 91:Some("int32: too short")...[pass]
Checking 92:Some("int64: 0")...[pass]
Checking 93:Some("int64: 1")...[pass]
Checking 94:Some("int64: 255")...[pass]
Checking 95:Some("int64: 256")...[pass]
Checking 96:Some("int64: 65535")...[pass]
Checking 97:Some("int64: 4294967295")...[pass]
Checking 98:Some("int64: -1")...[pass]
Checking 99:Some("int64: too short")...[pass]
Checking 100:Some("int64: too short")...[pass]
Checking 101:Some("int64: too short")...[pass]
Checking 102:Some("int64: too short")...[pass]
Checking 103:Some("int64: too short")...[pass]
Checking 104:Some("int64: too short")...[pass]
Checking 105:Some("int64: too short")...[pass]
Checking 106:Some("int64: too short")...[pass]
Checking 107:Some("text: empty string")...[pass]
Checking 108:Some("text: Motoko")...[pass]
Checking 109:Some("text: too long")...[pass]
Checking 110:Some("text: too short")...[pass]
Checking 111:Some("text: too overlong length leb")...[Expect fail, but passed!]
Checking 112:Some("text: Unicode")...[pass]
Checking 113:Some("text: Unicode escape")...[pass]
Checking 114:Some("text: Unicode escape (unclosed)")...[pass]
Checking 115:Some("text: Invalid utf8")...[pass]
Checking 116:Some("text: Unicode overshoots")...[pass]
Checking 117:Some("text: Escape sequences")...[pass]
Checking 118:Some("reserved from null")...type mismatch: null cannot be of type Reserved
Checking 119:Some("reserved from bool")...type mismatch: null cannot be of type Reserved
Checking 120:Some("reserved from nat")...type mismatch: null cannot be of type Reserved
Checking 121:Some("reserved from text")...type mismatch: null cannot be of type Reserved
Checking 122:Some("reserved from too short text")...[pass]
Checking 123:Some("reserved from invalid utf8 text")...[pass]
Checking 124:Some("cannot decode empty type")...[pass]
thread 'decode_test_test_prim_test_did' panicked at 'called `Result::unwrap()` on an `Err` value:
Message: "114/125 passed"

chenyan-dfinity commented 4 years ago

Preview from JS library

  ✓ 1:empty (96ms)
  ✓ 2:no magic bytes (9ms)
  ✓ 3:wrong magic bytes (5ms)
  ✓ 4:wrong magic bytes (9ms)
  ✓ 5:overlong typ table length (6ms)
  ✓ 6:overlong arg length (12ms)
  ✓ 7 (4ms)
  ✓ 8:nullary: too long (4ms)
  ✓ 9:Additional parameters are ignored (6ms)
  ✓ 10 (5ms)
  ✓ 11:null: too long (39ms)
  ✓ 12:null: missing (4ms)
  ✓ 13:bool: false (4ms)
  ✓ 14:bool: true (12ms)
  ✕ 15:bool: missing (18ms)
  ✕ 16:bool: out of range (9ms)
  ✕ 17:bool: out of range (7ms)
  ✓ 18:nat: 0 (20ms)
  ✓ 19:nat: 1 (5ms)
  ✓ 20:nat: 0x7f (16ms)
  ✓ 21:nat: leb (two bytes) (4ms)
  ✓ 22:nat: leb (two bytes, all bits) (4ms)
  ✕ 23:nat: leb too short (17ms)
  ✓ 24:nat: leb overlong (10ms)
  ✓ 25:nat: leb overlong (5ms)
  ✓ 26:int: 0 (3ms)
  ✓ 27:int: 1 (19ms)
  ✓ 28:int: -1 (7ms)
  ✓ 29:int: -64 (5ms)
  ✓ 30:int: leb (two bytes) (4ms)
  ✕ 31:int: leb too short (5ms)
  ✓ 32:int: leb overlong (0s) (16ms)
  ✓ 33:int: leb overlong (1s) (5ms)
  ✓ 34:int: leb not overlong when signed (5ms)
  ✓ 35:int: leb not overlong when signed (6ms)
  ✓ 36:nat8: 0 (5ms)
  ✓ 37:nat8: 1 (5ms)
  ✓ 38:nat8: 255 (6ms)
  ✕ 39:nat8: too short (30ms)
  ✓ 40:nat16: 0 (8ms)
  ✓ 41:nat16: 1 (7ms)
  ✓ 42:nat16: 255 (7ms)
  ✓ 43:nat16: 256 (4ms)
  ✓ 44:nat16: 65535 (19ms)
  ✕ 45:nat16: too short (5ms)
  ✕ 46:nat16: too short (4ms)
  ✓ 47:nat32: 0 (4ms)
  ✓ 48:nat32: 1 (4ms)
  ✓ 49:nat32: 255 (32ms)
  ✓ 50:nat32: 256 (63ms)
  ✓ 51:nat32: 65535 (17ms)
  ✓ 52:nat32: 4294967295 (7ms)
  ✕ 53:nat32: too short (41ms)
  ✕ 54:nat32: too short (73ms)
  ✕ 55:nat32: too short (5ms)
  ✕ 56:nat32: too short (7ms)
  ✓ 57:nat64: 0 (5ms)
  ✓ 58:nat64: 1 (9ms)
  ✓ 59:nat64: 255 (6ms)
  ✓ 60:nat64: 256 (7ms)
  ✓ 61:nat64: 65535 (6ms)
  ✓ 62:nat64: 4294967295 (6ms)
  ✓ 63:nat64: 18446744073709551615 (100ms)
  ✕ 64:nat64: too short (12ms)
  ✕ 65:nat64: too short (79ms)
  ✕ 66:nat64: too short (5ms)
  ✕ 67:nat64: too short (20ms)
  ✕ 68:nat64: too short (37ms)
  ✕ 69:nat64: too short (5ms)
  ✕ 70:nat64: too short (8ms)
  ✕ 71:nat64: too short (4ms)
  ✓ 72:int8: 0 (4ms)
  ✓ 73:int8: 1 (32ms)
  ✓ 74:int8: -1 (11ms)
  ✕ 75:int8: too short (77ms)
  ✓ 76:int16: 0 (4ms)
  ✓ 77:int16: 1 (4ms)
  ✓ 78:int16: 255 (10ms)
  ✓ 79:int16: 256 (4ms)
  ✓ 80:int16: -1 (4ms)
  ✕ 81:int16: too short (35ms)
  ✕ 82:int16: too short (21ms)
  ✓ 83:int32: 0 (4ms)
  ✓ 84:int32: 1 (4ms)
  ✓ 85:int32: 255 (82ms)
  ✓ 86:int32: 256 (5ms)
  ✓ 87:int32: 65535 (8ms)
  ✓ 88:int32: -1 (4ms)
  ✕ 89:int32: too short (4ms)
  ✕ 90:int32: too short (44ms)
  ✕ 91:int32: too short (4ms)
  ✕ 92:int32: too short (20ms)
  ✓ 93:int64: 0 (4ms)
  ✓ 94:int64: 1 (4ms)
  ✓ 95:int64: 255 (8ms)
  ✓ 96:int64: 256 (4ms)
  ✓ 97:int64: 65535 (66ms)
  ✓ 98:int64: 4294967295 (5ms)
  ✓ 99:int64: -1 (10ms)
  ✕ 100:int64: too short (5ms)
  ✕ 101:int64: too short (7ms)
  ✕ 102:int64: too short (48ms)
  ✕ 103:int64: too short (6ms)
  ✕ 104:int64: too short (5ms)
  ✕ 105:int64: too short (5ms)
  ✕ 106:int64: too short (5ms)
  ✕ 107:int64: too short (5ms)
  ✓ 108:text: empty string (84ms)
  ✓ 109:text: Motoko (16ms)
  ✓ 110:text: too long (5ms)
  ✕ 111:text: too short (17ms)
  ✓ 112:text: overlong length leb (4ms)
  ✓ 113:text: Unicode (4ms)
  ✕ 116:text: Invalid utf8 (4ms)
  ✓ 117:text: Unicode overshoots (5ms)
  ✓ 118:text: Escape sequences (3ms)
  ✓ 125:cannot decode empty type (4ms)

nomeata commented 4 years ago

Nice! Both the many ✓, but also the ✗, as it means that this effort is very useful.

NB: If you use ✗ (ballot crossmark) instead of ✕ (multiplication symbol) it looks better.

chenyan-dfinity commented 4 years ago

Yep, this is indeed useful and uncovered bugs in all candid implementations: Motoko, Rust and JS.

NB: If you use ✗ (ballot crossmark) instead of ✕ (multiplication symbol) it looks better.

This symbol chosen by jest. JavaScript is crazy, btw. It can even catch stack overflow....

nomeata commented 4 years ago

…and in the Haskell one too, I must admit :)

dfinity / motoko

Motoko Candid implementation: Run the candid test suite #1825