update to nom 4

[Geal]

Hello,

I'm currently testing the upgrade to nom 4 on various crates, to see if I forgot anything. Currently, I get this error after doing the following changes:

error[E0308]: if and else have incompatible types
  --> tests/clang.rs:67:15
   |
67 |       let result = if functional {
   |  __________________^
68 | |         let mut fnidents;
69 | |         let expr_tokens;
70 | |         match fn_macro_declaration(&tokens) {
...  |
90 | |         IdentifierParser::new(idents).macro_definition(&tokens).map(|(_,val)|val)
91 | |     };
   | |_____^ expected struct `cexpr::token::Token`, found u8
   |
   = note: expected type `std::result::Result<(&[cexpr::token::Token], cexpr::expr::EvalResult), cexpr::nom::Err<&[cexpr::token::Token], _>>`
              found type `std::result::Result<(&[u8], cexpr::expr::EvalResult), cexpr::nom::Err<&[cexpr::token::Token], _>>`

error: aborting due to previous error

error: Could not compile `cexpr`.

For a bit of context, here is the upgrade doc for nom 4.

[jethrogb]

The map at the end of line 90 used to turn IResult<&'a [Token], (&'a [u8], EvalResult), Error> into IResult<&'a [Token], EvalResult, Error>. I think you now need to call .map(|(i, (_,val))|(i, val)) to do the same. When I make that change, the test compiles, but there are a lot of failures.

[Geal]

Can you show me the errors? They're probably linked to the new behaviour around Incomplete, that should be easy to fix

[jethrogb]

---- int_signed stdout ----
    Failed test for Int_n3, expected Int(-3), got Err(Error(Code([Token { kind: Punctuation, raw: [45] }, Token { kind: Punctuation, raw: [40] }, Token { kind: Punctuation, raw: [45] }, Token { kind: Punctuation, raw: [40] }, Token { kind: Punctuation, raw: [45] }, Token { kind: Literal, raw: [51] }, Token { kind: Punctuation, raw: [41] }, Token { kind: Punctuation, raw: [41] }], Alt)))
Failed test for Int_n5, expected Int(-5), got Err(Error(Code([Token { kind: Punctuation, raw: [45] }, Token { kind: Literal, raw: [51] }, Token { kind: Punctuation, raw: [45] }, Token { kind: Literal, raw: [50] }], Alt)))
Failed test for Int_n9223372036854775808, expected Int(-9223372036854775808), got Err(Error(Code([Token { kind: Punctuation, raw: [45] }, Token { kind: Literal, raw: [57, 50, 50, 51, 51, 55, 50, 48, 51, 54, 56, 53, 52, 55, 55, 53, 56, 48, 56] }], Alt)))
thread 'int_signed' panicked at 'test_file', tests/clang.rs:244:0
note: Run with `RUST_BACKTRACE=1` for a backtrace.

---- fail stdout ----
    Failed test for Int_0, expected Int(0), got Err(Error(Code([Token { kind: Literal, raw: [48] }], Alt)))
Failed test for Str_str, expected Str([115, 116, 114]), got Err(Incomplete(Size(1)))
thread 'fail' panicked at 'test_file', tests/clang.rs:246:0

---- floats stdout ----
    Failed test for Float_0, expected Float(0), got Err(Error(Code([Token { kind: Literal, raw: [48, 46] }], Alt)))
Failed test for Float_1, expected Float(1), got Err(Incomplete(Size(1)))
Failed test for Float_p1, expected Float(0.1), got Err(Error(Code([Token { kind: Literal, raw: [46, 49] }], Alt)))
Failed test for Float_2, expected Float(2), got Err(Error(Code([Token { kind: Literal, raw: [50, 46, 48] }], Alt)))
Failed test for Float_1000, expected Float(1000), got Err(Error(Code([Token { kind: Literal, raw: [49, 101, 51] }], Alt)))
Failed test for Float_2000, expected Float(2000), got Err(Error(Code([Token { kind: Literal, raw: [50, 101, 43, 51] }], Alt)))
Failed test for Float_p001, expected Float(0.001), got Err(Error(Code([Token { kind: Literal, raw: [49, 101, 45, 51] }], Alt)))
Failed test for Float_80, expected Float(80), got Err(Error(Code([Token { kind: Literal, raw: [49, 48, 46, 48] }, Token { kind: Punctuation, raw: [42] }, Token { kind: Punctuation, raw: [40] }, Token { kind: Literal, raw: [49] }, Token { kind: Punctuation, raw: [60, 60] }, Token { kind: Literal, raw: [51] }, Token { kind: Punctuation, raw: [41] }], Alt)))
thread 'floats' panicked at 'test_file', tests/clang.rs:241:0

---- strings stdout ----
    Failed test for Str_, expected Str([]), got Err(Incomplete(Size(1)))
Failed test for Str_str, expected Str([115, 116, 114]), got Err(Incomplete(Size(1)))
Failed test for Str_unicode, expected Str([117, 110, 105, 99, 111, 100, 101]), got Err(Incomplete(Size(1)))
Failed test for Str_long, expected Str([108, 111, 110, 103]), got Err(Incomplete(Size(1)))
Failed test for Str_concat, expected Str([99, 111, 110, 99, 97, 116]), got Err(Incomplete(Size(1)))
Failed test for Str_concat_identifier, expected Str([99, 111, 110, 99, 97, 116, 95, 105, 100, 101, 110, 116, 105, 102, 105, 101, 114]), got Err(Error(Code([Token { kind: Punctuation, raw: [40] }, Token { kind: Identifier, raw: [83, 116, 114, 95, 99, 111, 110, 99, 97, 116] }, Token { kind: Literal, raw: [76, 34, 95, 105, 100, 101, 110, 116, 105, 102, 105, 101, 114, 34] }, Token { kind: Punctuation, raw: [41] }], Alt)))
Failed test for Fn_Str_no_args, expected Str([110, 111, 95, 97, 114, 103, 115]), got Err(Incomplete(Size(1)))
Failed test for Fn_Str_no_args_concat, expected Str([110, 111, 95, 97, 114, 103, 115, 95, 99, 111, 110, 99, 97, 116]), got Err(Error(Code([Token { kind: Identifier, raw: [83, 116, 114, 95, 99, 111, 110, 99, 97, 116] }], Custom(Partial))))
Failed test for Fn_Str_prepend_arg, expected Str([112, 114, 101, 112, 101, 110, 100, 95, 97, 114, 103]), got Err(Incomplete(Size(1)))
Failed test for Fn_Str_two_args, expected Str([116, 119, 111, 95, 97, 114, 103, 115]), got Err(Incomplete(Size(1)))
Failed test for Fn_Str_three_args, expected Str([116, 104, 114, 101, 101, 95, 97, 114, 103, 115]), got Err(Incomplete(Size(1)))
thread 'strings' panicked at 'test_file', tests/clang.rs:243:0

---- int_unsigned stdout ----
    Failed test for Int_456, expected Int(456), got Err(Error(Code([Token { kind: Literal, raw: [52, 53, 54] }], Alt)))
Failed test for Int_0, expected Int(0), got Err(Error(Code([Token { kind: Literal, raw: [48] }], Alt)))
Failed test for Int_1, expected Int(1), got Err(Error(Code([Token { kind: Literal, raw: [48, 98, 49] }], Alt)))
Failed test for Int_2, expected Int(2), got Err(Error(Code([Token { kind: Literal, raw: [48, 120, 50] }], Alt)))
Failed test for Int_63, expected Int(63), got Err(Error(Code([Token { kind: Literal, raw: [48, 55, 55] }], Alt)))
Failed test for Int_123, expected Int(123), got Err(Error(Code([Token { kind: Literal, raw: [49, 50, 51] }], Alt)))
Failed test for Int_124, expected Int(124), got Err(Error(Code([Token { kind: Literal, raw: [49, 50, 52, 117] }], Alt)))
Failed test for Int_125, expected Int(125), got Err(Error(Code([Token { kind: Literal, raw: [49, 50, 53, 117, 76] }], Alt)))
Failed test for Int_126, expected Int(126), got Err(Error(Code([Token { kind: Literal, raw: [49, 50, 54, 76, 117, 76] }], Alt)))
Failed test for Int_16, expected Int(16), got Err(Error(Code([Token { kind: Punctuation, raw: [40] }, Token { kind: Punctuation, raw: [40] }, Token { kind: Punctuation, raw: [40] }, Token { kind: Literal, raw: [49] }, Token { kind: Punctuation, raw: [41] }, Token { kind: Punctuation, raw: [60, 60] }, Token { kind: Literal, raw: [52, 85, 76, 76] }, Token { kind: Punctuation, raw: [41] }, Token { kind: Punctuation, raw: [41] }], Alt)))
Failed test for Int_13, expected Int(13), got Err(Error(Code([Token { kind: Literal, raw: [49] }, Token { kind: Punctuation, raw: [124] }, Token { kind: Literal, raw: [56] }, Token { kind: Punctuation, raw: [94] }, Token { kind: Literal, raw: [54] }, Token { kind: Punctuation, raw: [38] }, Token { kind: Literal, raw: [50] }, Token { kind: Punctuation, raw: [60, 60] }, Token { kind: Literal, raw: [49] }], Alt)))
Failed test for Int_47, expected Int(47), got Err(Error(Code([Token { kind: Literal, raw: [51, 50] }, Token { kind: Punctuation, raw: [124] }, Token { kind: Literal, raw: [49, 53] }], Alt)))
Failed test for Int_38, expected Int(38), got Err(Error(Code([Token { kind: Punctuation, raw: [40] }, Token { kind: Literal, raw: [51, 50] }, Token { kind: Punctuation, raw: [124] }, Token { kind: Literal, raw: [49, 53] }, Token { kind: Punctuation, raw: [41] }, Token { kind: Punctuation, raw: [94] }, Token { kind: Literal, raw: [57] }], Alt)))
Failed test for Int_6, expected Int(6), got Err(Error(Code([Token { kind: Punctuation, raw: [40] }, Token { kind: Punctuation, raw: [40] }, Token { kind: Literal, raw: [51, 50] }, Token { kind: Punctuation, raw: [124] }, Token { kind: Literal, raw: [49, 53] }, Token { kind: Punctuation, raw: [41] }, Token { kind: Punctuation, raw: [94] }, Token { kind: Literal, raw: [57] }, Token { kind: Punctuation, raw: [41] }, Token { kind: Punctuation, raw: [38] }, Token { kind: Literal, raw: [55] }], Alt)))
Failed test for Int_12, expected Int(12), got Err(Error(Code([Token { kind: Punctuation, raw: [40] }, Token { kind: Punctuation, raw: [40] }, Token { kind: Punctuation, raw: [40] }, Token { kind: Literal, raw: [51, 50] }, Token { kind: Punctuation, raw: [124] }, Token { kind: Literal, raw: [49, 53] }, Token { kind: Punctuation, raw: [41] }, Token { kind: Punctuation, raw: [94] }, Token { kind: Literal, raw: [57] }, Token { kind: Punctuation, raw: [41] }, Token { kind: Punctuation, raw: [38] }, Token { kind: Literal, raw: [55] }, Token { kind: Punctuation, raw: [41] }, Token { kind: Punctuation, raw: [60, 60] }, Token { kind: Literal, raw: [49] }], Alt)))
Failed test for Int_17, expected Int(17), got Err(Error(Code([Token { kind: Punctuation, raw: [40] }, Token { kind: Punctuation, raw: [40] }, Token { kind: Punctuation, raw: [40] }, Token { kind: Punctuation, raw: [40] }, Token { kind: Literal, raw: [51, 50] }, Token { kind: Punctuation, raw: [124] }, Token { kind: Literal, raw: [49, 53] }, Token { kind: Punctuation, raw: [41] }, Token { kind: Punctuation, raw: [94] }, Token { kind: Literal, raw: [57] }, Token { kind: Punctuation, raw: [41] }, Token { kind: Punctuation, raw: [38] }, Token { kind: Literal, raw: [55] }, Token { kind: Punctuation, raw: [41] }, Token { kind: Punctuation, raw: [60, 60] }, Token { kind: Literal, raw: [49] }, Token { kind: Punctuation, raw: [41] }, Token { kind: Punctuation, raw: [43] }, Token { kind: Literal, raw: [53] }], Alt)))
Failed test for Int_15, expected Int(15), got Err(Error(Code([Token { kind: Punctuation, raw: [40] }, Token { kind: Punctuation, raw: [40] }, Token { kind: Punctuation, raw: [40] }, Token { kind: Punctuation, raw: [40] }, Token { kind: Punctuation, raw: [40] }, Token { kind: Literal, raw: [51, 50] }, Token { kind: Punctuation, raw: [124] }, Token { kind: Literal, raw: [49, 53] }, Token { kind: Punctuation, raw: [41] }, Token { kind: Punctuation, raw: [94] }, Token { kind: Literal, raw: [57] }, Token { kind: Punctuation, raw: [41] }, Token { kind: Punctuation, raw: [38] }, Token { kind: Literal, raw: [55] }, Token { kind: Punctuation, raw: [41] }, Token { kind: Punctuation, raw: [60, 60] }, Token { kind: Literal, raw: [49] }, Token { kind: Punctuation, raw: [41] }, Token { kind: Punctuation, raw: [43] }, Token { kind: Literal, raw: [53] }, Token { kind: Punctuation, raw: [41] }, Token { kind: Punctuation, raw: [45] }, Token { kind: Literal, raw: [50] }], Alt)))
Failed test for Int_60, expected Int(60), got Err(Error(Code([Token { kind: Punctuation, raw: [40] }, Token { kind: Punctuation, raw: [40] }, Token { kind: Punctuation, raw: [40] }, Token { kind: Punctuation, raw: [40] }, Token { kind: Punctuation, raw: [40] }, Token { kind: Punctuation, raw: [40] }, Token { kind: Literal, raw: [51, 50] }, Token { kind: Punctuation, raw: [124] }, Token { kind: Literal, raw: [49, 53] }, Token { kind: Punctuation, raw: [41] }, Token { kind: Punctuation, raw: [94] }, Token { kind: Literal, raw: [57] }, Token { kind: Punctuation, raw: [41] }, Token { kind: Punctuation, raw: [38] }, Token { kind: Literal, raw: [55] }, Token { kind: Punctuation, raw: [41] }, Token { kind: Punctuation, raw: [60, 60] }, Token { kind: Literal, raw: [49] }, Token { kind: Punctuation, raw: [41] }, Token { kind: Punctuation, raw: [43] }, Token { kind: Literal, raw: [53] }, Token { kind: Punctuation, raw: [41] }, Token { kind: Punctuation, raw: [45] }, Token { kind: Literal, raw: [50] }, Token { kind: Punctuation, raw: [41] }, Token { kind: Punctuation, raw: [42] }, Token { kind: Literal, raw: [52] }], Alt)))
Failed test for Int_30, expected Int(30), got Err(Error(Code([Token { kind: Punctuation, raw: [40] }, Token { kind: Punctuation, raw: [40] }, Token { kind: Punctuation, raw: [40] }, Token { kind: Punctuation, raw: [40] }, Token { kind: Punctuation, raw: [40] }, Token { kind: Punctuation, raw: [40] }, Token { kind: Punctuation, raw: [40] }, Token { kind: Literal, raw: [51, 50] }, Token { kind: Punctuation, raw: [124] }, Token { kind: Literal, raw: [49, 53] }, Token { kind: Punctuation, raw: [41] }, Token { kind: Punctuation, raw: [94] }, Token { kind: Literal, raw: [57] }, Token { kind: Punctuation, raw: [41] }, Token { kind: Punctuation, raw: [38] }, Token { kind: Literal, raw: [55] }, Token { kind: Punctuation, raw: [41] }, Token { kind: Punctuation, raw: [60, 60] }, Token { kind: Literal, raw: [49] }, Token { kind: Punctuation, raw: [41] }, Token { kind: Punctuation, raw: [43] }, Token { kind: Literal, raw: [53] }, Token { kind: Punctuation, raw: [41] }, Token { kind: Punctuation, raw: [45] }, Token { kind: Literal, raw: [50] }, Token { kind: Punctuation, raw: [41] }, Token { kind: Punctuation, raw: [42] }, Token { kind: Literal, raw: [52] }, Token { kind: Punctuation, raw: [41] }, Token { kind: Punctuation, raw: [47] }, Token { kind: Literal, raw: [50] }], Alt)))
Failed test for Int_39, expected Int(39), got Err(Error(Code([Token { kind: Literal, raw: [51, 50] }, Token { kind: Punctuation, raw: [124] }, Token { kind: Literal, raw: [49, 53] }, Token { kind: Punctuation, raw: [94] }, Token { kind: Literal, raw: [57] }, Token { kind: Punctuation, raw: [38] }, Token { kind: Literal, raw: [55] }, Token { kind: Punctuation, raw: [60, 60] }, Token { kind: Literal, raw: [49] }, Token { kind: Punctuation, raw: [43] }, Token { kind: Literal, raw: [53] }, Token { kind: Punctuation, raw: [45] }, Token { kind: Literal, raw: [50] }, Token { kind: Punctuation, raw: [42] }, Token { kind: Literal, raw: [52] }, Token { kind: Punctuation, raw: [47] }, Token { kind: Literal, raw: [50] }], Alt)))
Failed test for Int_n1, expected Int(-1), got Err(Error(Code([Token { kind: Literal, raw: [49, 56, 52, 52, 54, 55, 52, 52, 48, 55, 51, 55, 48, 57, 53, 53, 49, 54, 49, 53] }], Alt)))
Failed test for Int_n9223372036854775808, expected Int(-9223372036854775808), got Err(Error(Code([Token { kind: Literal, raw: [57, 50, 50, 51, 51, 55, 50, 48, 51, 54, 56, 53, 52, 55, 55, 53, 56, 48, 56] }], Alt)))
Failed test for Fn_Int_9, expected Int(9), got Err(Error(Code([Token { kind: Punctuation, raw: [42] }, Token { kind: Literal, raw: [51] }], Custom(Partial))))
thread 'int_unsigned' panicked at 'test_file', tests/clang.rs:245:0

[Geal]

alright, so those are mostly about the Incomplete behaviour. nom is much stricter on this now, but on the other hand, I introduced a way to consider the input as complete, and it's already usable with the following types: nom::types::CompleteStr and nom::types::CompleteByteSlice (it's possible to alias those type names, they're a bit long). Is your code assuming it will have the whole input to parse at once?

[jethrogb]

Is your code assuming it will have the whole input to parse at once?

Yes, but most of the code uses &[Token] as input.

[ignatenkobrain]

any news here?

[jethrogb]

The PR doesn't build and pass tests as-is.

[infinity0]

Any details on the difficulty, @Geal ? We're trying to avoid having duplicate versions of the same dependency in Debian, and have already packaged nom 4.

[Geal]

Hello, now it compiles and the tests are passing. I did not do the conversion to the Complete* types, but it could simplify the code significantly

[Geal]

friendly ping on this :)

[jethrogb]

Apologies, missed the update. I'll take a look in a few hours.

[jethrogb]

Again, sorry for the delay. Just published cexpr 0.3.0 with this change. Thanks!