winnow-rs / winnow

Making parsing a breeze
https://docs.rs/winnow
Other
525 stars 40 forks source link

`tag(Caseless(...))` panicks while matching certain unicode capitalized characters. #414

Closed epage closed 7 months ago

epage commented 9 months ago

Adapted from rust-bakery/nom#1719

fn main() {
    let _ = nom::bytes::complete::tag_no_case::<_,_,nom::error::Error<&str>>("k")("K");
}

The tag_no_case function can panic whenever a character in a to be matched string lowercases to a character with a shorter byte pattern.

In the above example the K to be matched is actually the kelvin sign with the UTF-8 byte pattern of 0xE2 0x84 0xAA. This character has a the normal single byte k as its lowercase character and therefore matches the tag.

The tag_no_case implementation, after matching a string to the tag, splits the given string with the length of the tag.

This assumes that the length in bytes of the matched characters is the same as the tag it was matched to. But, as in the above example, this is not always the case, resulting in the function sometimes trying to split a string outside of a character boundary and panicking

epage commented 9 months ago

We need CompareResult::Ok to include the byte offset for where the difference occurs and to split on that.