search

rrevenantt / antlr4rust

ANTLR4 parser generator runtime for Rust programming laguage
Other
398 stars 70 forks source link

Integer overflow w/ generated lexer #47

Closed netshade closed 2 years ago

netshade commented 2 years ago

I was attempting to compile a lexer generated w/ the Rust backend, and I got the following errors on my tokens:

error: this arithmetic operation will overflow
     |
8379 | ...ze << (FLOAT - 1)) | (1usize << (VALID_IDENTIFIER - 1)) | (1usize << (FIELD - 1)) | (1usize << (INDEXED_FIELD - 1)))) != 0) {
     |                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ attempt to shift left by `61_isize`, which would overflow

I think this is because the template portion that is doing a bitset emulation of the Antlr code is presuming that 1usize is 64-bit width trying to match the original implementation?.

If my understanding here is correct, should it be that the template is instead masking with 1u64 instead ? Happy to provide a PR if this is right.

netshade commented 2 years ago

( This came about because I was compiling for wasm32-unknown-unknown - on other architectures it worked as expected )

netshade commented 2 years ago

( After looking at the template some more I see that this definitely is not quite so simple as that. Are the semantics of the parser more or less locked in at "everything must be word sized which is also 64 bits", or is it possible to be explicit that everything is 64 bits here? )