Added flags to LexerGenerator

[MarkusH]

This commit allows specifying regex flags when adding new rules to a lexer generator, for both, token and ignore patterns

[alex]

Are all flags processed at regex compile time, or are they needed at evaluation time as well? (I'm thinking about the RPython specific regex paths)

[alex]

This probably needs a note somewhere in the docs, but assuming the flags are all handled at re.compile time, this patch looks great to me. Thanks!

[alex]

It looks like you do need to pass flags down through the RPython layer, let me know if there are any questions about that.

I believe the places you need to update are:

• RuleRepr.__init__: add a flags field to the struct
• RuleRepr.convert_const: pass the flags through to the instance
• LLRule.ll_matches: pass the flags along where 0 is passed now.

[MarkusH]

Hey @alex . I have absolutely no clue about RPython, but I'll try my best :)

[alex]

Can you add a note to the docs somewhere? After that, I can merge it, thanks!

[MarkusH]

Done. Thank you for your support.

[alex]

Thanks!