microsoft / llguidance

Low-level Guidance Parser
MIT License
30 stars 7 forks source link

[perf] large `allowed_lexemes` sets #64

Open mmoskal opened 4 days ago

mmoskal commented 4 days ago

If the lexer is large, the bitsets will be large. However, the Earley rows are limited (currently to 200 items), so it may make more sense to use a HashSet of lexemes not bitset if the lexer is large.

Second option: put limit on size of lexer.