search

jflex-de / jflex

The fast scanner generator for Java™ with full Unicode support
http://jflex.de
Other
581 stars 114 forks source link

Optimize memory usage #1106

Open hurricup opened 1 year ago

hurricup commented 1 year ago
ice1000 commented 4 months ago

Unfortunately the project is dead :( can we have JetBrains jflex published to maven central?

lsf37 commented 4 months ago

Not sure which project you mean is dead, but I'm not going to add an Emitter change with just the explanation "optimisation" without any benchmarks, use case or anything else provided.

For small scanners this is going to be slower, so from which size on is it better? Is it actually ever better? How much?

ice1000 commented 4 months ago

Not sure which project you mean is dead

Ohh!! I'm happy to see you're still active. I assumed this project is no longer receiving updates due to the amount of time it is being inactive.

lsf37 commented 4 months ago

Still around, just not always with a lot of time. Sorry I was sounding a bit annoyed, PRs with subtle changes to the runtime engine like this one take an inordinate amount of time to review properly compared to the benefit they might (or might not) bring, so I've put it off for quite a while now.

hurricup commented 4 months ago

@lsf37 why is it going to be slower? This was inspired by memory snapshots analysis from intellij idea, which uses slightly modified version of jflex for its lexers.

  1. No need to allocate buffers if they are never used
  2. Bitset is more memory effective than boolean array

Not sure what benchmarks would you like to see. I did measured performance as well on some huge samples with warmup and stuff, and difference was neglectable, not remember in which direction. Like a percent or so.

This change is in the intellij fork already, so just wanted to contribute for the common good. If you don't think it is worth it, feel free to drop this PR, i'm totally ok with that.