Memory leak found in src/scan/tokens.js

[Magic-Bytes]

Global variable tokenIdxToClass stores all TokenType instances when I create Lexer instance every time. Can I clear the related TokenType instances when I destroy the Lexer Instance?

[bd82]

This is a known issue, see: #1056

However, I am not sure why it should cause a memory leak. The lexer is stateless, so you could just re-use the same Lexer instance and and store these TokenTypes once. This could also improve performance due to various optimizations in JavaScript engines.

• https://chevrotain.io/docs/guide/performance.html#using-a-singleton-parser

[bd82]

ping and I will re-open if using a single instance does not solve the leak issue.

[Magic-Bytes]

This is a known issue, see: #1056

However, I am not sure why it should cause a memory leak. The lexer is stateless, so you could just re-use the same Lexer instance and and store these TokenTypes once. This could also improve performance due to various optimizations in JavaScript engines.

• https://chevrotain.io/docs/guide/performance.html#using-a-singleton-parser

@bd82 Yes, we have considered reusing the same Lexing instance. However, we use the lexer in NodeJS server. We create different tokens and lexers related to request contexts, so they cannot be reused. According to the usage of global variable tokenIdxToClass, they cannot be released, causing memory leak. In fact, we want an API to dispose lexer and token instances. Thanks.

[bd82]

@Magic-Bytes

I see, well until the global state is fixed you will need to find a workaround. Either use something like patch-package to clean up the global state or if you can limit the number of permutations of your lexer instances cache and re-use them.