In the current implementation of both mutable and immutable Document types, each token string is assigned a unique instance. This approach is inefficient, as many source code tokens typically share the same content.
To optimize memory usage, the underlying text storage for source code should be replaced with string interning, which can potentially improve computational performance as well.
Ideally, the implementation should support both options — interning and non-interning — while enabling string interning by default.
In the current implementation of both mutable and immutable Document types, each token string is assigned a unique instance. This approach is inefficient, as many source code tokens typically share the same content.
To optimize memory usage, the underlying text storage for source code should be replaced with string interning, which can potentially improve computational performance as well.
Ideally, the implementation should support both options — interning and non-interning — while enabling string interning by default.