This uses an inner Arc to decrease the size of GreenToken. This uses the same caching logic as GreenNode through GreenNodeBuilder. Do note that this means that to get a decently memory-thoughtful tree, either the use of GreenNodeBuilder or a manual cache in the same style is required. We should probably consider eventually exposing some version of the Cache implemented here, such that bottom-up builders can use it and get the deduplication for free.
The main reason for shrinking GreenToken is the target of shrinking GreenElement, as each node carries a [GreenElement], and by reducing overhead per-element, we can considerably decrease overall memory usage. By boxing GreenToken we make room to shrink GreenNode and benefit from said shrinkage.
This uses an inner
Arc
to decrease the size ofGreenToken
. This uses the same caching logic asGreenNode
throughGreenNodeBuilder
. Do note that this means that to get a decently memory-thoughtful tree, either the use ofGreenNodeBuilder
or a manual cache in the same style is required. We should probably consider eventually exposing some version of theCache
implemented here, such that bottom-up builders can use it and get the deduplication for free.The main reason for shrinking
GreenToken
is the target of shrinkingGreenElement
, as each node carries a[GreenElement]
, and by reducing overhead per-element, we can considerably decrease overall memory usage. By boxingGreenToken
we make room to shrinkGreenNode
and benefit from said shrinkage.r? @matklad
Size comparison
Before: ``` GreenNode 24 GreenToken 32 GreenElement 40 SyntaxNode 8 SyntaxToken 16 SyntaxElement 24 ``` After: ``` GreenNode 24 GreenToken 8 GreenElement 32 SyntaxNode 8 SyntaxToken 16 SyntaxElement 24 ```