The pathological test case was taking ~10 seconds (!!). Now it's just 200ms.
The issue was that it was relying on ast.FileInfo to provide things like the line numbers for tokens. But those are all computed on-demand. And this operation was being done enough that it induced pathological bad performance.
So now we have just a little bit more book-keeping in the lexer so that we can decide when to donate a comment to prior token as a trailing comment without having to ask the file info to compute things.
The pathological test case was taking ~10 seconds (!!). Now it's just 200ms.
The issue was that it was relying on
ast.FileInfoto provide things like the line numbers for tokens. But those are all computed on-demand. And this operation was being done enough that it induced pathological bad performance.So now we have just a little bit more book-keeping in the lexer so that we can decide when to donate a comment to prior token as a trailing comment without having to ask the file info to compute things.