It turned out using the debug_tt consumed a lot of memory. So in the next rewrite the plugin directly gets debug info from the low level VM in hardhat. This makes it comparatively very memory efficient. Please checkout the v2-beta branch for more info.
(Documentation)
Interesting things that could be traced: