The dumper is sightly too memory hungry. As I first looked at dumper code, the obvious reason was because the dumper keeps all 150 live threads, with all of its posts in memory, but a first attempt at writing code to purge those seems to shave no more than 10% of memory (I was able to drop mem usage for a dumper that wasn't archiving media or previews from 161 to 147 MiB, which checks out, at ~100 KiB per thread.) This suggests shared data structures aren't the issue here.
Since Perl threads aren't lightweight by design, the other possible reason might be that there's just too many modules and other constant data structures that are loaded with each thread, and this issue might be unfixable without rewriting the dumper in another language.
The dumper is sightly too memory hungry. As I first looked at dumper code, the obvious reason was because the dumper keeps all 150 live threads, with all of its posts in memory, but a first attempt at writing code to purge those seems to shave no more than 10% of memory (I was able to drop mem usage for a dumper that wasn't archiving media or previews from 161 to 147 MiB, which checks out, at ~100 KiB per thread.) This suggests shared data structures aren't the issue here.
Since Perl threads aren't lightweight by design, the other possible reason might be that there's just too many modules and other constant data structures that are loaded with each thread, and this issue might be unfixable without rewriting the dumper in another language.