rster2002
commented
4 months ago So, it seems the issue lies in these lines of code
It's very likely because of the mem take and it needing to loop over 100's of items items every for every single other item (O(n^2) baby!). One fix could be to make it so you need to manually activate the cleanup of the left-overs.
The current implementation is just too slow. This is probably because it creates a lot of independent allocations which just cause a lot of overhead.
Currently it mutates a lot of state which allows us to create a single state, but means that a lot of data is cloned and lots of allocations need to happen.
One idea that was initially considered is just adding every entry that was read to a single vec and perform operations on that, but that would mean that you would need to iterate over most if not all items if you want to perform some lookup or action.
Maybe a hybrid method would work where everything is added to a single vec, but state is kept through references to the vec? This would require a lot of references and probably some unsafe code.