quixdb / squash

Compression abstraction library and utilities
https://quixdb.github.io/squash/
MIT License
407 stars 53 forks source link

Custom allocators #170

Closed nemequ closed 8 years ago

nemequ commented 8 years ago

We should probably allow for custom memory allocators. It would allow better integration with some higher-level languages (such as PHP, IIRC, Python). Also, we could have a realloc-like function which doesn't necessarily move existing data (for example, for jemalloc we could call rallocx and, if that fails, free and malloc a new buffer).

Lots of plugins allow them, so in those cases we should pass them along, but I don't think it will ever be possible to completely eliminate malloc/free in all plugins, especially those written in C++. If, however, we did manage to completely eliminate malloc, it would be trivial to add memory consumption information to the benchmark…

One concern is that I'm not sure all plugins would support a user data field where we could put a SquashContext*, meaning that either the information would have to be truly global (which doesn't mesh with Squash's design) or any time we invoke something in the library we would have to store the context in a thread local variable (there is a wrapper in tinycthread for _Thread_local) then extract it in a wrapper.