search

kaspanet / rusty-kaspa

Kaspa full-node and related libraries in the Rust programming language. This is a stable version at the initial rollout phases.
ISC License
359 stars 109 forks source link

Move to mimalloc #361

Closed someone235 closed 5 months ago

someone235 commented 5 months ago

Since rusty-kaspa uses a lot of allocations, the RSS remains high even after the memory is free. This should be mitigated by using mimalloc which utilizes the use of MADV_FREE and MADV_DONTNEED (and equivalent techniques on non unix-like systems) which tend to release memory earlier and help in keeping the RSS small.

elichai commented 5 months ago

Why only unix?

michaelsutton commented 5 months ago
  1. we should also move the C API call to a more accessible module and should probably set it (the allocator and the purge_decommits setting) for tests as well
  2. should we leave the version free version = "*" @elichai ?
elichai commented 5 months ago
  1. should we leave the version free version = "*" @elichai ?

Hmm no, in case some of the API(the C API or the allocator name) changes we don't want our code to just break

aspect commented 5 months ago

We don't want any dependencies to be "*" as that exposes the project to supply-chain attacks whenever Cargo.lock is reset.

aspect commented 5 months ago

If mimalloc is to be kept, can its initialization please be moved to daemon.rs or better, a dedicated kaspad/src/memory.rs module (which is then invoked from main.rs via a public fn). If this becomes a requirement for rusty kaspa, it should be loadable externally alongside the daemon itself (with pre-configured parameters within this module/loader fn, not in main).

elichai commented 5 months ago

If mimalloc is to be kept, can its initialization please be moved to daemon.rs or better, a dedicated kaspad/src/memory.rs module (which is then invoked from main.rs via a public fn). If this becomes a requirement for rusty kaspa, it should be loadable externally alongside the daemon itself (with pre-configured parameters within this module/loader fn, not in main).

that might not work well if you have C dependencies, if you want the override feature to work properly then mimalloc has to be passed before any C dependency, meaning it should be the first dependency of the top-most crate

Note that this is only for the C deps, for the Rust deps it shouldn't matter where in the dep graph are you setting the global allocator