search

danielealbano / cachegrand

cachegrand - a modern data ingestion, processing and serving platform built for today's hardware
BSD 3-Clause "New" or "Revised" License
975 stars 34 forks source link

Add support for FFMA without hugepages and using only mmap #296

Closed danielealbano closed 1 year ago

danielealbano commented 1 year ago

This branch adds support for Fast Fixed Memory Allocator (FFMA) to function without hugepages and using only mmap.

Overall, it makes FFMA more versatile by allowing it to work without hugepages and with only mmap, which can improve its performance and efficiency in certain scenarios, but most importantly as FFMA doesn't suffer from memory fragmentation makes it a perfect choice for a long-running software like cachegrand as reduce the scope of any garbage collector that will be implemented down the line.

The only downside, currently, is that the storage_db has to operate using blocks of 64kb and this limits the maximum size of the chunks to about ~170MB of data.

This limitation will be removed with a future PR.

There are also some additional PRs planned to make FFMA less dependant on malloc and rely instead on mmap for its own internal allocations.

codecov[bot] commented 1 year ago

Codecov Report

Patch coverage: 87.86% and project coverage change: -0.02 :warning:

Comparison is base (d0202e2) 81.71% compared to head (1584935) 81.70%.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #296 +/- ## ========================================== - Coverage 81.71% 81.70% -0.02% ========================================== Files 161 163 +2 Lines 11009 11031 +22 ========================================== + Hits 8996 9012 +16 - Misses 2013 2019 +6 ``` | Flag | Coverage Δ | | |---|---|---| | unittests | `81.70% <87.86%> (-0.02%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Daniele+Salvatore+Albano#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/danielealbano/cachegrand/pull/296?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Daniele+Salvatore+Albano) | Coverage Δ | | |---|---|---| | [...rc/module/redis/command/module\_redis\_command\_get.c](https://codecov.io/gh/danielealbano/cachegrand/pull/296?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Daniele+Salvatore+Albano#diff-c3JjL21vZHVsZS9yZWRpcy9jb21tYW5kL21vZHVsZV9yZWRpc19jb21tYW5kX2dldC5j) | `100.00% <ø> (ø)` | | | [...module/redis/command/module\_redis\_command\_getdel.c](https://codecov.io/gh/danielealbano/cachegrand/pull/296?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Daniele+Salvatore+Albano#diff-c3JjL21vZHVsZS9yZWRpcy9jb21tYW5kL21vZHVsZV9yZWRpc19jb21tYW5kX2dldGRlbC5j) | `88.00% <ø> (ø)` | | | [.../module/redis/command/module\_redis\_command\_getex.c](https://codecov.io/gh/danielealbano/cachegrand/pull/296?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Daniele+Salvatore+Albano#diff-c3JjL21vZHVsZS9yZWRpcy9jb21tYW5kL21vZHVsZV9yZWRpc19jb21tYW5kX2dldGV4LmM=) | `91.23% <ø> (ø)` | | | [...dule/redis/command/module\_redis\_command\_getrange.c](https://codecov.io/gh/danielealbano/cachegrand/pull/296?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Daniele+Salvatore+Albano#diff-c3JjL21vZHVsZS9yZWRpcy9jb21tYW5kL21vZHVsZV9yZWRpc19jb21tYW5kX2dldHJhbmdlLmM=) | `93.10% <ø> (ø)` | | | [...module/redis/command/module\_redis\_command\_getset.c](https://codecov.io/gh/danielealbano/cachegrand/pull/296?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Daniele+Salvatore+Albano#diff-c3JjL21vZHVsZS9yZWRpcy9jb21tYW5kL21vZHVsZV9yZWRpc19jb21tYW5kX2dldHNldC5j) | `78.57% <ø> (ø)` | | | [...c/module/redis/command/module\_redis\_command\_mget.c](https://codecov.io/gh/danielealbano/cachegrand/pull/296?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Daniele+Salvatore+Albano#diff-c3JjL21vZHVsZS9yZWRpcy9jb21tYW5kL21vZHVsZV9yZWRpc19jb21tYW5kX21nZXQuYw==) | `81.25% <ø> (ø)` | | | [...module/redis/command/module\_redis\_command\_rename.c](https://codecov.io/gh/danielealbano/cachegrand/pull/296?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Daniele+Salvatore+Albano#diff-c3JjL21vZHVsZS9yZWRpcy9jb21tYW5kL21vZHVsZV9yZWRpc19jb21tYW5kX3JlbmFtZS5j) | `88.37% <ø> (ø)` | | | [...dule/redis/command/module\_redis\_command\_renamenx.c](https://codecov.io/gh/danielealbano/cachegrand/pull/296?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Daniele+Salvatore+Albano#diff-c3JjL21vZHVsZS9yZWRpcy9jb21tYW5kL21vZHVsZV9yZWRpc19jb21tYW5kX3JlbmFtZW54LmM=) | `89.13% <ø> (ø)` | | | [...rc/module/redis/command/module\_redis\_command\_set.c](https://codecov.io/gh/danielealbano/cachegrand/pull/296?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Daniele+Salvatore+Albano#diff-c3JjL21vZHVsZS9yZWRpcy9jb21tYW5kL21vZHVsZV9yZWRpc19jb21tYW5kX3NldC5j) | `88.61% <ø> (ø)` | | | [.../module/redis/command/module\_redis\_command\_setex.c](https://codecov.io/gh/danielealbano/cachegrand/pull/296?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Daniele+Salvatore+Albano#diff-c3JjL21vZHVsZS9yZWRpcy9jb21tYW5kL21vZHVsZV9yZWRpc19jb21tYW5kX3NldGV4LmM=) | `89.47% <ø> (ø)` | | | ... and [9 more](https://codecov.io/gh/danielealbano/cachegrand/pull/296?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Daniele+Salvatore+Albano) | | ... and [3 files with indirect coverage changes](https://codecov.io/gh/danielealbano/cachegrand/pull/296/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Daniele+Salvatore+Albano) Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Daniele+Salvatore+Albano). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Daniele+Salvatore+Albano)

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.