search

danielealbano / cachegrand

cachegrand - a modern data ingestion, processing and serving platform built for today's hardware
BSD 3-Clause "New" or "Revised" License
975 stars 34 forks source link

Multi database support #371

Closed danielealbano closed 1 year ago

danielealbano commented 1 year ago

This PR implements the multi database support in cachegrand, allowing cachegrand to have separated namespaces where to store data meanwhile sharing the database index.

Although this approach is different from what Redis does, it opens the doors to different scenarios, e.g. transactional operations across different namespaces will be possible because the indexes are shared.

This approach also will open the door to mapping specific database numbers to specific module configurations in the config file (currently not implemented), allowing to have more endpoint exposed with access to different databases (e.g. an endpoint with access to the db numbers from 0 to 15, and another, maybe in the future shared with memcache, from 16 to 64). Although internally the database number is an uint32, the configuration allows only up to UINT8_MAX (255) namespaces.

One of the biggest downsides of having to support multiple databases is having separated counters, one global and one per database. To avoid a performance loss, a number of optimizations have been implemented in the hashtable_spsc and it's now capable of getting the value associated with a key in ~1.09ns in average with the worst case being around ~1.30ns.

Here a bench run of hashtable-spsc-op-get

2023-04-28T23:00:29+02:00
Running /home/daalbano/dev/cachegrand/cmake-build-release/benches/bench-hashtable-spsc-op-get
Run on (32 X 4200 MHz CPU s)
CPU Caches:
  L1 Data 32 KiB (x16)
  L1 Instruction 32 KiB (x16)
  L2 Unified 512 KiB (x16)
  L3 Unified 16384 KiB (x4)
Load Average: 3.27, 2.28, 2.98
------------------------------------------------------------------------------------------------------------------------------------------------
Benchmark                                                                                                      Time             CPU   Iterations
------------------------------------------------------------------------------------------------------------------------------------------------
HashtableSpscOpGetFixture/FindTokenInHashtableWorstCase1Benchmark/iterations:100000000                      14.3 ns         14.3 ns    100000000
HashtableSpscOpGetFixture/FindTokenInHashtableAvgCaseCIBenchmark/iterations:100000000                       8.78 ns         8.78 ns    100000000
HashtableSpscOpGetFixture/FindTokenInHashtableAvgCaseCSBenchmark/iterations:100000000                       6.98 ns         6.98 ns    100000000
HashtableSpscOpGetKeyAsIntFixture/FindTokenInHashtableBypassHashAndKey1Benchmark/iterations:100000000       1.09 ns         1.09 ns    100000000
HashtableSpscOpGetKeyAsIntFixture/FindTokenInHashtableBypassHashAndKey2Benchmark/iterations:100000000       1.36 ns         1.36 ns    100000000
HashtableSpscOpGetKeyAsIntFixture/FindTokenInHashtableBypassHashAndKey3Benchmark/iterations:100000000       1.09 ns         1.09 ns    100000000
HashtableSpscOpGetKeyAsIntFixture/FindTokenInHashtableBypassHashAndKey4Benchmark/iterations:100000000       1.20 ns         1.20 ns    100000000
HashtableSpscOpGetKeyAsIntFixture/FindTokenInHashtableBypassHashAndKey5Benchmark/iterations:100000000       1.60 ns         1.60 ns    100000000

These amazing performances render almost null the impact of the extra counters.

The PR also introduces the SELECT command.

Although some tests have been implemented for the SELECT command to test, end-to-end, the process, it's necessary to implement further tests.

Closes #368 Closes #369

codecov[bot] commented 1 year ago

Codecov Report

Patch coverage: 53.77% and project coverage change: -0.50 :warning:

Comparison is base (b151815) 79.03% compared to head (9f1f01f) 78.53%.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #371 +/- ## ========================================== - Coverage 79.03% 78.53% -0.50% ========================================== Files 183 184 +1 Lines 12614 12871 +257 ========================================== + Hits 9969 10108 +139 - Misses 2645 2763 +118 ``` | Flag | Coverage Δ | | |---|---|---| | unittests | `78.53% <53.77%> (-0.50%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Daniele+Salvatore+Albano#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/danielealbano/cachegrand/pull/371?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Daniele+Salvatore+Albano) | Coverage Δ | | |---|---|---| | [src/config\_cyaml\_schema.c](https://codecov.io/gh/danielealbano/cachegrand/pull/371?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Daniele+Salvatore+Albano#diff-c3JjL2NvbmZpZ19jeWFtbF9zY2hlbWEuYw==) | `100.00% <ø> (ø)` | | | [...tures/hashtable/mcmp/hashtable\_op\_get\_random\_key.c](https://codecov.io/gh/danielealbano/cachegrand/pull/371?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Daniele+Salvatore+Albano#diff-c3JjL2RhdGFfc3RydWN0dXJlcy9oYXNodGFibGUvbWNtcC9oYXNodGFibGVfb3BfZ2V0X3JhbmRvbV9rZXkuYw==) | `100.00% <ø> (ø)` | | | [...a\_structures/hashtable/mcmp/hashtable\_support\_op.c](https://codecov.io/gh/danielealbano/cachegrand/pull/371?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Daniele+Salvatore+Albano#diff-c3JjL2RhdGFfc3RydWN0dXJlcy9oYXNodGFibGUvbWNtcC9oYXNodGFibGVfc3VwcG9ydF9vcC5j) | `100.00% <ø> (ø)` | | | [...nd/helpers/module\_redis\_command\_helper\_incr\_decr.c](https://codecov.io/gh/danielealbano/cachegrand/pull/371?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Daniele+Salvatore+Albano#diff-c3JjL21vZHVsZS9yZWRpcy9jb21tYW5kL2hlbHBlcnMvbW9kdWxlX3JlZGlzX2NvbW1hbmRfaGVscGVyX2luY3JfZGVjci5j) | `83.75% <ø> (ø)` | | | [...module/redis/command/module\_redis\_command\_append.c](https://codecov.io/gh/danielealbano/cachegrand/pull/371?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Daniele+Salvatore+Albano#diff-c3JjL21vZHVsZS9yZWRpcy9jb21tYW5kL21vZHVsZV9yZWRpc19jb21tYW5kX2FwcGVuZC5j) | `77.01% <ø> (ø)` | | | [...c/module/redis/command/module\_redis\_command\_copy.c](https://codecov.io/gh/danielealbano/cachegrand/pull/371?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Daniele+Salvatore+Albano#diff-c3JjL21vZHVsZS9yZWRpcy9jb21tYW5kL21vZHVsZV9yZWRpc19jb21tYW5kX2NvcHkuYw==) | `75.26% <ø> (ø)` | | | [...module/redis/command/module\_redis\_command\_dbsize.c](https://codecov.io/gh/danielealbano/cachegrand/pull/371?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Daniele+Salvatore+Albano#diff-c3JjL21vZHVsZS9yZWRpcy9jb21tYW5kL21vZHVsZV9yZWRpc19jb21tYW5kX2Ric2l6ZS5j) | `100.00% <ø> (ø)` | | | [...rc/module/redis/command/module\_redis\_command\_del.c](https://codecov.io/gh/danielealbano/cachegrand/pull/371?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Daniele+Salvatore+Albano#diff-c3JjL21vZHVsZS9yZWRpcy9jb21tYW5kL21vZHVsZV9yZWRpc19jb21tYW5kX2RlbC5j) | `100.00% <ø> (ø)` | | | [...module/redis/command/module\_redis\_command\_exists.c](https://codecov.io/gh/danielealbano/cachegrand/pull/371?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Daniele+Salvatore+Albano#diff-c3JjL21vZHVsZS9yZWRpcy9jb21tYW5kL21vZHVsZV9yZWRpc19jb21tYW5kX2V4aXN0cy5j) | `100.00% <ø> (ø)` | | | [...module/redis/command/module\_redis\_command\_expire.c](https://codecov.io/gh/danielealbano/cachegrand/pull/371?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Daniele+Salvatore+Albano#diff-c3JjL21vZHVsZS9yZWRpcy9jb21tYW5kL21vZHVsZV9yZWRpc19jb21tYW5kX2V4cGlyZS5j) | `94.87% <ø> (ø)` | | | ... and [66 more](https://codecov.io/gh/danielealbano/cachegrand/pull/371?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Daniele+Salvatore+Albano) | | ... and [4 files with indirect coverage changes](https://codecov.io/gh/danielealbano/cachegrand/pull/371/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Daniele+Salvatore+Albano)

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.