search

danielealbano / cachegrand

cachegrand - a modern data ingestion, processing and serving platform built for today's hardware
BSD 3-Clause "New" or "Revised" License
975 stars 34 forks source link

Implement snapshotting at shutdown #363

Closed danielealbano closed 1 year ago

danielealbano commented 1 year ago

This PR introduces the ability to trigger a snapshot when the workers start to shutdown.

To achieve this quite some changes have been done to the workers events loops: meanwhile before the initialization and shutdown of the internal database and the networking could have performed before entering the loop and at its end, with the need to be able to perform I/O the loop has still to run at the shutdown but it's necessary to terminate all the connections and pending operations. The PR introduces the required changes and therefore it's now possible perform operations both at the startup and at the shutdown, within the loop!

The PR also introduces a new field in the snapshot structure to keep track of the number of iteration performed, very handy to check if a snapshot has been taken, used now internally.

A number of additional fixes have been introduced, e.g. to better sync the shutdown of the signal handler thread or of the epoch gc workers., better tests for the SAVE and BGSAVE commands, improve tests for SHUTDOWN, etc.

Closes #316

codecov[bot] commented 1 year ago

Codecov Report

Patch coverage: 82.83% and project coverage change: +0.13 :tada:

Comparison is base (8475737) 78.97% compared to head (0fdc880) 79.09%.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #363 +/- ## ========================================== + Coverage 78.97% 79.09% +0.13% ========================================== Files 182 182 Lines 12546 12609 +63 ========================================== + Hits 9907 9973 +66 + Misses 2639 2636 -3 ``` | Flag | Coverage Δ | | |---|---|---| | unittests | `79.09% <82.83%> (+0.13%)` | :arrow_up: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Daniele+Salvatore+Albano#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/danielealbano/cachegrand/pull/363?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Daniele+Salvatore+Albano) | Coverage Δ | | |---|---|---| | [src/config\_cyaml\_schema.c](https://codecov.io/gh/danielealbano/cachegrand/pull/363?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Daniele+Salvatore+Albano#diff-c3JjL2NvbmZpZ19jeWFtbF9zY2hlbWEuYw==) | `100.00% <ø> (ø)` | | | [src/storage/db/storage\_db.h](https://codecov.io/gh/danielealbano/cachegrand/pull/363?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Daniele+Salvatore+Albano#diff-c3JjL3N0b3JhZ2UvZGIvc3RvcmFnZV9kYi5o) | `30.77% <ø> (ø)` | | | [src/support/io\_uring/io\_uring\_support.c](https://codecov.io/gh/danielealbano/cachegrand/pull/363?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Daniele+Salvatore+Albano#diff-c3JjL3N1cHBvcnQvaW9fdXJpbmcvaW9fdXJpbmdfc3VwcG9ydC5j) | `82.42% <0.00%> (-3.79%)` | :arrow_down: | | [...orker/fiber/worker\_fiber\_storage\_db\_snapshot\_rdb.c](https://codecov.io/gh/danielealbano/cachegrand/pull/363?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Daniele+Salvatore+Albano#diff-c3JjL3dvcmtlci9maWJlci93b3JrZXJfZmliZXJfc3RvcmFnZV9kYl9zbmFwc2hvdF9yZGIuYw==) | `72.22% <ø> (ø)` | | | [src/worker/storage/worker\_storage\_iouring\_op.c](https://codecov.io/gh/danielealbano/cachegrand/pull/363?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Daniele+Salvatore+Albano#diff-c3JjL3dvcmtlci9zdG9yYWdlL3dvcmtlcl9zdG9yYWdlX2lvdXJpbmdfb3AuYw==) | `87.50% <ø> (ø)` | | | [src/worker/worker\_context.h](https://codecov.io/gh/danielealbano/cachegrand/pull/363?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Daniele+Salvatore+Albano#diff-c3JjL3dvcmtlci93b3JrZXJfY29udGV4dC5o) | `100.00% <ø> (ø)` | | | [src/worker/network/worker\_network\_op.c](https://codecov.io/gh/danielealbano/cachegrand/pull/363?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Daniele+Salvatore+Albano#diff-c3JjL3dvcmtlci9uZXR3b3JrL3dvcmtlcl9uZXR3b3JrX29wLmM=) | `71.52% <50.00%> (+1.46%)` | :arrow_up: | | [src/program.c](https://codecov.io/gh/danielealbano/cachegrand/pull/363?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Daniele+Salvatore+Albano#diff-c3JjL3Byb2dyYW0uYw==) | `39.83% <60.00%> (+3.70%)` | :arrow_up: | | [src/worker/worker.c](https://codecov.io/gh/danielealbano/cachegrand/pull/363?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Daniele+Salvatore+Albano#diff-c3JjL3dvcmtlci93b3JrZXIuYw==) | `83.49% <91.67%> (+2.24%)` | :arrow_up: | | [...dule/redis/command/module\_redis\_command\_shutdown.c](https://codecov.io/gh/danielealbano/cachegrand/pull/363?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Daniele+Salvatore+Albano#diff-c3JjL21vZHVsZS9yZWRpcy9jb21tYW5kL21vZHVsZV9yZWRpc19jb21tYW5kX3NodXRkb3duLmM=) | `85.71% <100.00%> (+10.71%)` | :arrow_up: | | ... and [7 more](https://codecov.io/gh/danielealbano/cachegrand/pull/363?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Daniele+Salvatore+Albano) | | ... and [9 files with indirect coverage changes](https://codecov.io/gh/danielealbano/cachegrand/pull/363/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Daniele+Salvatore+Albano) Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Daniele+Salvatore+Albano). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Daniele+Salvatore+Albano)

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.