search

star-whale / starwhale

an MLOps/LLMOps platform
https://starwhale.ai
Apache License 2.0
208 stars 35 forks source link

feat(datastore): support checkpoint in server side #3093

Open jialeicui opened 10 months ago

jialeicui commented 10 months ago

Description

Provides support for checkpoints, which later allows:

  1. Only needing to save the data version indicated by the checkpoint, effectively reducing the disk and memory space occupied by the data
  2. Fetch the statistical information of the data pointed by a checkpoint with constant complexity, including data row count, minimum key, maximum key, etc.

Issues to be considered: #3098

  1. It is necessary to support old versions of data, and a migration logic is needed to ensure that all historical data checkpoints can be correctly processed.
  2. Consider whether a Controller can rollback after an upgrade (data versions might have been garbage collected due to the checkpoint function).

Modules

Checklist

codecov[bot] commented 10 months ago

Codecov Report

Attention: 20 lines in your changes are missing coverage. Please review.

Comparison is base (cf58e90) 82.68% compared to head (ff4446e) 82.74%.

Files Patch % Lines
...tarwhale/mlops/datastore/impl/MemoryTableImpl.java 89.18% 6 Missing and 10 partials :warning:
.../java/ai/starwhale/mlops/datastore/Checkpoint.java 87.50% 0 Missing and 2 partials :warning:
...n/java/ai/starwhale/mlops/datastore/DataStore.java 90.00% 1 Missing and 1 partial :warning:
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #3093 +/- ## ============================================ + Coverage 82.68% 82.74% +0.06% - Complexity 3182 3232 +50 ============================================ Files 573 575 +2 Lines 31969 32181 +212 Branches 1865 1897 +32 ============================================ + Hits 26433 26628 +195 - Misses 4712 4715 +3 - Partials 824 838 +14 ``` | [Flag](https://app.codecov.io/gh/star-whale/starwhale/pull/3093/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=star-whale) | Coverage Δ | | |---|---|---| | [console](https://app.codecov.io/gh/star-whale/starwhale/pull/3093/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=star-whale) | `72.09% <ø> (ø)` | | | [controller](https://app.codecov.io/gh/star-whale/starwhale/pull/3093/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=star-whale) | `73.55% <89.94%> (+0.19%)` | :arrow_up: | | [standalone](https://app.codecov.io/gh/star-whale/starwhale/pull/3093/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=star-whale) | `91.89% <100.00%> (+0.03%)` | :arrow_up: | | [unittests](https://app.codecov.io/gh/star-whale/starwhale/pull/3093/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=star-whale) | `91.63% <100.00%> (+0.03%)` | :arrow_up: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=star-whale#carryforward-flags-in-the-pull-request-comment) to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.