search

toeverything / OctoBase

🐙 OctoBase is the open-source database behind AFFiNE, local-first, yet collaborative. A light-weight, scalable, data engine written in Rust.
https://octobase.dev
GNU Affero General Public License v3.0
1.38k stars 91 forks source link

feat(storage): init s3 backend for blob storage #466

Closed thorseraq closed 1 year ago

thorseraq commented 1 year ago

We plan to put blob data into s3 storage, and keep local db to make metadata querying speed unaffected.

This draft PR plans to adapt S3Storage to implement BlobStorage, so we can delegate relevant operations to it. It's worth noting that some behavior under BlobStorage may not be needed, and may need further discussion.

Upstream issue: https://github.com/toeverything/OctoBase/issues/374, This PR will close #467

S3Storage was renamed to BucketStorage, which may represent more general key-value like storage medium

vercel[bot] commented 1 year ago

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name Status Preview Comments Updated (UTC)
octobase ✅ Ready (Inspect) Visit Preview 💬 Add feedback Jun 15, 2023 11:23am
codecov[bot] commented 1 year ago

Codecov Report

Patch coverage: 26.42% and project coverage change: -0.32 :warning:

Comparison is base (c94cf10) 69.96% compared to head (42aab0b) 69.65%.

:exclamation: Current head 42aab0b differs from pull request most recent head cd3e4fb. Consider uploading reports for the commit cd3e4fb to get more accurate results

Additional details and impacted files ```diff @@ Coverage Diff @@ ## master #466 +/- ## ========================================== - Coverage 69.96% 69.65% -0.32% ========================================== Files 161 164 +3 Lines 17732 17872 +140 ========================================== + Hits 12407 12448 +41 - Misses 5325 5424 +99 ``` | [Impacted Files](https://app.codecov.io/gh/toeverything/OctoBase/pull/466?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=toeverything) | Coverage Δ | | |---|---|---| | [libs/jwst-storage/src/entities/bucket\_blobs.rs](https://app.codecov.io/gh/toeverything/OctoBase/pull/466?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=toeverything#diff-bGlicy9qd3N0LXN0b3JhZ2Uvc3JjL2VudGl0aWVzL2J1Y2tldF9ibG9icy5ycw==) | `0.00% <0.00%> (ø)` | | | [.../jwst-storage/src/storage/blobs/bucket\_local\_db.rs](https://app.codecov.io/gh/toeverything/OctoBase/pull/466?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=toeverything#diff-bGlicy9qd3N0LXN0b3JhZ2Uvc3JjL3N0b3JhZ2UvYmxvYnMvYnVja2V0X2xvY2FsX2RiLnJz) | `0.00% <0.00%> (ø)` | | | [libs/jwst-storage/src/storage/blobs/local\_db.rs](https://app.codecov.io/gh/toeverything/OctoBase/pull/466?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=toeverything#diff-bGlicy9qd3N0LXN0b3JhZ2Uvc3JjL3N0b3JhZ2UvYmxvYnMvbG9jYWxfZGIucnM=) | `79.00% <0.00%> (ø)` | | | [libs/jwst-storage/src/storage/blobs/mod.rs](https://app.codecov.io/gh/toeverything/OctoBase/pull/466?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=toeverything#diff-bGlicy9qd3N0LXN0b3JhZ2Uvc3JjL3N0b3JhZ2UvYmxvYnMvbW9kLnJz) | `92.98% <ø> (ø)` | | | [libs/jwst-storage/src/types.rs](https://app.codecov.io/gh/toeverything/OctoBase/pull/466?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=toeverything#diff-bGlicy9qd3N0LXN0b3JhZ2Uvc3JjL3R5cGVzLnJz) | `0.00% <ø> (ø)` | | | [libs/jwst/src/lib.rs](https://app.codecov.io/gh/toeverything/OctoBase/pull/466?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=toeverything#diff-bGlicy9qd3N0L3NyYy9saWIucnM=) | `54.54% <ø> (ø)` | | | [libs/jwst/src/types/blob.rs](https://app.codecov.io/gh/toeverything/OctoBase/pull/466?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=toeverything#diff-bGlicy9qd3N0L3NyYy90eXBlcy9ibG9iLnJz) | `0.00% <ø> (ø)` | | | [.../src/m20230614\_000001\_initial\_bucket\_blob\_table.rs](https://app.codecov.io/gh/toeverything/OctoBase/pull/466?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=toeverything#diff-bGlicy9qd3N0LXN0b3JhZ2Uvc3JjL21pZ3JhdGlvbi9zcmMvbTIwMjMwNjE0XzAwMDAwMV9pbml0aWFsX2J1Y2tldF9ibG9iX3RhYmxlLnJz) | `79.54% <79.54%> (ø)` | | | [libs/jwst-storage/src/migration/src/lib.rs](https://app.codecov.io/gh/toeverything/OctoBase/pull/466?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=toeverything#diff-bGlicy9qd3N0LXN0b3JhZ2Uvc3JjL21pZ3JhdGlvbi9zcmMvbGliLnJz) | `100.00% <100.00%> (ø)` | | | [libs/jwst-storage/src/migration/src/schema.rs](https://app.codecov.io/gh/toeverything/OctoBase/pull/466?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=toeverything#diff-bGlicy9qd3N0LXN0b3JhZ2Uvc3JjL21pZ3JhdGlvbi9zcmMvc2NoZW1hLnJz) | `100.00% <100.00%> (ø)` | | ... and [5 files with indirect coverage changes](https://app.codecov.io/gh/toeverything/OctoBase/pull/466/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=toeverything)

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

github-actions[bot] commented 1 year ago

Benchmark Results

Linux

group                                  master                                 pr
-----                                  ------                                 --
codec/jwst decode var_int (32 bit)     1.06    346.6±5.09µs        ? ?/sec    1.00    327.4±4.82µs        ? ?/sec
codec/jwst decode var_int (64 bit)     1.00    316.2±0.32µs        ? ?/sec    1.22    386.9±2.83µs        ? ?/sec
codec/jwst decode var_uint (32 bit)    1.00    272.1±3.21µs        ? ?/sec    1.01    274.8±2.23µs        ? ?/sec
codec/jwst decode var_uint (64 bit)    1.03    292.6±4.10µs        ? ?/sec    1.00    284.8±1.55µs        ? ?/sec
codec/jwst encode var_int (32 bit)     1.00    392.0±0.66µs        ? ?/sec    1.00    391.9±0.35µs        ? ?/sec
codec/jwst encode var_int (64 bit)     1.10    434.6±5.97µs        ? ?/sec    1.00    395.9±4.35µs        ? ?/sec
codec/jwst encode var_uint (32 bit)    1.00    315.1±4.47µs        ? ?/sec    1.00    314.1±4.62µs        ? ?/sec
codec/jwst encode var_uint (64 bit)    1.00    313.9±4.04µs        ? ?/sec    1.00    313.9±3.63µs        ? ?/sec
codec/lib0 decode var_int (64 bit)     1.00    270.8±1.35µs        ? ?/sec    1.03    278.9±5.49µs        ? ?/sec
codec/lib0 decode var_uint (32 bit)    1.03    264.2±2.72µs        ? ?/sec    1.00    255.9±7.15µs        ? ?/sec
codec/lib0 decode var_uint (64 bit)    1.06    276.6±5.98µs        ? ?/sec    1.00    262.0±5.12µs        ? ?/sec
codec/lib0 encode var_int (64 bit)     1.00    384.5±5.86µs        ? ?/sec    1.25   479.3±14.62µs        ? ?/sec
codec/lib0 encode var_uint (32 bit)    1.00    300.9±7.29µs        ? ?/sec    1.02    305.8±2.50µs        ? ?/sec
codec/lib0 encode var_uint (64 bit)    1.00    298.5±3.20µs        ? ?/sec    1.15    342.9±0.91µs        ? ?/sec
operations/jwst array insert           1.00    255.4±0.46µs        ? ?/sec    1.01    256.7±0.20µs        ? ?/sec
operations/jwst array insert range     1.00   469.4±29.96µs        ? ?/sec    1.00   469.3±29.04µs        ? ?/sec
operations/yrs array insert            1.00     85.0±1.31µs        ? ?/sec    1.07     91.1±0.26µs        ? ?/sec
operations/yrs array insert range      1.00    113.3±0.83µs        ? ?/sec    1.04    118.0±0.62µs        ? ?/sec
update/parse with jwst/basic.bin       1.00     60.2±0.21µs    90.7 MB/sec    1.01     60.8±0.22µs    89.8 MB/sec
update/parse with jwst/database.bin    1.00     65.3±0.39µs    50.8 MB/sec    1.00     65.6±0.23µs    50.5 MB/sec
update/parse with jwst/large.bin       1.01      4.1±0.01ms    55.2 MB/sec    1.00      4.1±0.01ms    55.9 MB/sec
update/parse with yrs/basic.bin        1.00     41.4±0.15µs   132.1 MB/sec    1.00     41.2±0.04µs   132.4 MB/sec
update/parse with yrs/database.bin     1.00     33.3±0.02µs    99.4 MB/sec    1.00     33.2±0.02µs    99.8 MB/sec
update/parse with yrs/large.bin        1.00      2.2±0.00ms   105.3 MB/sec    1.00      2.2±0.01ms   105.0 MB/sec

Windows

group                                  master                                 pr
-----                                  ------                                 --
codec/jwst decode var_int (32 bit)     1.00   342.1±18.70µs        ? ?/sec    1.05   357.6±23.90µs        ? ?/sec
codec/jwst decode var_int (64 bit)     1.00   339.3±14.32µs        ? ?/sec    1.00   340.8±17.44µs        ? ?/sec
codec/jwst decode var_uint (32 bit)    1.02   303.0±18.13µs        ? ?/sec    1.00   298.5±15.16µs        ? ?/sec
codec/jwst decode var_uint (64 bit)    1.02   309.8±18.73µs        ? ?/sec    1.00   303.4±14.69µs        ? ?/sec
codec/jwst encode var_int (32 bit)     1.00   414.5±27.03µs        ? ?/sec    1.06   439.5±37.72µs        ? ?/sec
codec/jwst encode var_int (64 bit)     1.01   407.5±25.24µs        ? ?/sec    1.00   404.2±22.01µs        ? ?/sec
codec/jwst encode var_uint (32 bit)    1.00   274.4±12.39µs        ? ?/sec    1.03   283.9±19.43µs        ? ?/sec
codec/jwst encode var_uint (64 bit)    1.02   279.3±20.99µs        ? ?/sec    1.00   273.3±12.72µs        ? ?/sec
codec/lib0 decode var_int (64 bit)     1.00   276.9±13.25µs        ? ?/sec    1.01   278.6±14.03µs        ? ?/sec
codec/lib0 decode var_uint (32 bit)    1.01   243.8±14.11µs        ? ?/sec    1.00   242.0±13.78µs        ? ?/sec
codec/lib0 decode var_uint (64 bit)    1.02   239.9±12.36µs        ? ?/sec    1.00    234.1±8.65µs        ? ?/sec
codec/lib0 encode var_int (64 bit)     1.02   413.9±30.75µs        ? ?/sec    1.00   404.7±19.49µs        ? ?/sec
codec/lib0 encode var_uint (32 bit)    1.00   274.4±17.13µs        ? ?/sec    1.06   292.1±12.53µs        ? ?/sec
codec/lib0 encode var_uint (64 bit)    1.07   300.4±25.05µs        ? ?/sec    1.00   280.6±13.81µs        ? ?/sec
operations/jwst array insert           1.10   319.3±38.72µs        ? ?/sec    1.00   289.4±11.57µs        ? ?/sec
operations/yrs array insert            1.01    100.7±3.71µs        ? ?/sec    1.00     99.3±3.41µs        ? ?/sec
update/parse with jwst/basic.bin       1.00    113.9±4.60µs    47.9 MB/sec    1.03    116.9±4.53µs    46.7 MB/sec
update/parse with jwst/database.bin    1.00     91.5±3.06µs    36.2 MB/sec    1.00     91.8±3.34µs    36.1 MB/sec
update/parse with jwst/large.bin       1.02      5.8±0.30ms    39.1 MB/sec    1.00      5.7±0.26ms    39.9 MB/sec
update/parse with yrs/basic.bin        1.00     66.0±3.46µs    82.7 MB/sec    1.03     67.7±3.37µs    80.6 MB/sec
update/parse with yrs/database.bin     1.00     50.8±1.87µs    65.3 MB/sec    1.04     52.6±2.31µs    62.9 MB/sec
update/parse with yrs/large.bin        1.01      3.1±0.17ms    73.0 MB/sec    1.00      3.1±0.13ms    74.0 MB/sec