uber / kraken

P2P Docker registry capable of distributing TBs of data in seconds
Apache License 2.0
6.13k stars 422 forks source link

Migrate SQL and shadow backend from Uber internal repo #294

Closed SQUIDwarrior closed 3 years ago

SQUIDwarrior commented 3 years ago

This PR adds a new, SQL-based backend option for the tag datastore. The SQL backend offers increased performance over filesystem based backends for very large Docker registries as it takes advantage of optimizations such as indexing. Block-based storage systems are particularly bad at handling many small files, or walking directory structures. With the SQL backend, when listing the entire Docker catalog you are doing a simple SELECT over an indexed table, rather than a costly filesystem walk.

This also includes a backend abstraction option called a "shadow" backend. This backend is designed for transitioning Kraken to a new backend by allowing admins to specify an "active" and "shadow" backend. Writes are sent to both backends, but reads only occur from the active. This ensures data consistency between the backends, and allows the old, proven backed to act as a safety net in the case where the new backend fails or has to be taken offline due to some unforeseen problems.

codecov[bot] commented 3 years ago

Codecov Report

Merging #294 (6070a08) into master (1e07dd7) will increase coverage by 0.12%. The diff coverage is 69.81%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #294      +/-   ##
==========================================
+ Coverage   64.76%   64.89%   +0.12%     
==========================================
  Files         186      188       +2     
  Lines        7822     8044     +222     
==========================================
+ Hits         5066     5220     +154     
- Misses       2001     2037      +36     
- Partials      755      787      +32     
Impacted Files Coverage Δ
lib/backend/shadowbackend/client.go 64.94% <64.94%> (ø)
lib/backend/sqlbackend/client.go 73.60% <73.60%> (ø)
core/peer_info.go 85.71% <0.00%> (-7.15%) :arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 1e07dd7...6070a08. Read the comment docs.

codygibb commented 3 years ago

(Talked offline, posting for posterity).

Can we add the license to the top of each file?

// Copyright (c) 2016-2020 Uber Technologies, Inc.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
//     http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

Also, can we remove these shell scripts? They seem pretty bespoke and will be difficult to maintain going forward.

SQUIDwarrior commented 3 years ago

@codygibb Made updates, let me know if this looks good.