clyso / chorus

s3 multi provider data lifecycle management
Apache License 2.0
64 stars 7 forks source link
backup go migration rclone replication s3 s3-storage

CI GitHub go.mod Go version GoDoc Go Report Card

Chorus

chorus.png

Chorus is vendor-agnostic S3-compatible backup, replication, and routing software. Once configured it can:

Listed features can be configured per S3 user and per bucket with management CLI or webUI.

Components

Chorus S3 Proxy service responsible for routing s3 requests and capturing data change events. Chorus Agent can be used as an alternative solution for capturing events instead of proxy. Chorus Worker service does actual data replication with the help of RClone. Communication between Proxy/Agent and worker is done over work queue. Asynq with Redis is used as a work queue.

diagram.png

For more details, see:

Documentation

Documentation available at docs.clyso.com.

Run

From source

REQUIREMENTS:

Run all-in-one standalone binry from Go:

go run ./cmd/chorus

Or run each service separately:

REQUIREMENTS:

# run chorus worker
go run ./cmd/worker

# run chorus worker with a custom YAML config file
go run ./cmd/worker -config <path to worker yaml config>

# run chorus proxy with a custom YAML config file
go run ./cmd/proxy -config <path to proxy yaml config>

# run chorus agent with a custom YAML config file
go run ./cmd/agent -config <path to agent yaml config>

Standalone binary

See: service/standalone

Docker-compose

REQUIREMENTS:

See: docker-compose

With Helm

REQUIREMENTS:

Install the Chorus Helm chart from the OCI registry:

helm install <release name> oci://harbor.clyso.com/chorus/chorus

See: deploy/chorus

Develop

test package contains e2e tests for replications between s3 storages. It runs:

all listed tools are written in go so test can be run without external dependencies just by:

go test ./test/...