pingcap-inc / tidb2dw

Replicate data changes from TiDB to Data Warehouses in real-time
MIT License
13 stars 7 forks source link

feature: support aws s3 as a target type #64

Closed Icemap closed 1 year ago

Icemap commented 1 year ago

Support AWS S3 not only as a temporary storage but also as a final target.

$ ./bin/tidb2dw s3 --help
Replicate snapshot and incremental data from TiDB to AWS S3

Usage:
  tidb2dw s3 [flags]

Flags:
      --api.host string               API service listen host, only available in --mode=cloud (default "0.0.0.0")
      --api.port int                  API service listen port, only available in --mode=cloud (default 8185)
      --aws.access-key string         aws access key
      --aws.secret-key string         aws secret key
      --cdc.file-size int              (default 67108864)
      --cdc.flush-interval duration    (default 1m0s)
      --cdc.host string               TiCDC server host (default "127.0.0.1")
      --cdc.port int                  TiCDC server port (default 8300)
      --help                          help for this command
      --log.file string               log file path
      --log.level string              log level (default "info")
      --mode mode                     replication mode: full, snapshot-only, incremental-only, cloud (default full)
      --snapshot-concurrency int      the number of concurrent snapshot workers (default 8)
  -s, --storage string                storage path: s3://<bucket>/<path> or gcs://<bucket>/<path>
  -t, --table stringArray             tables full qualified name, e.g. -t <db1>.<table1> -t <db2>.<table2>
  -h, --tidb.host string              TiDB host (default "127.0.0.1")
  -p, --tidb.pass string              TiDB password
  -P, --tidb.port int                 TiDB port (default 4000)
      --tidb.ssl-ca string            TiDB SSL CA
  -u, --tidb.user string              TiDB user (default "root")
      --tz string                     specify time zone of storage consumer (default "System")
Icemap commented 1 year ago

Change another way to archive it.