yzsfly
commented
3 years ago we met the same issue that the compressed file cannot be upload into OSS. Could anyone provide support? Urgent for me ,thanks!
Open vczyh opened 3 years ago
yzsfly
commented
3 years ago we met the same issue that the compressed file cannot be upload into OSS. Could anyone provide support? Urgent for me ,thanks!
vczyh
commented
3 years ago we met the same issue that the compressed file cannot be upload into OSS. Could anyone provide support? Urgent for me ,thanks!
We have solved the problem. We have changed backup_storage_implementation to s3 before using ceph. Here are the steps:
vttablet \
$TOPOLOGY_FLAGS \
-log_dir $VTDATAROOT/tmp \
-log_queries_to_file $VTDATAROOT/tmp/$tablet_logfile \
-tablet-path $alias \
-tablet_hostname "$tablet_hostname" \
-init_keyspace $keyspace \
-init_shard $shard \
-init_tablet_type $tablet_type \
-health_check_interval 5s \
-enable_semi_sync \
-enable_replication_reporter \
-backup_storage_implementation s3 \
-s3_backup_aws_endpoint "http://north1.minio.local" \
-s3_backup_force_path_style \
-s3_backup_storage_root "/root/data" \
-s3_backup_storage_bucket "commerce" \
-backup_engine_implementation xtrabackup \
-xtrabackup_root_path "/usr/bin" \
-xtrabackup_user root \
-xtrabackup_stream_mode xbstream \
-restore_from_backup \
-port $port \
-grpc_port $grpc_port \
-service_map 'grpc-queryservice,grpc-tabletmanager,grpc-updatestream' \
-pid_file $VTDATAROOT/$tablet_dir/vttablet.pid \
-vtctld_addr http://$hostname:$vtctld_web_port/ \
> $VTDATAROOT/$tablet_dir/vttablet.out 2>&1 &export AWS_ACCESS_KEY_ID="minioadmin"
export AWS_SECRET_ACCESS_KEY="minioadmin"
grsmith-projects
commented
3 years ago I can confirm the same behavior -- it seems to potentially be an issue with the minio version the client uses.
askdba
commented
3 years ago @grsmith-projects does that mean the built-in backup works? Have you tried that?
grsmith-projects
commented
3 years ago @askdba built-in does not work
The backup succeeds, the errors occur when the files start being transferred -- only 1 file succeeds and the rest fail due to the connection being signalled to exit.
Also, we rebuilt the system w/ various versions of minio lib (6.x and 7.x) and had the same issue.
My gut feeling right now is that some default setting for miniolib might be conflicting with the way newer ceph does multipart uploads -- we are currently investigating tuning on the ceph side.
Minio client itself works
Minio client from a container in the same pod works.. but vtbackup does not.
(All of this is speculation at this point)
deepthi
commented
3 years ago from ceph.go:
client, err := minio.NewV2(url, accessKey, secretKey, useSSL)Could it be that your server does not support v2 clients?
L3o-pold
commented
11 months ago I can reproduce the issue with vitess/lite:v18.0.0-rc1 as s3 backup_storage_implementation no longer works too with Minio. On the server side we see 499 http status code when trying to upload.
I tried with minio.NewV4 without more luck. Also tried by updating minio client to github.com/minio/minio-go/v7 still no luck...
Overview
Hi~, I try the local demo , everything was fine at first, but the error happend when I backed an instance up.
Only one
MANIFESTfile is stored on OSS.backup.xbstream.gznot found.And error information
So I need some help, THANK YOU!
Version Infomation
Reproduction Steps
Steps to reproduce this issue, example:
Modify scripts/vtctld-up.sh
Modify scripts/vttablet-up.sh
Ceph config(
/etc/ceph_conf.json)Run local demo
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 0.0.0.0:22 0.0.0.0: LISTEN - tcp 0 0 127.0.0.1:2379 0.0.0.0: LISTEN 124382/etcd tcp 0 0 127.0.0.1:2380 0.0.0.0: LISTEN 124382/etcd tcp6 0 0 :::17102 ::: LISTEN 124674/mysqld tcp6 0 0 :::22 ::: LISTEN - tcp6 0 0 :::15991 ::: LISTEN 124802/vtgate tcp6 0 0 :::15000 ::: LISTEN 124419/vtctld tcp6 0 0 :::15001 ::: LISTEN 124802/vtgate tcp6 0 0 :::15100 ::: LISTEN 124504/vttablet tcp6 0 0 :::15101 ::: LISTEN 124607/vttablet tcp6 0 0 :::15102 ::: LISTEN 124709/vttablet tcp6 0 0 :::15999 ::: LISTEN 124419/vtctld tcp6 0 0 :::16100 ::: LISTEN 124504/vttablet tcp6 0 0 :::16101 ::: LISTEN 124607/vttablet tcp6 0 0 :::16102 ::: LISTEN 124709/vttablet tcp6 0 0 :::15306 ::: LISTEN 124802/vtgate tcp6 0 0 :::17100 ::: LISTEN 124469/mysqld tcp6 0 0 :::17101 ::: LISTEN 124569/mysqld
My Question
MANIFESTupload successfully means OSS work well. I don't understand why tar upload failed. Maybe I ignore some flags or version Incompatibility.Please give me some advice.
If need more logs and information, I can provide it.
THANK YOU!