search

Websoft9Archive / role_os

useful tools for OS image, e.g GNOME
Other
0 stars 1 forks source link

Use rclone, copy azure's vhd to aliyun's oss #13

Closed qiaofeng1227 closed 2 years ago

qiaofeng1227 commented 2 years ago

rclone:https://github.com/rclone/rclone

qiaofeng1227 commented 2 years ago

Test: User rclone, copy websoft9's vhd to yourself oss.

qiaofeng1227 commented 2 years ago 
[root@websoft9linuxnodesktop-OracleLinux7 websoft9]# cloud-init --version
/bin/cloud-init 19.4
Preparing...                          ################################# [100%]
Updating / installing...
   1:aliyun_assist-2.2.3.247-1        ################################# [100%]
install linux aliyun-assist at /usr/local/share/aliyun-assist/2.2.3.247 ...
install assist ok
start assist ok
need to install assist_daemon
create ok
start ok
qiaofeng1227 commented 2 years ago

websoft9_linuxnodesktop-OracleLinux7.9.vhd

qiaofeng1227 commented 2 years ago

azure下载链接

zengxc-1996 commented 2 years ago
  1. use rclone install rclone: 
    curl https://rclone.org/install.sh | sudo bash
  2. rclone config 
    
[root@zengxc-vpn ~]# rclone config
No remotes found - make a new one
n) New remote
s) Set configuration password
q) Quit config
n/s/q> n
name> oss
Type of storage to configure.
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
1 / 1Fichier
\ "fichier"
2 / Alias for an existing remote
\ "alias"
3 / Amazon Drive
\ "amazon cloud drive"
4 / Amazon S3 Compliant Storage Providers including AWS, Alibaba, Ceph, Digital Ocean, Dreamhost, IBM COS, Minio, SeaweedFS, and Tencent COS
\ "s3"
5 / Backblaze B2
\ "b2"
6 / Box
\ "box"
7 / Cache a remote
\ "cache"
8 / Citrix Sharefile
\ "sharefile"
9 / Compress a remote
\ "compress"
10 / Dropbox
\ "dropbox"
11 / Encrypt/Decrypt a remote
\ "crypt"
12 / Enterprise File Fabric
\ "filefabric"
13 / FTP Connection
\ "ftp"
14 / Google Cloud Storage (this is not Google Drive)
\ "google cloud storage"
15 / Google Drive
\ "drive"
16 / Google Photos
\ "google photos"
17 / Hadoop distributed file system
\ "hdfs"
18 / Hubic
\ "hubic"
19 / In memory object storage system.
\ "memory"
20 / Jottacloud
\ "jottacloud"
21 / Koofr
\ "koofr"
22 / Local Disk
\ "local"
23 / Mail.ru Cloud
\ "mailru"
24 / Mega
\ "mega"
25 / Microsoft Azure Blob Storage
\ "azureblob"
26 / Microsoft OneDrive
\ "onedrive"
27 / OpenDrive
\ "opendrive"
28 / OpenStack Swift (Rackspace Cloud Files, Memset Memstore, OVH)
\ "swift"
29 / Pcloud
\ "pcloud"
30 / Put.io
\ "putio"
31 / QingCloud Object Storage
\ "qingstor"
32 / SSH/SFTP Connection
\ "sftp"
33 / Sugarsync
\ "sugarsync"
34 / Tardigrade Decentralized Cloud Storage
\ "tardigrade"
35 / Transparently chunk/split large files
\ "chunker"
36 / Union merges the contents of several upstream fs
\ "union"
37 / Uptobox
\ "uptobox"
38 / Webdav
\ "webdav"
39 / Yandex Disk
\ "yandex"
40 / Zoho
\ "zoho"
41 / http Connection
\ "http"
42 / premiumize.me
\ "premiumizeme"
43 / seafile
\ "seafile"
Storage> s3
Choose your S3 provider.
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
1 / Amazon Web Services (AWS) S3
\ "AWS"
2 / Alibaba Cloud Object Storage System (OSS) formerly Aliyun
\ "Alibaba"
3 / Ceph Object Storage
\ "Ceph"
4 / Digital Ocean Spaces
\ "DigitalOcean"
5 / Dreamhost DreamObjects
\ "Dreamhost"
6 / IBM COS S3
\ "IBMCOS"
7 / Minio Object Storage
\ "Minio"
8 / Netease Object Storage (NOS)
\ "Netease"
9 / Scaleway Object Storage
\ "Scaleway"
10 / SeaweedFS S3
\ "SeaweedFS"
11 / StackPath Object Storage
\ "StackPath"
12 / Tencent Cloud Object Storage (COS)
\ "TencentCOS"
13 / Wasabi Object Storage
\ "Wasabi"
14 / Any other S3 compatible provider
\ "Other"
provider> Alibaba
Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars).
Only applies if access_key_id and secret_access_key is blank.
Enter a boolean value (true or false). Press Enter for the default ("false").
Choose a number from below, or type in your own value
1 / Enter AWS credentials in the next step
\ "false"
2 / Get AWS credentials from the environment (env vars or IAM)
\ "true"
env_auth> 1
AWS Access Key ID.
Leave blank for anonymous access or runtime credentials.
Enter a string value. Press Enter for the default ("").
access_key_id> oss_key_id 
AWS Secret Access Key (password)
Leave blank for anonymous access or runtime credentials.
Enter a string value. Press Enter for the default ("").
secret_access_key> oss_key         
Endpoint for OSS API.
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
1 / Global Accelerate
\ "oss-accelerate.aliyuncs.com"
2 / Global Accelerate (outside mainland China)
\ "oss-accelerate-overseas.aliyuncs.com"
3 / East China 1 (Hangzhou)
\ "oss-cn-hangzhou.aliyuncs.com"
4 / East China 2 (Shanghai)
\ "oss-cn-shanghai.aliyuncs.com"
5 / North China 1 (Qingdao)
\ "oss-cn-qingdao.aliyuncs.com"
6 / North China 2 (Beijing)
\ "oss-cn-beijing.aliyuncs.com"
7 / North China 3 (Zhangjiakou)
\ "oss-cn-zhangjiakou.aliyuncs.com"
8 / North China 5 (Hohhot)
\ "oss-cn-huhehaote.aliyuncs.com"
9 / North China 6 (Ulanqab)
\ "oss-cn-wulanchabu.aliyuncs.com"
10 / South China 1 (Shenzhen)
\ "oss-cn-shenzhen.aliyuncs.com"
11 / South China 2 (Heyuan)
\ "oss-cn-heyuan.aliyuncs.com"
12 / South China 3 (Guangzhou)
\ "oss-cn-guangzhou.aliyuncs.com"
13 / West China 1 (Chengdu)
\ "oss-cn-chengdu.aliyuncs.com"
14 / Hong Kong (Hong Kong)
\ "oss-cn-hongkong.aliyuncs.com"
15 / US West 1 (Silicon Valley)
\ "oss-us-west-1.aliyuncs.com"
16 / US East 1 (Virginia)
\ "oss-us-east-1.aliyuncs.com"
17 / Southeast Asia Southeast 1 (Singapore)
\ "oss-ap-southeast-1.aliyuncs.com"
18 / Asia Pacific Southeast 2 (Sydney)
\ "oss-ap-southeast-2.aliyuncs.com"
19 / Southeast Asia Southeast 3 (Kuala Lumpur)
\ "oss-ap-southeast-3.aliyuncs.com"
20 / Asia Pacific Southeast 5 (Jakarta)
\ "oss-ap-southeast-5.aliyuncs.com"
21 / Asia Pacific Northeast 1 (Japan)
\ "oss-ap-northeast-1.aliyuncs.com"
22 / Asia Pacific South 1 (Mumbai)
\ "oss-ap-south-1.aliyuncs.com"
23 / Central Europe 1 (Frankfurt)
\ "oss-eu-central-1.aliyuncs.com"
24 / West Europe (London)
\ "oss-eu-west-1.aliyuncs.com"
25 / Middle East 1 (Dubai)
\ "oss-me-east-1.aliyuncs.com"
endpoint> 3
Canned ACL used when creating buckets and storing or copying objects.

This ACL is used for creating objects and if bucket_acl isn't set, for creating buckets too.

For more info visit https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl

Note that this ACL is applied when server-side copying objects as S3 doesn't copy the ACL from the source but rather writes a fresh one. Enter a string value. Press Enter for the default (""). Choose a number from below, or type in your own value 1 / Owner gets FULL_CONTROL. No one else has access rights (default). \ "private" 2 / Owner gets FULL_CONTROL. The AllUsers group gets READ access. \ "public-read" / Owner gets FULL_CONTROL. The AllUsers group gets READ and WRITE access. 3 | Granting this on a bucket is generally not recommended. \ "public-read-write" 4 / Owner gets FULL_CONTROL. The AuthenticatedUsers group gets READ access. \ "authenticated-read" / Object owner gets FULL_CONTROL. Bucket owner gets READ access. 5 | If you specify this canned ACL when creating a bucket, Amazon S3 ignores it. \ "bucket-owner-read" / Both the object owner and the bucket owner get FULL_CONTROL over the object. 6 | If you specify this canned ACL when creating a bucket, Amazon S3 ignores it. \ "bucket-owner-full-control" acl> 1 The storage class to use when storing new objects in OSS. Enter a string value. Press Enter for the default (""). Choose a number from below, or type in your own value 1 / Default \ "" 2 / Standard storage class \ "STANDARD" 3 / Archive storage mode. \ "GLACIER" 4 / Infrequent access storage mode. \ "STANDARD_IA" storage_class> 1 Edit advanced config? y) Yes n) No (default) y/n> n

[oss] type = s3 provider = Alibaba access_key_id = oss_key_id secret_access_key = oss_key endpoint = oss-cn-hangzhou.aliyuncs.com acl = private

y) Yes this is OK (default) e) Edit this remote d) Delete this remote y/e/d> y Current remotes:

Name Type ==== ==== oss s3

e) Edit existing remote n) New remote d) Delete remote r) Rename remote c) Copy remote s) Set configuration password q) Quit config e/n/d/r/c/s/q> 

configure file path:  /root/.config/rclone/rclone.conf

3.  *To be verified， please give me oss Authentication account password*
List oss remote file tree：

reclone ls oss:rclone-os

4. Copy url resources to oss

rclone copyurl URL oss:rclone-oss


Test tomorrow. Please contact me if you have any questions.
qiaofeng1227 commented 2 years ago

download test: wget on a hongkong server: 7M/s rclone copyurl URL oss:zhangjiakou :100K rclone copyurl URL oss:hongkong :3M/s

conclusion: The main problem affecting the speed is the region of the server, but rclone can also be tuned

zengxc-1996 commented 2 years ago

uploding oss:

nohup rclone copyurl -a -p -P  "https://websoft9imagedisks.blob.core.windows.net/vhds/websoft9_linuxnodesktop-OracleLinux7.9.vhd?sp=rl&st=2021-10-02T05:39:00Z&se=2021-12-03T05:39:00Z&sv=2020-08-04&sr=c&sig=8BpZjguP3VEPUbcQDE%2F9iYH0bMcPjxPV4d8EfOtnyu8%3D" oss:websoft9xg > logs &

logs:

tail -f logs

config file:

[root@VM-66-244-centos ~]# cat /root/.config/rclone/rclone.conf 
[oss]
type = s3
provider = Alibaba
access_key_id = key_id
secret_access_key = key
endpoint = oss-cn-hongkong.aliyuncs.com
acl = public-read-write
bucket_acl = public-read-write
chunk_size = 10M
disable_checksum = true
zengxc-1996 commented 2 years ago

Change the chunk_size size to change the upload speed.

If the memory keeps increasing, the Settings are too large.

zengxc-1996 commented 2 years ago

image

qiaofeng1227 commented 2 years ago

The server execute rclone's shell will consumption flow:

before rclone: image

start：rclone image

image

qiaofeng1227 commented 2 years ago

API:https://help.aliyun.com/document_detail/25542.html doc:https://help.aliyun.com/document_detail/25464.html