search

vesoft-inc / nebula-br

Backup and restore utility for Nebula Graph
Apache License 2.0
15 stars 11 forks source link

BR backup to s3 failed #48

Closed munichandra-nageti closed 1 year ago

munichandra-nageti commented 2 years ago

I am trying to backup my nebula-cluster data to s3 using nebula-br but facing the below issues. 

Cleanup backup BACKUP_2022_09_21_18_09_25 successfully after backup failed.Error: upload file by agent failed: rpc error: code = Unknown desc = upload from /usr/local/nebula/data/meta/nebula/0/checkpoints/BACKUP_2022_09_21_18_09_25/__disk_parts__.sst to nebula-backup-restore/BACKUP_2022_09_21_18_09_25/meta/__disk_parts__.sst failed: upload from /usr/local/nebula/data/meta/nebula/0/checkpoints/BACKUP_2022_09_21_18_09_25/__disk_parts__.sst to nebula-backup-restore/BACKUP_2022_09_21_18_09_25/meta/__disk_parts__.sst failed: SerializationError: failed to unmarshal error message
    status code: 405, request id: KMW29S7EGFNE1Y8X, host id: +bWJDdJCflu7yR8AgY7snKHajR4YZyvbYOD2qjuMDG82u+j9GNe9eAd7GVLmlrfrMXMrZPasqQE=
caused by: UnmarshalError: failed to unmarshal error message
    00000000 3c 68 74 6d 6c 3e 0a 3c 68 65 61 64 3e 3c 74 69 |<html>.<head><ti|
00000010 74 6c 65 3e 34 30 35 20 4d 65 74 68 6f 64 20 4e |tle>405 Method N|
00000020 6f 74 20 41 6c 6c 6f 77 65 64 3c 2f 74 69 74 6c |ot Allowed</titl|
00000030 65 3e 3c 2f 68 65 61 64 3e 0a 3c 62 6f 64 79 3e |e></head>.<body>|
00000040 0a 3c 68 31 3e 34 30 35 20 4d 65 74 68 6f 64 20 |.<h1>405 Method |
00000050 4e 6f 74 20 41 6c 6c 6f 77 65 64 3c 2f 68 31 3e |Not Allowed</h1>|
00000060 0a 3c 75 6c 3e 0a 3c 6c 69 3e 43 6f 64 65 3a 20 |.<ul>.<li>Code: |
00000070 4d 65 74 68 6f 64 4e 6f 74 41 6c 6c 6f 77 65 64 |MethodNotAllowed|
00000080 3c 2f 6c 69 3e 0a 3c 6c 69 3e 4d 65 73      |</li>.<li>Mes|
caused by: expected element type <Error> but have <html>

Please note that, I am able to backup the data to local storage with the same cluster setup. (edited)

wey-gu commented 2 years ago

@kqzh any clue on why the s3 post got 405?

kqzh commented 2 years ago

Hi @munichandra-nageti , I think it's your s3 setting or br args wrong which you input cause the problem, can you show your br args and check something in your s3 bucket

  1. S3 bucket naming convention is incorrect
  2. S3 bucket does not have appropriate permissions.
  3. S3 bucket name is not specified at session level.
munichandra-nageti commented 2 years ago

Hi @kqzh , There was an issue with br args(s3.epndpoint) and I have corrected it but now getting some permission errors.

BR command : sudo ./bin/br backup full --meta "127.0.0.1:9559" --s3.endpoint "http://nebula-s3-kg.s3.us-east-2.amazonaws.com/" --storage="s3://nebula-s3-kg/nebula-backup-restore/" --s3.access_key=<acc-key> --s3.secret_key=<sec-key> --s3.region=us-east-2

Error : Cleanup backup BACKUP_2022_09_26_12_34_43 successfully after backup failed.Error: upload file by agent failed: rpc error: code = Unknown desc = upload from /usr/local/nebula/data/meta/nebula/0/checkpoints/BACKUP_2022_09_26_12_34_43/disk_parts.sst to nebula-backup-restore/BACKUP_2022_09_26_12_34_43/meta/disk_parts.sst failed: upload from /usr/local/nebula/data/meta/nebula/0/checkpoints/BACKUP_2022_09_26_12_34_43/disk_parts.sst to nebula-backup-restore/BACKUP_2022_09_26_12_34_43/meta/disk_parts.sst failed: AccessDenied: Access Denied status code: 403, request id: NZV44P0AQ7YQY642, host id: ZwLRYLYwkP9KRHNaZetVGppQ4LF9TOFfoBjbIKemvXHqjVXQG6sAyTH76VEd3m3BklJ2+sIFiaY=

munichandra-nageti commented 2 years ago

Hi @kqzh , it looks like we need to assume certain IAM role inorder to access the s3 bucket mentioned above, so is there anyway to assume a specific role for nebula-br command, or do you have any plans to provision this in nebula-br command (add one more parameter something like --s3.role=<role-name>) in your upcoming releases. Thanks.

kqzh commented 2 years ago

hi @munichandra-nageti , thank you for your reply, now we can confirm that the problem lies in the permission of s3 bucket, you can check whether your access key has permission to access s3, you also can config your iam permissions in aws console like https://us-east-1.console.aws.amazon.com/iamv2/home#/users

QingZ11 commented 1 year ago

@munichandra-nageti hi, I have noticed that the issue you created hasn’t been updated for nearly a month, is this issue been resolved? If not resolved, can you provide some more information? If solved, can you close this issue?

Thanks a lot for your contribution anyway 😊