search

skyplane-project / skyplane

🔥 Blazing fast bulk data transfers between any cloud 🔥
https://skyplane.org
Apache License 2.0
1.08k stars 62 forks source link

[bug] AWS S3-to-S3 cp, PendingVerification error #799

Closed connerkdmi01 closed 1 year ago

connerkdmi01 commented 1 year ago

Describe the bug AWS PendingVerification error when cp'ing from us-west-2 to us-east-2. Transferring small file but with native_cmd_enabled = False to force use of Skyplane. Checked that AWS account is verified for all us regions (verified by default).

To Reproduce Steps to reproduce the behavior (please include the full Skyplane command you ran):

  1. Set up two s3 buckets on AWS, upload small file to one of them.
  2. Disable native_cmd_enabled with skyplane config set native_cmd_enabled false
  3. skyplane cp -r s3://skyplane-test-1 s3://skyplane-test-2
  4. See error

Expected behavior Successful cp from first bucket to second bucket.

Screenshots If applicable, add screenshots to help explain your problem.

Transfer client log In the log output from Skyplane, please upload the debug log from the CLI. You can find the path to the file in the log output:

18:08:25 [INFO]  [SkyplaneClient.direct_dataplane] Topology: {"replication_topology_edges": [{"src": {"type": "ReplicationTopologyObjectStore", "fields": {"region": "aws:us-west-2"}}, "dest": {"type": "ReplicationTopologyGateway", "fields": {"region": "aws:us-west-2", "instance": 0}}, "num_connections": 0}, {"src": {"type": "ReplicationTopologyGateway", "fields": {"region": "aws:us-west-2", "instance": 0}}, "dest": {"type": "ReplicationTopologyGateway", "fields": {"region": "aws:us-east-2", "instance": 0}}, "num_connections": 32}, {"src": {"type": "ReplicationTopologyGateway", "fields": {"region": "aws:us-east-2", "instance": 0}}, "dest": {"type": "ReplicationTopologyObjectStore", "fields": {"region": "aws:us-east-2"}}, "num_connections": 0}]}
18:08:25 [DEBUG] Using dataplane: <skyplane.api.dataplane.Dataplane object at 0x000001D779ECED10>
18:08:25 [INFO]  [SkyplaneClient.direct_dataplane] Topology: {"replication_topology_edges": [{"src": {"type": "ReplicationTopologyObjectStore", "fields": {"region": "aws:us-west-2"}}, "dest": {"type": "ReplicationTopologyGateway", "fields": {"region": "aws:us-west-2", "instance": 0}}, "num_connections": 0}, {"src": {"type": "ReplicationTopologyGateway", "fields": {"region": "aws:us-west-2", "instance": 0}}, "dest": {"type": "ReplicationTopologyGateway", "fields": {"region": "aws:us-east-2", "instance": 0}}, "num_connections": 32}, {"src": {"type": "ReplicationTopologyGateway", "fields": {"region": "aws:us-east-2", "instance": 0}}, "dest": {"type": "ReplicationTopologyObjectStore", "fields": {"region": "aws:us-east-2"}}, "num_connections": 0}]}
18:08:25 [DEBUG] Using dataplane: <skyplane.api.dataplane.Dataplane object at 0x000001D779ECEA10>
18:08:25 [DEBUG] [SkyplaneClient] Queued copy job CopyJob(src_path='s3://skyplane-test-1', dst_path='s3://skyplane-test-2', recursive=True, requester_pays=False, uuid='546a5147-d68b-4464-a8f3-8c49069a74d7', transfer_list=[], multipart_transfer_list=[])
18:08:25 [DEBUG] Generating transfer pairs for external access, <skyplane.obj_store.s3_interface.S3Interface object at 0x000001D779ECF850> -> <skyplane.obj_store.s3_interface.S3Interface object at 0x000001D779ECFCD0>
18:08:26 [DEBUG] Querying objects in s3://skyplane-test-1
18:09:30 [DEBUG] User confirmed transfer
18:09:30 [INFO]  [Provisioner.add_task] Queue ProvisionerTask(cloud_provider='aws', region='us-east-2', vm_type='m5.8xlarge', spot=False, autoterminate_minutes=15, tags={'skyplane': 'true', 'skyplaneclientid': '00000000000000000000f8e4e33dc63c'}, uuid='c95af13a-bf34-410f-b5ee-077416fa252d')
18:09:30 [INFO]  [Provisioner.add_task] Queue ProvisionerTask(cloud_provider='aws', region='us-west-2', vm_type='m5.8xlarge', spot=False, autoterminate_minutes=15, tags={'skyplane': 'true', 'skyplaneclientid': '00000000000000000000f8e4e33dc63c'}, uuid='f45d3cd7-efec-41db-89e5-2321938cc1df')
18:09:30 [INFO]  [Provisioner.init_global] Initializing global resources for aws=True, azure=False, gcp=False
18:09:37 [DEBUG] [aws_network]:us-west-2 Authorizing SSH 0.0.0.0/0:22 in skyplane
18:09:38 [DEBUG] [aws_network]:us-east-2 Authorizing SSH 0.0.0.0/0:22 in skyplane
18:09:39 [DEBUG] [AWS] Creating keypair skyplane-00000000000000000000f8e4e33dc63c-us-west-2 in us-west-2
18:09:40 [DEBUG] [AWS] Creating keypair skyplane-00000000000000000000f8e4e33dc63c-us-east-2 in us-east-2
18:09:40 [INFO]  [Provisioner.provision] Configured AWS regions {'us-west-2', 'us-east-2'}
18:09:40 [INFO]  [Provisioner.provision] Provisioning 2 VMs
18:09:43 [DEBUG] [wait_for] Waiting fn=<function AWSCloudProvider.provision_instance.<locals>.check_iam_role at 0x000001D701543D90> completed in 0.36s
18:09:43 [DEBUG] [wait_for] Waiting fn=<function AWSCloudProvider.provision_instance.<locals>.check_iam_role at 0x000001D7015A2170> completed in 0.32s
18:09:44 [DEBUG] [wait_for] Waiting fn=<function AWSCloudProvider.provision_instance.<locals>.check_instance_profile at 0x000001D7015D03A0> completed in 0.11s
18:09:44 [DEBUG] [wait_for] Waiting fn=<function AWSCloudProvider.provision_instance.<locals>.check_instance_profile at 0x000001D7015D0280> completed in 0.10s
18:09:57 [WARN]  An error occurred (PendingVerification) when calling the RunInstances operation: Your request for accessing resources in this region is being validated, and you will not be able to launch additional resources in this region until the validation is complete. We will notify you by email once your request has been validated. While normally resolved within minutes, please allow up to 4 hours for this process to complete. If the issue still persists, please let us know by writing to aws-verification@amazon.com for further assistance.
18:10:06 [WARN]  An error occurred (PendingVerification) when calling the RunInstances operation: Your request for accessing resources in this region is being validated, and you will not be able to launch additional resources in this region until the validation is complete. We will notify you by email once your request has been validated. While normally resolved within minutes, please allow up to 4 hours for this process to complete. If the issue still persists, please let us know by writing to aws-verification@amazon.com for further assistance.
18:10:12 [DEBUG] [Provisioner._provision_task] Provisioned AWSServer(region_tag=aws:us-west-2, instance_id=i-07643c7a06042a89f) in 31.28s
18:10:12 [DEBUG] [wait_for] Waiting for aws:us-west-2:i-07643c7a06042a89f to be ready fn=<function Server.wait_for_ssh_ready.<locals>.is_up at 0x000001D77ED56EF0> completed in 0.46s
18:10:16 [WARN]  An error occurred (PendingVerification) when calling the RunInstances operation: Your request for accessing resources in this region is being validated, and you will not be able to launch additional resources in this region until the validation is complete. We will notify you by email once your request has been validated. While normally resolved within minutes, please allow up to 4 hours for this process to complete. If the issue still persists, please let us know by writing to aws-verification@amazon.com for further assistance.
18:10:25 [WARN]  An error occurred (PendingVerification) when calling the RunInstances operation: Your request for accessing resources in this region is being validated, and you will not be able to launch additional resources in this region until the validation is complete. We will notify you by email once your request has been validated. While normally resolved within minutes, please allow up to 4 hours for this process to complete. If the issue still persists, please let us know by writing to aws-verification@amazon.com for further assistance.
18:10:35 [ERROR] Error running _provision_task: An error occurred (PendingVerification) when calling the RunInstances operation: Your request for accessing resources in this region is being validated, and you will not be able to launch additional resources in this region until the validation is complete. We will notify you by email once your request has been validated. While normally resolved within minutes, please allow up to 4 hours for this process to complete. If the issue still persists, please let us know by writing to aws-verification@amazon.com for further assistance.
18:10:35 [ERROR] Exception: An error occurred (PendingVerification) when calling the RunInstances operation: Your request for accessing resources in this region is being validated, and you will not be able to launch additional resources in this region until the validation is complete. We will notify you by email once your request has been validated. While normally resolved within minutes, please allow up to 4 hours for this process to complete. If the issue still persists, please let us know by writing to aws-verification@amazon.com for further assistance.
Traceback (most recent call last):
  File "C:\Python310\lib\site-packages\skyplane\cli\cli_transfer.py", line 382, in cp
    dp.provision(spinner=True)
  File "C:\Python310\lib\site-packages\skyplane\api\dataplane.py", line 131, in provision
    uuids = self.provisioner.provision(
  File "C:\Python310\lib\site-packages\skyplane\api\provisioner.py", line 205, in provision
    results: List[Tuple[ProvisionerTask, compute.Server]] = do_parallel(
  File "C:\Python310\lib\site-packages\skyplane\utils\fn.py", line 57, in do_parallel
    args, result = future.result()
  File "C:\Python310\lib\concurrent\futures\_base.py", line 451, in result
    return self.__get_result()
  File "C:\Python310\lib\concurrent\futures\_base.py", line 403, in __get_result
    raise self._exception
  File "C:\Python310\lib\concurrent\futures\thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "C:\Python310\lib\site-packages\skyplane\utils\fn.py", line 43, in wrapped_fn
    return args, func(args)
  File "C:\Python310\lib\site-packages\skyplane\api\provisioner.py", line 151, in _provision_task
    server = self.aws.provision_instance(task.region, task.vm_type, use_spot_instances=task.spot, tags=task.tags)
  File "C:\Python310\lib\site-packages\skyplane\utils\imports.py", line 33, in wrapped
    return fn(*modules_imported, *args, **kwargs)
  File "C:\Python310\lib\site-packages\skyplane\compute\aws\aws_cloud_provider.py", line 224, in provision_instance
    instance = start_instance(subnets[current_subnet_id].id)
  File "C:\Python310\lib\site-packages\skyplane\compute\aws\aws_cloud_provider.py", line 186, in start_instance
    return ec2.create_instances(
  File "C:\Python310\lib\site-packages\boto3\resources\factory.py", line 580, in do_action
    response = action(self, *args, **kwargs)
  File "C:\Python310\lib\site-packages\boto3\resources\action.py", line 88, in __call__
    response = getattr(parent.meta.client, operation_name)(*args, **params)
  File "C:\Python310\lib\site-packages\botocore\client.py", line 530, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "C:\Python310\lib\site-packages\botocore\client.py", line 960, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (PendingVerification) when calling the RunInstances operation: Your request for accessing resources in this region is being validated, and you will not be able to launch additional resources in this region until the validation is complete. We will notify you by email once your request has been validated. While normally resolved within minutes, please allow up to 4 hours for this process to complete. If the issue still persists, please let us know by writing to aws-verification@amazon.com for further assistance.
18:10:35 [WARN]  Attempting to deprovision dataplane that is not provisioned, this may be from auto_deprovision.
18:10:35 [INFO]  [Provisioner.deprovision] Deprovisioning 1 VMs
18:10:37 [INFO]  [Provisioner.deprovision] Terminated AWSServer(region_tag=aws:us-west-2, instance_id=i-07643c7a06042a89f)
18:10:37 [INFO]  [Provisioner.deprovision] Deauthorizing AWS gateways with firewalls: ['35.91.249.42']
18:10:38 [WARN]  Deprovisioning dataplane
18:10:38 [WARN]  Attempting to deprovision dataplane that is not provisioned, this may be from auto_deprovision.

Environment info (please complete the following information):

Additional context Add any other context about the problem here.

sarahwooders commented 1 year ago

Thanks for reporting this @connerkdmi01 - I think this is actually just because you don't have verification to create instances in us-west-2. Did you end up getting verified by AWS? Could you try the transfer again and see if it works?