search

skyplane-project / skyplane

🔥 Blazing fast bulk data transfers between any cloud 🔥
https://skyplane.org
Apache License 2.0
1k stars 58 forks source link

[bug] Inter cloud transfer AWS to GCP and vice versa not working #939

Open JimAdediran opened 4 weeks ago

JimAdediran commented 4 weeks ago

Describe the bug A clear and concise description of what the bug is. When I try an inter cloud transfer AWS to GCP or vice versa the process just stops abruptly Screenshot: skyplane sync s3://gorilla-weights-eadvd/ gs://omics-test/

/ | | / /\ \ / / \ | / _ \ | \ | || | \ `--.| |/ / \ V /| |/ / | / /\ | | || |
`--. \ \ \ / | /| | | || . ` || | /__/ / |\ \ | | | | | || | | || |\ || |_ __/_| _/ _/ _| _/_| |/_| \/____/

Azure resource group and UMI name not configured correctly. Please reinit Azure with skyplane init --reinit-azure. Logging to: /tmp/skyplane/transfer_logs/20240820_133054-3d91bcd3/client.log Using Skyplane version 0.3.2 Will transfer objects from aws:ap-south-1 to gcp:us-central1-a VMs to provision: 1x aws:ap-south-1, 1x gcp:us-central1-a Estimated egress cost: $0.11/GB s3://gorilla-weights-eadvd/reheader_tool_notes.docx => gs://omics-test/reheader_tool_notes.docx (679.21KB) Transfer starting

To Reproduce Steps to reproduce the behavior (please include the full Skyplane command you ran):

  1. Run sync command to transfer files from AWS to GCP bucket
  2. Wait for it to process the transfer
  3. Confirm transfer
  4. After transfer started message process halts abruptly

Expected behavior I expect the file from the aws bucket to be transfered to GCP after the Transfer started message instead of it halting abruptly

Screenshots If applicable, add screenshots to help explain your problem.

Transfer client log In the log output from Skyplane, please upload the debug log from the CLI. You can find the path to the file in the log output: 13:30:54 [DEBUG] Using pipeline: <skyplane.api.pipeline.Pipeline object at 0x14ccfdbd0> 13:30:56 [DEBUG] [SkyplaneClient] Queued sync job SyncJob() 13:31:10 [DEBUG] Querying objects in omics-test 13:31:13 [DEBUG] Querying objects in s3://gorilla-weights-eadvd 13:31:24 [DEBUG] Querying objects in omics-test 13:31:27 [DEBUG] Querying objects in s3://gorilla-weights-eadvd

... Storing debug information for transfer in /tmp/skyplane/transfer_logs/... ...



**Environment info (please complete the following information):**

MacOS Sonoma 14.5
Skyplane 0.3.2
Python 3.11.9

**Additional context**
Add any other context about the problem here.

I've also tried inter cloud transfer on a windows computer and observed the same issue.  I have also confirmed that skyplane is working correctly when I run commands for AWS to AWS bucket transfers and GCP to GCP bucket transfers