Closed nick-sturrock closed 2 years ago
sarahwooders
commented
2 years ago Hi @nick-sturrock - do you have a GCP project yourself, even if its not the bucket owner? Skyplane needs an associated GCP project to create service account keys to have permissions to access buckets and also create VM instances to execute the transfer.
nick-sturrock
commented
2 years ago Yes, we have multiple projects and have configured Skyplane to be connected via one of them, however it then doesn't have permission to access content in the Google-created takeout bucket, and we are unable to modify the permissions on the takeout bucket to allow access. We are also unable to transfer the content into a project-owned bucket using Google Transfer.
sarahwooders
commented
2 years ago Hi Nick - Could you please copy past the permission error you're seeing? Also are you able to access the contents of the bucket from any account? If you can't programmatically access the bucket from any account Skyplane wont be able to access the data either.
I'm trying to find a way to sync to S3 the results of a google takeout export, which has generated 250+ Gb of data on a GCP bucket that isn't associated with any project, and I don't apparently have any permissions within Cloud Console to transfer it to a project-based bucket. I have configured Skyplane accordingly but it's unable to read the contents of the bucket, since it appears only to be able to read project-connected buckets (?). Gsutil does allow this export bucket to be read from the command line, however it throws loads of connection and broken pipe errors and is not a viable way to move the contents of the bucket from what I've experienced so far. Should Skyplane be able to handle this?