Copy file from bucket to bucket is not supported

[AvihayTsayeg]

Hi, Thx for the fake gcs server.

When trying to copy file from one bucket to another I got this error.

self = <google.cloud.storage._http.Connection object at 0x7fafe923ae80>
method = 'POST'
path = '/b/data-transfer1/o/test_file.txt/copyTo/b/data-transfer2/o/ntbrpeljbr'
query_params = {}, data = None, content_type = None, headers = None
api_base_url = None, api_version = None, expect_json = True
_target_object = <Blob: data-transfer2, ntbrpeljbr, None>
...
raise exceptions.from_http_response(response)
google.api_core.exceptions.MethodNotAllowed: 405 POST https://localhost:33107/storage/v1/b/data-transfer1/o/test_file.txt/copyTo/b/data-transfer2/o/ntbrpeljbr: unknown error

I'm using python API google-cloud-storage==1.18.0 source_bucket.copy_blob(remote_file, to_bucket, target_gcs_path)

Regards, Avihay

[fsouza]

Hi @AvihayTsayeg, thanks for the detailed report!

Indeed that's not supported at the moment, but I'll see if I can find some time to add it later today or this week.

[fsouza]

(in the meantime, you should probably be able to get away with rewrite: https://github.com/googleapis/google-cloud-python/blob/3d29c750b528cbeeb7215c05bc34d355ef3b7480/storage/google/cloud/storage/blob.py#L1530)

The Go SDK actually replaced copyTo with rewriteTo, but I'm not sure why. I'll still work on adding copyTo to fake-gcs-server.

[AvihayTsayeg]

@fsouza Thank you very much

[Szasza]

@fsouza Thank you for maintaining this library, it is much appreciated.

I would like to ask if this feature will be added soon. My use-case would be to use multiple buckets for simulating ETL pipelines locally using Airflow (Cloud Composer in GCP) while not having to pay for a k8s cluster and whatnot in GCP, where this feature is a missing piece.

Can work around it if needed, just asking.

Thank you.

[fsouza]

@Szasza hi there. Thanks for pinging me, to be honest I had forgotten about this issue 🙈 The reason I didn't implement right away is because fake-gcs-server's test suite uses the Go SDK and the Go SDK doesn't support copying files with copyTo, I think, so I was lazy to implement a more sophisticated test 😁

I currently don't have any plans on implementing this feature, but I can take a look before the end of this week and check what it would take to implement/properly test it.

Do you think you could use the alternative rewrite API instead, since it's already supported: https://cloud.google.com/storage/docs/json_api/v1/objects/rewrite (you can see the relationship between copy and rewrite in copy's docs: https://cloud.google.com/storage/docs/json_api/v1/objects/copy)

[Szasza]

@fsouza Thank you for the quick response, much appreciated.

Yes, can do, no problem.