thomasborgen / storage-bucket

For easy GCP Storage Bucket interaction
MIT License
6 stars 0 forks source link

Add move file functionality between storage buckets #56

Open ChameleonTartu opened 2 years ago

ChameleonTartu commented 2 years ago

Is your feature request related to a problem? Please describe. I figured out that I need to use gsutil mv gs://bucketA/fileA gs://bucketB/fileB

Describe the solution you'd like It includes several stages:

  1. Move inside the same bucket
  2. Move between different buckets
  3. Move from the local dir to the bucket
  4. Move from the bucket to the local dir

Possible solution is to have 4 different functions or have the same function for all 4 scenarios. Both solutions seem okay the 2nd one sounds be a bit more universal while the first may make code cleaner and implementation more straightforward.

@thomasborgen

thomasborgen commented 2 years ago

@ChameleonTartu Great that you can work on this.

I like 1 and 2. I think it could be implemented by creating a copy and a copy_to_bucket function that takes a keyword argument delete_source: bool. Then we could have a move and move_to_bucket helper functions which basically just calls the copy functions with delete_source=True

But to be sure the above is a good idea we should have a look into the gcp storage bucket library. If theres anything we can call directly there to move something we don't need to have the move function call some copy function, it can juts call gcp storage buckets move directly.

For 3 and 4 I think our current upload_file and download_file could handle those with a delete_source argument.

I also think we have an issue with our function naming download_file reads the data and returns bytes, it doesn't write to a file which the function name says. upload_file takes bytes as input and not a file which one would actually expect from the function name. I'll make another issue on this.