search

move-coop / parsons

A python library of connectors for the progressive community.
Other
255 stars 125 forks source link

fixes copy_s3 by using generic GCS function #976

Open willyraedy opened 5 months ago

willyraedy commented 5 months ago

During the migration, we generalized a function to use the Transfer Service to copy file(s) from an arbitrary bucket rather than having bespoke functions for S3, GCS etc. However, we missed one reference to the old file.

Austin this should get you started. Could stand some more manual testing and we should add a unit test here obviously. I can get to that probably post the Feb 5th release of BQ to the rest of the members but wanted to get you what I had.

shaunagm commented 5 months ago

Looks like there's a test failure here

shaunagm commented 3 months ago

@bmos thanks for your help with #1019 but it seems the tests are still failing

bmos commented 3 months ago

I will investigate in the next few days.

shaunagm commented 1 month ago

Thanks to @bmos the tests are now passing. Is this otherwise ready to merge? @austinweisgrau it seems like you have more context than me, if you can give it the thumbs up I will merge it.