search

delta-io / delta-rs

A native Rust library for Delta Lake, with bindings into Python
https://delta-io.github.io/delta-rs/
Apache License 2.0
1.98k stars 365 forks source link

Remove s3_cli module from test_utils #1558

Open rtyler opened 11 months ago

rtyler commented 11 months ago

In rust/src/test_utils.rs we have some shell invocations of the AWS cli which aren't necessary (IMHO) because we can do all of these operations directly from Rusoto without adding the system dependency.

marijncv commented 11 months ago

Hey @rtyler, I started taking a look at this issue and I'm wondering if it's going in the right direction with my draft PR https://github.com/delta-io/delta-rs/pull/1570.

For now just creating the bucket with rusoto_s3 using their async API, but it required me to make a lot more functions async as well. Is that an issue? Or is there perhaps a better way of doing it? Curious for your feedback!

Edit: https://github.com/delta-io/delta-rs/pull/1570/commits/0f015aac73e5e42efdd202fcfa4ab43497d6f0a6 uses futures::executor::block_on() to deal with the async API of rusoto_s3 as an alternative to the approach of making more functions asyc, but it requires the tests that use it to have the #[tokio::test(flavor = "multi_thread")] macro