durch / rust-s3

Rust library for interfacing with S3 API compatible services
MIT License
498 stars 195 forks source link

MalformedXML on `put_object_stream` with minio #297

Closed ololduck closed 1 year ago

ololduck commented 1 year ago

Describe the bug I am encountering a MalformedXML error when trying to upload a file to a bucket with Bucket::put_object_stream.

To Reproduce The minimal example i found to present my issue is from the put_object_stream docblock.

First, I launched [minio] with the following command:

docker run --rm -p 9045:9000 -p 9001:9001 \
    -e MINIO_ROOT_USER=testuser -e MINIO_ROOT_PASSWORD=deadbeef \
    --name=minio minio/minio server /data --console-address :9001

I used the UI to create a public bucket, named rust-s3-test.

Then, I created a sample rust project with the following content in Cargo.toml:

[package]
name = "rust-s3-issue"
version = "0.1.0"
edition = "2021"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

[dependencies]
tokio = { version = "1.21.1", features = ["full"] }
rust-s3 = { version = "0.32.3", features = ["with-tokio"] }

Here's its src/main.rs:

use std::fs::File;
use std::io::Write;

use s3::bucket::Bucket;
use s3::creds::Credentials;
use s3::region::Region;

#[tokio::main]
async fn main() {
    let bucket_name = "rust-s3-test";
    let region = Region::Custom {
        region: "local-minio".to_string(),
        endpoint: "http://localhost:9045".to_string(),
    };
    let credentials =
        Credentials::new(Some("testuser"), Some("deadbeef"), None, None, None).unwrap();
    let bucket = Bucket::new(bucket_name, region, credentials).unwrap();
    let path = "path";
    let test: Vec<u8> = (0..1024).map(|_| 42).collect();
    let mut file = File::create(path).unwrap();
    file.write_all(&test).unwrap();

    let mut path = tokio::fs::File::open(path).await.unwrap();
    let status_code = bucket.put_object_stream(&mut path, "/path").await.unwrap();
    assert_eq!(status_code, 201); // whatever
}

And when trying to run it:

$ cargo run
   Compiling rust-s3-issue v0.1.0 (/tmp/rust-s3-issue)
    Finished dev [unoptimized + debuginfo] target(s) in 2.58s
     Running `target/debug/rust-s3-issue`
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Http(400, "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>MalformedXML</Code><Message>The XML you provided was not well-formed or did not validate against our published schema.</Message><BucketName>path</BucketName><Resource>/path</Resource><RequestId>17187E79EEB8626D</RequestId><HostId>21c8a2f9-df75-4c41-8a33-39e12c263005</HostId></Error>")', src/main.rs:24:74
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

Here's the beautified XML:

<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n
<Error>
    <Code>MalformedXML</Code>
    <Message>The XML you provided was not well-formed or did not validate against our published schema.</Message>
    <BucketName>path</BucketName>
    <Resource>/path</Resource>
    <RequestId>17187E79EEB8626D</RequestId>
    <HostId>21c8a2f9-df75-4c41-8a33-39e12c263005</HostId>
</Error>

Expected behavior I expected the file to be uploaded to the public bucket.

Environment

I hope you will find this useful & wish you well.

durch commented 1 year ago

@ololduck I think minio buckets still need to be configured with_path_style like so.

I've added tests for both small and big file streaming and they seem to be passing, so my guess is that its the path_style thing

ololduck commented 1 year ago

I can confirm that my tests run green using with_path_style. I'll try to present a merge request for more docs if i can find some time.

Thank you for your quick response!