Google Cloud Storage: Sharing storage config across multiple threads

[grindarius]

Hello everyone. First of all thank you for this amazing library.

What I am doing with this library is that we have a file on gcs as csv files with s3 url we need to download, then save all the files onto google cloud storage.

I will be using buffer_unordered from futures crate to download a bunch of files in parallel. This is a mocked up code on how I wrote the code.

#[tokio::main]
async fn main() {
    let urls = vec!["https://blah-bucket.aws.amazon.com/where-this-that"];

    let google_client = GoogleRestApi::new().await?;
    let configuration = google_client.create_google_storage_v1_config().await?;
    let conf_arc = Arc::new(configuration);

    let streams = futures::stream::iter(urls).map(|url| {
        let conf_arc = Arc::clone(&conf_arc);
        tokio::spawn(async move { download_and_save(conf_arc).await })
    }).buffer_unordered(20);

}

I am not sure if this is related but I found that when the code has been executing for some time, the client that uses the cloned arc will get 401 unauthorized request for some reason, why is that so? Thank you for any help.

[abdolence]

Hello,

The reason it fails is because configuration is a confusing concept/term (not invented by this crate). It actually contains OAuth token which can expire. So, instead of sharing configuration you actually need to share client and create configuration for each call.

Hope this helps, and let me know if you need more clarity or it doesn't work

[grindarius]

Thank you so much for quick response, after I change my code to

#[tokio::main]
async fn main() {
    let urls = vec!["https://blah-bucket.aws.amazon.com/where-this-that"];

    let google_client = GoogleRestApi::new().await?;

    let streams = futures::stream::iter(urls).map(|url| {
        let client = client.clone();
        tokio::spawn(async move { download_and_save(client).await })
    }).buffer_unordered(20);
}

Then inside the client I do a call to get the config instead, which now helps. Thank you so much!