Optimise upload speed of multiple files

jenkinsci / google-storage-plugin

This plugin provides the “Google Cloud Storage Uploader” post-build step for publishing build artifacts to Google Cloud Storage.

https://plugins.jenkins.io/google-storage-plugin/

Apache License 2.0

31 stars 57 forks source link

Optimise upload speed of multiple files #51

Open danpalmer opened 5 years ago

danpalmer commented 5 years ago

We're trying to upload ~2.5k small files, and currently this takes around 8 minutes, even though we're on a gigabit connection (and GCS being quite fast).

It would be great to get some sort of bulk or parallel upload working. I'd expect this could reduce our upload time to well under a minute. I think my preference would be for batched uploads rather than parallel uploads, as those should use far fewer system resources.

nigesh-pugazhendhi commented 4 years ago

@stephenashank any update on this enhancement? @danpalmer did you find a workaround for this ?

we are trying to upload ~16GB worth of multiple files and it takes ~16mins whereas s3plugin takes in seconds! would love to hear any workaround to this.

danpalmer commented 4 years ago

I’m afraid we never found a workaround to this, and have now moved away from Jenkins entirely.

Recently I’ve been thinking about doing this again for GCS uploads, and found https://rclone.org which looks like it could be much more optimal for large sets of small files where not all have changed (our use-case, and a classic problem that rsync is great at solving). You may find it useful to investigate this, but we haven’t done any testing with it yet so I don’t know how well it works in practice.