Transparently handles large uploads (>=100MB) on GCS by using the composite upload feature. This splits uploads into 100MB parts and then merges them 32 parts at a time recursively. The smaller part files are deleted.
Part format: filename.{depth}.{partno}.part
A few other goodies are added here:
Faster local file transfers
FileInterface also accepts a file handle
Fixes broken transcoding for single files on CLI
Transfers of compressed files from GCS to local fs will get a compression suffix
Transparently handles large uploads (>=100MB) on GCS by using the composite upload feature. This splits uploads into 100MB parts and then merges them 32 parts at a time recursively. The smaller part files are deleted.
Part format:
filename.{depth}.{partno}.partA few other goodies are added here: