We have seen cases where due to aborts in earlier steps
the upload stage would eagerly upload lots of files that
were supposed to go into an archive to S3.
This new optional parameter allows specifying a regular
expression for file names that actually should be uploaded,
regardless of what else might be in the source directory.
For example specify --upload.file_regex='.+\.tar$' to ensure only
tars are ever sent.
We have seen cases where due to aborts in earlier steps the upload stage would eagerly upload lots of files that were supposed to go into an archive to S3. This new optional parameter allows specifying a regular expression for file names that actually should be uploaded, regardless of what else might be in the source directory.
For example specify --upload.file_regex='.+\.tar$' to ensure only tars are ever sent.