Using 'scp' to copy a lot of data in a file-by-file manner is very
inefficient. This patch changes the way path extraction works:
it creates a tar.gz archive containing all desired files on remote side
uses ssh to redirect the output of tar|gzip pipe directly to local side (there are no files created on the remote side)
extracts .tar.gz file to a temporary directory on the local side
moves all the extracted files from temporary directory to target destination, at the same time renaming them to desired names
This algorithm offers drastic speed-ups when dealing with collection of a large dataset i.e. for oVirt System Tests, it reduces the time needed to collect artifacts from ~20 secs to ~2 secs. This in turn, trims down the running time of each OST suite by a couple of minutes.
Using 'scp' to copy a lot of data in a file-by-file manner is very inefficient. This patch changes the way path extraction works:
This algorithm offers drastic speed-ups when dealing with collection of a large dataset i.e. for oVirt System Tests, it reduces the time needed to collect artifacts from ~20 secs to ~2 secs. This in turn, trims down the running time of each OST suite by a couple of minutes.