lsst-uk / csd3-echo-somerville

Code to backup from CSD3 to Echo S3, curate at STFC cloud and expose to Somerville
Apache License 2.0
0 stars 0 forks source link

Give backup progress #47

Open davedavemckay opened 3 months ago

davedavemckay commented 3 months ago

As os.walk returns an Iterator, its length is unknown. Do a quick os.walk at the beginning to count files and folders and use this to provide a progress percentage.

davedavemckay commented 3 months ago
%%timeit
folder_count = 0
file_count = 0
for folder, subfolders, files in os.walk(path):
    folder_count += 1
    file_count += len(files)
print(f'Folders: {folder_count}, Files: {file_count}')

Folders: 28, Files: 28360 22.2 ms ± 30.4 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)