Currently we download the zip file from GCS and unzip it when using retrieve-experiment. After unzipping we are left with a folder containing many empty subfolders. I don't like that. Either add helper that recursively flattens the folder structure (https://stackoverflow.com/questions/17547273/flatten-complex-directory-structure-in-python) or directly only upload the relevant folder with the logs/results to GCP. Second option is preferred.
import shutil
import os
def move(destination, depth=None):
if not depth:
depth = []
for file_or_dir in os.listdir(os.path.join([destination] + depth, os.sep)):
if os.path.isfile(file_or_dir):
shutil.move(file_or_dir, destination)
else:
move(destination, os.path.join(depth + [file_or_dir], os.sep))
Currently we download the zip file from GCS and unzip it when using
retrieve-experiment
. After unzipping we are left with a folder containing many empty subfolders. I don't like that. Either add helper that recursively flattens the folder structure (https://stackoverflow.com/questions/17547273/flatten-complex-directory-structure-in-python) or directly only upload the relevant folder with the logs/results to GCP. Second option is preferred.