mle-infrastructure / mle-toolbox

Lightweight Tool to Manage Distributed ML Experiments 🛠
https://mle-infrastructure.github.io/mle_toolbox/toolbox/
MIT License
3 stars 1 forks source link

Recursively flatten retrieved/uploaded experiment #5

Closed RobertTLange closed 3 years ago

RobertTLange commented 3 years ago

Currently we download the zip file from GCS and unzip it when using retrieve-experiment. After unzipping we are left with a folder containing many empty subfolders. I don't like that. Either add helper that recursively flattens the folder structure (https://stackoverflow.com/questions/17547273/flatten-complex-directory-structure-in-python) or directly only upload the relevant folder with the logs/results to GCP. Second option is preferred.

import shutil
import os

def move(destination, depth=None):
    if not depth:
        depth = []
    for file_or_dir in os.listdir(os.path.join([destination] + depth, os.sep)):
        if os.path.isfile(file_or_dir):
            shutil.move(file_or_dir, destination)
        else:
            move(destination, os.path.join(depth + [file_or_dir], os.sep))
RobertTLange commented 3 years ago

Addressed in PR #29.