ai2cm / fv3config

Manipulate FV3GFS run directories
Apache License 2.0
1 stars 0 forks source link

Add ability to download files from cloud storage bucket #10

Closed oliverwm1 closed 4 years ago

oliverwm1 commented 4 years ago

If user provides a path that begins with gs:// for an option that can be specified as a path to a file or directory (specifically diag_table, data_table, forcing or initial_conditions) then have fv3config download that data and put it in the right place.

I think it's fair to assume the user will have appropriate authorization if pointing to a private storage bucket, but we should ensure fv3config fails gracefully if they do not.

nbren12 commented 4 years ago

I think this should be relatively straightforward, but currently code like this

    elif os.path.isdir(config['initial_conditions']):

is in several functions. It would be easier if we refactor it one common place. Maybe a resolve_url function like this would work:

def resolve_url(url: str):
    if url.startswith('gs://'):
        check_call(['gsutil', 'cp', '-r', url, abs_local_path])
        return resolve_url(abs_local_path)
    elif url.startswith('/'):
        # check if file present and create symlinks
    else:
        # url is a default "tag"

Then, get_initial_conditions could use it like this:

def get_initial_conditions_directory(config):
     return resolve_url(config['initial_conditions'])
oliverwm1 commented 4 years ago

Improve handling of options in fv3config