Closed oliverwm1 closed 4 years ago
I think this should be relatively straightforward, but currently code like this
elif os.path.isdir(config['initial_conditions']):
is in several functions. It would be easier if we refactor it one common place. Maybe a resolve_url
function like this would work:
def resolve_url(url: str):
if url.startswith('gs://'):
check_call(['gsutil', 'cp', '-r', url, abs_local_path])
return resolve_url(abs_local_path)
elif url.startswith('/'):
# check if file present and create symlinks
else:
# url is a default "tag"
Then, get_initial_conditions
could use it like this:
def get_initial_conditions_directory(config):
return resolve_url(config['initial_conditions'])
If user provides a path that begins with gs:// for an option that can be specified as a path to a file or directory (specifically diag_table, data_table, forcing or initial_conditions) then have fv3config download that data and put it in the right place.
I think it's fair to assume the user will have appropriate authorization if pointing to a private storage bucket, but we should ensure fv3config fails gracefully if they do not.