Allow per-file specification of config files

[oliverwm1]

(Edited 2019-11-20 to reflect current PR)

Closes #8, closes #23, closes #25.

In order to allow a per-file specification of initial conditions and forcing files (as well as whether to link or copy particular files) this PR introduces a low-level representation of the config dictionary called asset_list. This asset_list is a list of dicts. Each dict represents one file, and has information about its source name/location, target name and location relative to the root of the run-directory, and whether to copy or link.

API changes:

• no breaking changes
• user can now specify initial conditions or forcing files through a list of dicts
• user can specify a set of patch files through the new, optional, config['patch_files'] (these will overwrite any files specified in initial conditions or forcing)
• no files are symlinked. By default, the orography and forcing files are hard-linked and the initial conditions are copied.

Other notes:

The functions link_or_copy_directory, link_directory have been removed. copy_file and gsutil_is_installed have been moved to the new _asset_list.py. Some test refactoring had to be done because of this.

I put a few of the tests for asset_list in test_forcingdata.py because they rely on the existence of downloaded input data and lists of forcing/initial condition data, which already exist in test_forcingdata.py.

Added a missing required file (fv_core.res.nc) to the restart initial conditions required filenames in test_forcingdata.py

My method for generating an asset_list from a google cloud storage url is not great (some funky string manipulation on output from gsutil ls). We should probably move from using gsutil to gcsfs but I suggest doing that in a later PR.

[oliverwm1]

Allow per-file specification of run directory in fv3config

[mcgibbon]

"We should probably move from using gsutil to gcsfs but I suggest doing that in a later PR." please open an issue for this, before we forget.

[oliverwm1]

"We should probably move from using gsutil to gcsfs but I suggest doing that in a later PR." please open an issue for this, before we forget.

Issued opened. #33

[nbren12]

@mcgibbon Can we merge this? It seems like we have both approved it.