ai2cm / fv3config

Manipulate FV3GFS run directories
Apache License 2.0
1 stars 0 forks source link

Remove "options" for file locations in favor of automatic cache #46

Open mcgibbon opened 4 years ago

mcgibbon commented 4 years ago

We should move towards a system that no longer has "options" available for data, and instead uses remote URLs that get cached locally. Motivations for this change are listed below.

This involves a few changes:

A sample configuration dictionary (excluding the namelist) might look like the following yaml (note this is a mockup and doesn't represent a valid run directory):

experiment_name: default_experiment
forcing: gs://vcm-fv3config/data/base_forcing/v1.1
initial_conditions: gfs_example
file_sources: [
    gs://vcm-fv3config/data/base_forcing/v1.1,
    gs://vcm-fv3config/config/data_table/v1.0/data_table,
    {
        source_location: gs://vcm-fv3config/data/initial_conditions/gfs_initial_conditions/v1.0,
        source_name: file.nc,
        target_location: ,
        target_name: file.nc,
        copy_method: copy,
    },
]
orographic_data: gs://vcm-fv3config/data/orographic_data/v1.0
field_tables: gs://vcm-fv3config/config/field_tables/v1.0
diag_table: gs://vcm-fv3config/config/diag_table/v1.0/diag_table

Motivations:

mcgibbon commented 4 years ago

When we do this, file_sources should always be a list, even if it contains a single item.