Create CI configurations for repositories

[agstephens]

Integrate the two prototype repositories with:

• CEDA gitlab CI
• Travis-CI (in the cloud)

Use mini example data repository set up separately.

[ellesmith88]

@agstephens I'm not having much luck working out how to use this data in the tests when running the CI. Do you know anything about how this works?

[ellesmith88]

Also travis needs to be authorised to work, I've sent a request over.

[agstephens]

@ellesmith88 , I have just granted access to the request.

[ellesmith88]

Thank @agstephens, I've worked out a way to get the test data in as well.

[agstephens]

@agstephens I'm not having much luck working out how to use this data in the tests when running the CI. Do you know anything about how this works?

@ellesmith88: Here is my take on how things could work:

1. The mini-esgf-data repository could be included as a git "sub-module" in the other repositories. See: https://gist.github.com/gitaarik/8735255 for info about this approach.
2. This should allow the clone of the main repository (that you are testing) to automatically pull down the sub-module (mini-esgf-data). You may need to force this with extra arguments and/or commands. See: https://www.atlassian.com/git/tutorials/git-submodule - section called "Cloning git submodules"
3. Then the data should be ready for use in tests.

And/or you could hook up the pulling of the submodule as setup stage for all the relevant unit tests.

Does that make sense? Please come back to me if I've missed the point of your question.

[agstephens]

@ellesmith88: also, it might make sense to define the test data sets as different "projects" in the configuration of the repositories. E.g.:

At present, in the lib.options module we define:

project_base_dirs = {
    'cmip5': '/badc/cmip5/data',
    'cmip6': '/badc/cmip6/data',
    'cordex': '/badc/cordex/data'
}

known_projects = project_base_dirs.keys()

facet_rules = {
    'cmip5': 'activity product institute model experiment frequency realm mip_table ensemble_member version variable'.split(),
    'cmip6': 'mip_era activity_id institution_id source_id experiment_id member_id table_id variable_id grid_label version'.split(),
    'cordex': 'NOT DEFINED YET'
}

As part of the setup for the unit tests you could do something like:

from lib import options

...
CMIP5_TEST = 'mini_esgf_cmip5'
options.project_base_dirs[CMIP5_TEST] = './test_data/badc/cmip5/data'
options.known_projects.append(CMIP5_TEST)
options.facet_rules[CMIP5_TEST] = options.facet_rules['cmip5'][:]
...

[agstephens]

Thank @agstephens, I've worked out a way to get the test data in as well.

Great stuff @ellesmith88

[agstephens]

All done.