Test data sets - Githubissues

bartnijssen commented 10 years ago

We need to develop a test data sets that can be used to ensure that changes to VIC do what they are supposed to do and do not have any unintended side consequences. I will open a series of issues for this, since we'll need different test datasets for different use cases. This issue will serve as the parent issue that can be closed when all the test data sets have been developed. Note that these test cases may need to be updated as we update VIC.

Eventual goal: Something that would allow us to run a series of automated tests after source code changes have been made to VIC (somewhat analogous to travis for compilation).

Note that:

Simplifying the number of options that are used within VIC will facilitate this testing configuration.
Small tests that test specific behaviors would be ideal (so they can help in isolating problems).

Test cases:

no change in output format or calculated values:
- Comparison with prior VIC output
change in output format but no change in calculated values:
- Comparison with prior VIC output
no change in output format but change in calculated values:
- Comparison with prior VIC output
- Comparison with observations

bartnijssen commented 9 years ago

Moved to 5.0 - will need test datasets that can be run with 4.2, but that can be done after the the 4.2 code base is released.

jhamman commented 9 years ago

@bartnijssen and @tbohn,

I've been working on pulling together a bunch of test datasets for this issue. I'm looking for suggestions on datasets for VIC simulations that can be compared to observations. I know there have been projects in the past that have compared VIC output to:

flux tower observations,
SNOTEL observations (Chiyu possibly),
soil temperature observations, and
streamflow observations.

I'd like to collect established cases of VIC setups and observations to be included in the test datasets. If either of you know where these are, please let me know (personal communication is fine).

jhamman commented 9 years ago

I should also add that we need to think of how we can test against observations for some of the lesser used options in VIC, e.g.:

Blowing snow,
Precipitation correction,
Lakes and wetlands, and
Carbon cycle.

If we can't test these options, we may want to rethink whether they belong in the released version of the model.

tbohn commented 9 years ago

I have setups for about 60 Ameriflux tower sites. I also have some setups for some EC towers in Russia, which I used to evaluate the carbon cycle features.

For lakes and wetlands, this is a harder thing to test because there are so many parameters - bathymetry is needed. However, I have setups for a) Lake Chad (comparison with lake level gauges), b) West Siberia (comparison with satellite surface water observations over large scales), c) multi-year site observations of water table depth, soil temperature, and some fluxes at a specific bog in West Siberia.

On Fri, Nov 14, 2014 at 12:16 PM, Joe Hamman notifications@github.com wrote:

I should also add that we need to think of how we can test against observations for some of the lesser used options in VIC, e.g.:

Blowing snow,

Precipitation correction,

Lakes and wetlands, and

Carbon cycle.

If we can't test these options, we may want to rethink whether they belong in the released version of the model.

— Reply to this email directly or view it on GitHub https://github.com/UW-Hydro/VIC/issues/79#issuecomment-63122881.

jhamman commented 9 years ago

I'm getting close to being done with this work for this issue. How do we feel about putting these datasets in new github repo (https://github.com/UW-Hydro/VIC_test_data)? Total size of the datasets is about 1GB and I'd like to use git-lfs to store the data on github.

I currently have four datasets formatted into netCDF4 ready for the test module:

Stehekin sample dataset
Snotel (from Chiyu Lin)
Ameriflux towers
Rosemount sample dataset

tbohn commented 9 years ago

Sounds good to me.

On Thu, Jul 23, 2015 at 11:12 AM, Joe Hamman notifications@github.com wrote:

I'm getting close to being done with this work for this issue. How do we feel about putting these datasets in new github repo ( https://github.com/UW-Hydro/VIC_test_data)? Total size of the datasets is about 1GB and I'd like to use git-lfs https://git-lfs.github.com/ to store the data on github.

I currently have four datasets formatted into netCDF4 ready for the test module:

Stehekin sample dataset

Snotel (from Chiyu Lin)

Ameriflux towers

Rosemount sample dataset

— Reply to this email directly or view it on GitHub https://github.com/UW-Hydro/VIC/issues/79#issuecomment-124191346.

bartnijssen commented 9 years ago

Same here - git-lfs may also be a good place for the SUMMA test data. I wasn't aware that they already started this (I know it was in the works)

jhamman commented 8 years ago

partially completed by #387

follow up PRs will add science and release test datasets.

jhamman commented 8 years ago

@dgergel's science testing PR has just been merged (#544). With that done, we just have to put all the test datasets together in one place. I'll work with @dgergel to pull this together. For now, let's just create a directory on our groups FTP. We'll want to put the science and release data in there so we can all start using it.

jhamman commented 8 years ago

This is done. I'll close this for now pointing to this FTP path: ftp://ftp.hydro.washington.edu/pub/gergel/VIC5_test_data/.

UW-Hydro / VIC

Test data sets #79