Start of Evaluation object (#191)

[jmccreight]

• A first try. The bulk collect still passes tests but the file grouping/chunking is not tested, the datasets looked at are equal but do not have identical reprs.

• fix

• Start of Evaluation object

• try a newer pytest

• try a newer pytest without typo

• Testing gof on streamflow, generalized to other variables

• Testing GOF for SOIL_M in LDASOUT.

• Teams for forecasts, not tested with a scheduler.

• add scipy

• Refactor teams code to separate file. Currently only used by cycles. Split out cycle run tests.

• Ensemble teams uses common code

• more groping in the dark about pytest failures

• Cycle team assignment tested. Ensemble team assignment not integrated.

• Cycle and ensemble teams tested, documented, using same teams module

• touchups and critical typo

• Cycle ensembles working and tested too

• Print the teams dict for clarity, useful for checking resources.

• Update README.md

• Serial calls to a parallel collect with saving on each chunk.

• Introducing xarray ufunc... WIP

• some xr and testing, but strane behavior.

• Fix xr evaluations via ravel, ndarray -> array.

• Evaluation with xarray except on groups, tested.

• minor semantics

• Extend xr.da.eval

• pep8, i didnt ignore you. i just couldnt wait to merge

• chunked ("hierarchical") collect tested to be equal to non-chunked collection

• hierarchical->group nomenclature change

• Cleanup grouped collection

• speedup ens cycle ldasout testing

• minor

• removing silent job failures for this branch

• minor evaluation changes

• eval: take default join dims from the observation dims/index

• Adding CRPS and Brier Score using proper scoring.

• Timelagged CRPS evaluation, but not tested

• CRPS for regular ensemble forecasts (member over reference,lead times), ensemble time-lagged (both member and lead time over valid time), and deterministic time-lagged (lead time over valid time). Not fully tested.

• evaluation gof and contingency. contingency tested only for known data.

• Evaluation testing; repr madness!

• contingency stats: made threshold a column that can vary across the group, removed label

• event stats: made threshold a column that can vary across the group, removed label

• Contingency stats improvements: handle NaNs, extra stats in output.

• Cleanup of contingency and event returned objects.

• Translate the txt file to a netcdf file with some qc steps.

• typo

• Dictionaries of each gof and contingency stats ideal value, for purposes of calculating improvement/skill score stats.

• update contingency stats

• Move to xarray 0.14.1 (#199)

• Move to xarray 0.14.1:

• remove use of xr ds reprs and add netcdf data to the collection download data

• accomodate xr api changes

• debugging travis failure: travis may be running some things concurrently. path.is_symlink fixed this

• update pandas requirement

• fix to previous PR (hierarchical collect) caused slow downs as a missing indexing argument

• update requirements and setup

• fix numpy

• fix fixing numpy

• Numpy to 1.18.1 and answer change.

• pep8 cleanup

[pep8speaks]

Hello @jmccreight! Thanks for updating this PR.

• In the file wrfhydropy/tests/data/evaluation_answer_reprs.py:

Line 5:101: E501 line too long (353 > 100 characters) Line 8:101: E501 line too long (354 > 100 characters) Line 11:101: E501 line too long (2023 > 100 characters) Line 14:101: E501 line too long (7542 > 100 characters) Line 17:101: E501 line too long (385 > 100 characters) Line 20:101: E501 line too long (2198 > 100 characters) Line 23:101: E501 line too long (396 > 100 characters) Line 26:101: E501 line too long (7968 > 100 characters) Line 27:1: W293 blank line contains whitespace Line 31:101: E501 line too long (1551 > 100 characters) Line 34:101: E501 line too long (438 > 100 characters)

• In the file wrfhydropy/tests/test_evaluation.py:

Line 235:24: E225 missing whitespace around operator Line 235:37: E225 missing whitespace around operator Line 235:38: E222 multiple spaces after operator Line 235:55: E225 missing whitespace around operator Line 235:56: E222 multiple spaces after operator Line 235:75: E225 missing whitespace around operator Line 235:76: E222 multiple spaces after operator Line 236:22: E222 multiple spaces after operator Line 236:36: E222 multiple spaces after operator Line 236:53: E222 multiple spaces after operator Line 236:68: E222 multiple spaces after operator Line 245:24: E225 missing whitespace around operator Line 245:25: E222 multiple spaces after operator Line 245:40: E225 missing whitespace around operator Line 245:41: E222 multiple spaces after operator Line 245:59: E225 missing whitespace around operator Line 245:60: E222 multiple spaces after operator Line 245:80: E225 missing whitespace around operator Line 246:22: E222 multiple spaces after operator Line 246:36: E222 multiple spaces after operator Line 246:53: E222 multiple spaces after operator Line 246:68: E222 multiple spaces after operator

Comment last updated at 2020-01-29 17:06:03 UTC

[coveralls]

Pull Request Test Coverage Report for Build 528

• 321 of 382 (84.03%) changed or added relevant lines in 4 files are covered.
• No unchanged relevant lines lost coverage.
• Overall coverage decreased (-0.3%) to 88.336%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
wrfhydropy/tests/test_evaluation.py	126	129	97.67%
wrfhydropy/core/evaluation.py	193	251	76.89%
<!--	Total:	321	382	84.03%	-->

Totals
Change from base Build 515:	-0.3%
Covered Lines:	3711
Relevant Lines:	4201

---

💛 - Coveralls