search

HydrologicEngineeringCenter / FIRO_TSEnsembles

Time series of ensembles in SqLite
Other
2 stars 2 forks source link

Scripting Issue: Subset ensemble to keep n largest n-day volumes. #212

Closed danhamill closed 1 year ago

danhamill commented 1 year ago

I am working on a script to subset an ensemble to only contain the top 20 largest n-day volumes. I am not able to reproduce the original n-day volume calculations. 

def indexToTopNMembers(nDayVolumes, nMembers):
    _nDayVolumes = list(nDayVolumes)
    #https://stackoverflow.com/questions/7851077/how-to-return-index-of-a-sorted-list
    index = sorted(range(len(_nDayVolumes)), key=lambda k: _nDayVolumes[k],  reverse=True)
    return index[:nMembers]

# Calculate n-day volumes
e = eTs.getEnsemble(issueDate)
nDayComputable = NDayMultiComputable(CumulativeComputable(), 3)
nDayVolumes = e.iterateForTracesAcrossTime(nDayComputable)

# find indicies to the top n nDay volumes
idxToSave = indexToTopNMembers(nDayVolumes, 20)

# create subset array containing the top n memvers
values = e.getValues()
newValues = []
for idx in idxToSave:
    chosenMember =list(values[idx])
    newValues.append(chosenMember)

# convert to jarray with correct java dtype
ensembleValues = array(newValues, java.lang.Class.forName("[F"))

# Create new Ensemble with subset of the original data
newE = Ensemble(issueDate, ensembleValues, e.getStartDateTime(), e.getInterval(), "cfs")
_values = newE.getValues()

# Check to make sure original and subset data are unchanged (no failed assertions)
for i, idx in enumerate(idxToSave):
    assert list(values[idx]) == list(_values[i]), "input data was changed for input index %s and output index %s" %(idx, i)

# Back calculate volumes identified from original source data
testVols = newE.iterateForTracesAcrossTime(nDayComputable)
print "test  volumes: %s" %(list(testVols))
print "input volumes: %s" %(sorted(list(nDayVolumes), reverse=True)[:20])

Which returns:

test  volumes: [131198.734375, 124981.015625, 124036.5234375, 122041.96875, 105067.03125, 101823.6015625, 101270.390625, 94239.75, 94062.9921875, 93955.4453125, 93163.2734375, 93062.046875, 92806.9609375, 92369.5390625, 90739.515625, 88281.203125, 87921.359375, 87638.625, 86804.5859375, 83009.53125]
input volumes: [1587504.75, 1512270.375, 1500841.75, 1476708.125, 1271311.375, 1232065.75, 1225371.5, 1140300.75, 1138162.125, 1136861.0, 1127275.75, 1126050.75, 1122964.125, 1117671.5, 1097947.75, 1068202.75, 1063848.375, 1060427.5, 1050335.75, 1004415.375]
danhamill commented 1 year ago

I can also check for equivalence between ensembleValues and newE.getValues(), which returns True

java.util.Arrays.equals(newE.getValues(), ensembleValues)

If I change the # create subset array containing the top n memvers bit of code to print our more info like (Column 1):

for idx in idxToSave:
    chosenMember =list(values[idx])
    newValues.append(chosenMember)
    print "================================="
    print "Chosen member sum : %s" %(sum(chosenMember[:73]))
    print "Actual nday volume: %s" %(nDayVolumes[idx])

And recalculate the volumes from ensembleValues, I am seeing consistent results (Column 2)

for idx in range(len(ensembleValues)):
    values = list(ensembleValues[idx])
    print "================================="
    print "Chosen member sum : %s\n\n" %(sum(values[:73]))

And recalculate the volumes from newE, I am also seeing consistent results (Column 3)

newE = Ensemble(issueDate, ensembleValues, e.getStartDateTime(), e.getInterval(), "cfs")
_values = newE.getValues()

for idx in range(len(_values)):
    values = list(_values[idx])
    print "================================="
    print "Chosen member sum : %s\n \n" %(sum(values[:73]))
Calcs from original data Calcs from data from ensembleValues Calcs from newE
================================= ================================= =================================
Chosen member sum : 1587504.92773 Chosen member sum : 1587504.92773 Chosen member sum : 1587504.92773
Actual nday volume: 1587504.75
================================= ================================= =================================
Chosen member sum : 1512270.41797 Chosen member sum : 1512270.41797 Chosen member sum : 1512270.41797
Actual nday volume: 1512270.375
================================= ================================= =================================
Chosen member sum : 1500841.84082 Chosen member sum : 1500841.84082 Chosen member sum : 1500841.84082
Actual nday volume: 1500841.75
================================= ================================= =================================
Chosen member sum : 1476708.10254 Chosen member sum : 1476708.10254 Chosen member sum : 1476708.10254
Actual nday volume: 1476708.125
================================= ================================= =================================
Chosen member sum : 1271311.10742 Chosen member sum : 1271311.10742 Chosen member sum : 1271311.10742
Actual nday volume: 1271311.375
================================= ================================= =================================
Chosen member sum : 1232065.69727 Chosen member sum : 1232065.69727 Chosen member sum : 1232065.69727
Actual nday volume: 1232065.75
================================= ================================= =================================
Chosen member sum : 1225371.62695 Chosen member sum : 1225371.62695 Chosen member sum : 1225371.62695
Actual nday volume: 1225371.5
================================= ================================= =================================
Chosen member sum : 1140300.89648 Chosen member sum : 1140300.89648 Chosen member sum : 1140300.89648
Actual nday volume: 1140300.75
================================= ================================= =================================
Chosen member sum : 1138162.19238 Chosen member sum : 1138162.19238 Chosen member sum : 1138162.19238
Actual nday volume: 1138162.125
================================= ================================= =================================
Chosen member sum : 1136860.90527 Chosen member sum : 1136860.90527 Chosen member sum : 1136860.90527
Actual nday volume: 1136861.0
================================= ================================= =================================
Chosen member sum : 1127275.62793 Chosen member sum : 1127275.62793 Chosen member sum : 1127275.62793
Actual nday volume: 1127275.75
================================= ================================= =================================
Chosen member sum : 1126050.76562 Chosen member sum : 1126050.76562 Chosen member sum : 1126050.76562
Actual nday volume: 1126050.75
================================= ================================= =================================
Chosen member sum : 1122964.27539 Chosen member sum : 1122964.27539 Chosen member sum : 1122964.27539
Actual nday volume: 1122964.125
================================= ================================= =================================
Chosen member sum : 1117671.35059 Chosen member sum : 1117671.35059 Chosen member sum : 1117671.35059
Actual nday volume: 1117671.5
================================= ================================= =================================
Chosen member sum : 1097947.88379 Chosen member sum : 1097947.88379 Chosen member sum : 1097947.88379
Actual nday volume: 1097947.75
================================= ================================= =================================
Chosen member sum : 1068202.79004 Chosen member sum : 1068202.79004 Chosen member sum : 1068202.79004
Actual nday volume: 1068202.75
================================= ================================= =================================
Chosen member sum : 1063848.49414 Chosen member sum : 1063848.49414 Chosen member sum : 1063848.49414
Actual nday volume: 1063848.375
================================= ================================= =================================
Chosen member sum : 1060427.33496 Chosen member sum : 1060427.33496 Chosen member sum : 1060427.33496
Actual nday volume: 1060427.5
================================= ================================= =================================
Chosen member sum : 1050335.58008 Chosen member sum : 1050335.58008 Chosen member sum : 1050335.58008
Actual nday volume: 1050335.75
================================= ================================= =================================
Chosen member sum : 1004415.39941 Chosen member sum : 1004415.39941 Chosen member sum : 1004415.39941
Actual nday volume: 1004415.375
danhamill commented 1 year ago

This was due to a stale master branch. There were strange unit conversions happening.