search

ucd-cws / calvin

reservoir network optimization
MIT License
21 stars 13 forks source link

Flow differences between S09I05 (CALVIN) and pyvin #34

Closed msdogan closed 7 years ago

msdogan commented 7 years ago

Now that pyvin is running without debug mode, we can compare flow results. Tables below show links with monthly average (TAF/m) flow differences (S09I05-pyvin) greater than 1 TAF/m and less than -1 TAF/m. Here is the cumulative distribution of errors: Link_Flow_Comparison.pdf

Avg. flow difference > 1 TAF/m (Sorted large to small)

Link    S09I05_avg  pyvin_avg
D523-D525   781.6787633 591.67432
D522-D523   819.9958928 630.3933275
D515-D522   528.008249  393.2333143
D513-D515   486.5349756 352.6453817
D616-C42    256.0514138 145.6792348
C42-D521    268.3969079 159.2392462
C309-D59    569.3837406 465.9362483
D525-D550   584.5392239 481.0940955
D550-C309   579.0554182 475.6215628
D676-D616   258.6287234 163.474426
D612-D676   193.596771  99.623167
D689-D612   147.0590394 53.08543553
D525-D528   197.1395393 110.5802245
D528-D509   196.8778724 110.3185578
D712-D722   211.4920777 127.3056605
D699-D683   113.8574162 31.53270243
D683-D689   165.615928  85.08676601
D511-D513   519.2325421 440.7415609
D695-D697   113.2036319 36.68345202
D697-D699   121.8469573 45.32677738
D710-D712   196.2670988 126.6142803
D694-D695   87.69708414 22.0825765
D619-D691   86.11558013 20.65716946
D691-D694   88.05623051 22.59781987
D731-D732   64.42653107 1.936525627
D732-D619   65.02854844 2.538543284
D608-D731   67.77790935 8.511235921
D722-D723   108.0343814 48.84486741
D723-D608   108.0343814 48.84486741
D521-D522   291.9876479 237.1600132
D801-D803   327.6546412 274.0828638
D800-D801   327.7888817 274.2201671
D803-D804   327.5845773 274.0779858
D804-D814   326.9431606 273.4365692
D59-PMP_BANKS   325.5374547 273.5503749
PMP_BANKS-D800  325.5374547 273.5503749
D59-PMP_TRACY   243.8462845 192.3858734
PMP_TRACY-D701  243.8462845 192.3858734
D703-D710   240.0539089 190.1751316
D700-D703   241.5948572 191.7160812
D701-D700   241.5948572 191.7160812
C30-HSU303C30   103.4576956 78.46079304
D722-C30    103.4576956 78.46079304
D37-C32 507.4699125 484.4740644
C31-D37 482.0339231 460.053713
C83-C31 151.1321929 129.2539676
C80-HSU202C80   66.60247714 44.86999146
C32-C80 66.57156612 44.83920776
HSU303C30-A303  74.37338561 52.98824876
HSU202C80-A202  58.13530012 39.01028786
D814-PMP_GIANEL 67.82482563 48.75842331
PMP_GIANEL-SR_SNL   67.82482563 48.75842331
D816-D742   326.4629903 308.0852808
PWP_GIANEL-D816 38.78282508 20.53766385
SR_SNL-PWP_GIANEL   38.78282508 20.53766385
D814-D816   302.9051435 288.2389971
D816-PWP_ONEILL 15.22497819 0.691380196
PWP_ONEILL-D712 15.22497819 0.691380196
D664-HSU302D664 19.52586112 5.451949758
C79-WTP301  13.64860477 0
D647-D649   35.80701506 23.21745529
D646-D647   38.30255871 25.71299902
D645-D646   27.54941947 14.9598598
D743-D743A  318.846535  306.4885273
D742-D743   319.9544192 307.596413
D745-D747   314.6986092 302.5986296
D747-D749   314.6986092 302.5986296
D743A-PMP_DAMIGO    318.5885043 306.4885273
PMP_DAMIGO-D744A    318.5885043 306.4885273
C92-D745    317.5306449 305.4306699
D744A-D744  317.5306449 305.4306699
D744-C92    317.5306449 305.4306699
D662-D663   56.23863223 44.22299142
D663-D664   73.11645779 61.10081729
D749-D750   257.1865286 245.4923298
D751-C75    253.3531936 241.6589964
D750-D751   254.1948587 242.5006631
HSU302D664-A302 16.01120609 4.470598762
D649-D695   24.15888663 13.25320583
SR_WHI-D73  11.21628471 1.136087984
D606-HSU306D606 15.26307935 6.415667275
D16-HSU302D16   22.74824393 14.12181611
HSU306D606-A306 14.34729473 6.030727223
HSU302D16-A302  18.65355975 11.57988911
C75-D845    247.4650748 240.6704342
D845-D850   229.7598306 222.9651903
D605-D606   28.23995003 21.53709347
SR_MIL-D605 15.33395013 8.631093809
C85-HSU303C85   6.766601838 0.48886783
SR_GNT-SR_ML    6.275451584 0.214308148
D742-C85    6.508571298 0.48886783
HSU303C85-A303  6.292939901 0.454647085
D850-D851   149.3076568 143.9506185
D851-D853   198.9501456 193.5931083
C95-D752    197.9584786 192.6014416
D853-C95    197.9584786 192.6014416
C25-WTP201  5.939398709 0.759735772
C75-N9  5.888119312 0.988562258
N9-HSU404AN9    4.81924606  0.008439114
D731-HSU306D731 7.07485776  2.373963042
SR_TUL-D16  80.6831387  76.06183871
D670-SR_TUL 78.16223905 73.61167338
HSU404AN9-A404A 4.481898826 0.007848376
HSU306D731-A306 6.225874813 2.089087466
D541-SURP_DELTA 766.33473   762.6269795
SURP_DELTA-SINK 766.33473   762.6269795
D509-D541   1182.482774 1178.775026
D752-C74    42.7752344  39.09655086
WTP204-U204 17.11132138 13.57653363
C53-HSU401C53   32.58497045 29.1254492
D860-D861   195.7541622 192.3439125
D861-PMP_WHEELR 195.4541609 192.0439123
D862A-D863  195.4541609 192.0439123
D863-PMPCHRIS   195.4541609 192.0439123
PMP_WHEELR-D862A    195.4541609 192.0439123
PMPCHRIS-D862B  195.4541609 192.0439123
D855-D859   173.4938085 170.0835602
D859-PMP_BUENA  190.0334425 186.6231944
PMP_BUENA-D860  190.0334425 186.6231944
D888-C161   55.3209544  52.02005447
SR_CAS-D888 63.29152746 60.00899147
D865-PMP_OSO    66.87248151 63.64238363
D884-D885   66.87248151 63.64238363
PMP_OSO-D884    66.87248151 63.64238363
D885-PWP_WARNE  66.71414748 63.4840503
PWP_WARNE-SR_PYM    66.71414748 63.4840503
C103-D865   171.6621655 168.4321003
D862B-PMP_EDMONS    171.6621655 168.4321003
PMP_EDMONS-C103 171.6621655 168.4321003
SR_PYM-D886 66.34205096 63.12430315
D886-PWP_CASTAI 66.08371728 62.8659699
PWP_CASTAI-D887 66.08371728 62.8659699
D887-SR_CAS 65.8837171  62.6659699
C98-D855    18.68333103 15.49752406
D752-D855   157.7585036 154.5860362
C8-WTP204   11.1863834  8.20115533
D692-D619   21.0870316  18.11862618
C47-D692    4.987438003 2.019032683
D855-C98    2.948026557 0
HSU401C53-A401  27.6972247  24.75663252
N2-N1   11.45064988 8.525521551
N1-N2   2.909262282 0.000626708
HGP08-U210  2.758320234 0.019366643
C153-C156   14.60094903 11.95462802
C156-WTP511 14.60094903 11.95462802
D31-D61 648.2002079 645.9033779
D61-C301    574.1111111 571.8142818
C48-C47 3.856772497 1.571581422
D649-HSU305D649 8.044713058 5.789660035
C313-C313B  29.1850803  26.96159061
C15-C313    24.81922715 22.59573775
C313B-C314  7.110882382 4.92907396
D507-HSU206D507 22.63862024 20.47871082
HSU206D507-A206 21.05391689 19.04520121
D42-D43 488.1520738 486.1907365
C12-D76B    24.51768342 22.58215773
C67-HSU204C67   24.793893   22.87836936
HSU303C30-HSR302    21.84227166 19.98028933
HSU305D649-A305 6.596664787 4.747521187
C69-C13 61.50130946 59.77415382
HSU204C67-A204  21.81862632 20.13296493
D77-D75 678.8182652 677.1783105
D75-C1  749.4542419 747.8142873
D66A-D66    725.4492731 723.8245471
N13-D66A    725.7947401 724.1700142
D76B-N13    721.9248721 720.3001471
D66-N14 752.7683776 751.1436529
N14-D30 786.7520156 785.1272912
D30-D31 654.1340779 652.5093556
D701-D800   2.251427187 0.669792188
C7-C8   1147.319722 1145.79009
C67-C7  1135.980717 1134.451088
C1-C4   748.2179941 746.7215023
C74-D752    2.575259538 1.081145449
SR_SKN-C153 60.70879245 59.22909242
D850-N12    80.45217429 79.0145718
C315-WTP304 6.464371612 5.040811262
D714-HARSC  16.24205625 14.82074005
C4-C69  750.8352162 749.4188606
C41-WTP210  5.325874823 3.971982198
C38-D98 82.53241114 81.22011558
D43-C67 1168.118259 1166.85168
C32-D42 478.1280184 476.8645313
D609-C48    1.51324129  0.259021716
C53-HSU403C53   45.35918535 44.10855896
N12-HSU408BN12  53.51405149 52.27116873
C74-HSU407C63   20.67207045 19.43108626
D672-D675   55.19289209 54.01219891
D675-D676   65.03195204 63.85125903
C104-C105   1.971935629 0.810711678
HSU408BN12-A408B    47.70748556 46.55160549
HSU407C63-A407  19.22502537 18.07091037
SR_OL-C120  15.79930722 14.69794415
C120-SR_OL  19.13180715 18.03044415
HSU403C53-A403  39.91608341 38.81553167
D98-HSU207D98   15.62842251 14.5422657
D74-C5  658.19441   657.109829
C87-D77 715.2067524 714.1225499
C5-C87  711.4139543 710.3297522
C37-C38 30.11519976 29.06270367
C53-WTP401  13.50921371 12.47653872
D624-C48    2.343531208 1.312559705
C308-D37    18.5807739  17.56513679
C33-C308    18.25191204 17.23627501

Avg. flow difference < -1 TAF/m (sorted small to large)

D624-HSU306D624 4.642579857 5.659813904
C49-WTP401  3.742574541 4.762225158
C8-C104 1349.292384 1350.328908
C37-HSU207C37   0.001771737 1.054267827
C65-HSU408BC65  6.146095074 7.203447451
D521-HSU206D521 9.106826652 10.17541229
HSU403C76-A403  0.005143002 1.105755993
D64-WTP204  3.362075038 4.521483896
C153-C154   46.10784397 47.2744644
N6-SR_SKN   94.48620779 95.65609002
C138-N6 64.46836709 65.63978749
C76-HSU403C76   0.00553011  1.18898493
HSU204C33-A204  10.15460734 11.36513442
C41-C42 8.573993216 9.788511462
HSU407C64-A407  22.41451607 23.72522962
C33-HSU204C33   11.53932632 12.91492555
C64-HSU407C64   24.10163052 25.51099939
C315-HARSC  3.757943747 5.179259948
HSU409AC68-A409A    9.913518882 11.41886889
C689-HSU409AC68 10.65969747 12.27835355
HSU305D662-A305 46.73235159 48.45542764
D664-D683   51.75851121 53.55406358
HSU303D731-HSR302   1.459898064 3.280752185
C64-C689    11.15244968 12.99103399
C12-C13 0.313141293 2.283852031
D662-HSU305D662 56.99067341 59.09198545
C67-C314    7.343647698 9.522223274
C104-D44    1347.320447 1349.518196
D44-D503    1371.395445 1373.593195
D503-D511   1371.395445 1373.593195
C15-C301    16.70667455 19.20692863
HSU401D606-A401 0.844646034 3.375156258
N8-WTP511   32.75375945 35.4000807
SR_SKN-N8   33.29111328 35.9374346
WTP210-U210 7.053346432 9.792300024
HXCMWD-ERES_CMWD    121.6311526 124.5512794
C161-WTP504 154.7601283 157.6802551
WTP504-U504 154.7601283 157.6802551
HXI504-HXCMWD   105.4017987 108.3219258
U504-HXI504 114.8451316 117.7652591
ERES_CMWD-SINK  4.444016902 7.364144817
D606-HSU401D606 1.030056143 4.116044253
C62-C64 35.45376283 38.70171634
HGP08-U204  10.01393551 13.40648498
C688-C62    36.43874119 40.0783141
C76-C688    121.4326631 125.2110164
D653B-D672  66.34002812 70.34515572
D16-D653A   57.93489497 61.9400226
D653A-D653B 57.93489497 61.9400226
C43-WTP210  1.727471609 5.820317826
HSU302D672-A302 9.140651507 13.39302447
HSU404AC52-A404A    14.81217809 19.42638299
D670-C43    7.39610892  12.24476806
HSU303D731-A303 4.975705492 9.890444649
C49-C76 123.1078828 128.0696912
C25-C31 332.5633859 337.6514731
D672-HSU302D672 11.14713595 16.33295681
C31-WTP201  1.66165803  6.851727642
D689-C10    1.542753609 7.013434163
D699-C10    4.303111494 9.975949731
C53-C54 26.28026123 32.04801696
C54-C52 16.6797006  22.44745637
C52-HSU404AC52  18.51522255 24.28297837
SR_GNT-SR_CRW   3.561948079 9.874774944
HSU302D689-A302 13.95159058 20.4900748
C116-C120   39.47307034 46.05707489
PWP_OWENS1-C116 15.13142157 21.71542621
SR_CRW-PWP_OWENS1   15.13142157 21.71542621
C120-SR_LA  30.65081318 37.23481789
SR_MIL-C49  127.6562018 134.3534422
C122-C161   30.44090823 37.23444592
PWP_OWENS2-C122 30.44090823 37.23444592
SR_LA-PWP_OWENS2    30.44090823 37.23444592
HSU302D662-A302 5.188653712 13.01207319
D731-HSU303D731 10.40432248 18.32854944
D689-HSU302D689 17.01413488 24.98789632
D5-D73  526.2667643 535.5880108
D662-HSU302D662 6.327626503 15.86838208
SR_WHI-D5   66.56458406 76.46066163
HSU306D645-A306 37.79101525 47.98350168
D645-HSU306D645 46.61482888 59.38876368
SR_SFAGG-WTP301 4.618895304 18.2675
C79-SR_SFAGG    12.9737563  27.13916667
D616-C10    4.192186262 19.41006823
HSU202C83-A202  2.537937501 21.74134421
D710-PMP_ONEILL 43.78680885 63.5608513
PMP_ONEILL-D814 43.78680885 63.5608513
C83-HSU202C83   2.884019835 24.7060731
HSU303C10-A303  8.532343801 30.93953517
C10-HSU303C10   10.03805131 36.39945212
D513-D521   32.6975663  88.09617922
D511-D55    852.1629026 932.8516343
D55-D507    995.1401354 1075.831792
D507-D509   972.5015154 1055.353081

S09I05 and pyvin link flows are attached S09I05_pyvin_flow.zip

jdherman commented 7 years ago

Thanks a lot Mustafa. Really excited that we're getting results now!

Some of these are very different -- but they seem to be in places that are similar to the differences we saw in Hobbes vs. PRM comparison. Could it be the case that the differences are in the network data, not the solver?

Another point is that our cost curve extensions are different than PRM, so we shouldn't expect exactly the same answer.

How do the objective function values compare?

jdherman commented 7 years ago

Just for comparison here is the portfolio plot from Max's script: image

(y-axis in units of average TAF/month)

compared to these figures from Dogan (2015)

image

Overall it matches up pretty well! (our new plot doesn't show shortage, and doesn't include the warm-dry scenario).

msdogan commented 7 years ago

Yes, looks like flow differences are occurring on links that we saw some differences while comparing CALVIN and HOBBES, but this time differences are even bigger. It is possible that the issue is input data. We need to go back check c, a, lb, ub parameters for these links to see if there is any problem.

For the objective function, values are close, at least in the same order of magnitude: S09I05 z = -40664453805.7 pyvin z = -40676283943.10019 difference = 11830137 So, since the objective is to minimize cost, CPLEX seems to do better job if we represent everything correctly.

msdogan commented 7 years ago

Thanks for the plots. I am guessing values on Max's plot are TAF/y instead of TAF/m. And looks like it is about 100 to 200 TAF/y less than what I calculated. The difference is scarcity?

jdherman commented 7 years ago

Hey they both get $40.6 Billion for the optimal cost -- that's pretty close! I feel good about that.

For the flow volumes I agree we can double-check the c,a,l,u parameters.

I'm pretty sure the bar plot is TAF/m. If you multiply by 12 and divide by 1000 (so basically divide by 100) you get values that are close to yours. And yes scarcity is not included, you're right.

msdogan commented 7 years ago

Sorry, yeah, ag and urban deliveries are looking close.

msdogan commented 7 years ago

Here is the latest comparison of CALVIN and PyVIN (run 12272016) after Justin's updates on MIF representation. PyVIN was run in debug mode but total debug flow amount is small #29. I compared some examples from network matrix (input) to CALVIN, and there is no issue (#35). Everything (bound & cost) seems to be correctly represented in network matrix, however there might be some local issues.

Classical cdf :) link_flow_comparison

links that have monthly average flow difference > 10 TAF/m

Network Link    S09I05  pyvin   S09I05-pyvin (difference)
D513-D521   32.70   55.85   -23.15
C83-HSU202C83   2.88    25.07   -22.19
D521-D522   291.99  313.26  -21.27
HSU202C83-A202  2.54    22.06   -19.53
D800-D801   327.79  338.69  -10.90
D801-D803   327.65  338.55  -10.90
D804-D814   326.94  337.83  -10.89
D803-D804   327.58  338.47  -10.89

C83-C31     151.13  128.93  22.20
C31-D37     482.03  459.82  22.22
HSU202C80-A202  58.14   35.62   22.51
D37-C32     507.47  483.43  24.04
C80-HSU202C80   66.60   41.02   25.58
C32-C80     66.57   40.99   25.58
D515-D522   528.01  501.13  26.88
D513-D515   486.53  458.96  27.57
jdherman commented 7 years ago

Oh much better!! Last time the biggest differences were 100-200 TAF/m on average, now they're 10x smaller. We'll run it by the group but I think this is close to done: the objective function is the same, debug flows are very small, so maybe these differences are just multiple optima.

msdogan commented 7 years ago

Yes, differences are much smaller in this run. We can also compare reservoir storage/release, ag and urban deliveries, where we apply economics, and then decide how to proceed.

jdherman commented 7 years ago

Ok that's great -- Let's do those comparisons after the break. @mfefer this would be a good application of your postprocessing scripts :)

msdogan commented 7 years ago

Objective value comparison of latest 82-year run (https://github.com/msdogan/pyvin/tree/master/examples/statewide_82-year_results)

model   Objective Value ($)
S09I05  -40,664,453,806
pyvin   -40,668,750,069
Difference ($)  -4,296,264
Difference (%)  0.011