Investigate differences in PRI code

qjhart commented 8 years ago

There are a few items in the current data that don't transfer over properly

Missing Nodes

The following nodes are missing from the calvin network data. CN1,CN3,CN4,CN5. They seem to be disconnected in the network anyways. (Except for CN2). There are also some errors in the ending storage for some reserviors.

for f in full full_dbg S09I05; do perl -ne '@F=unpack(a10a10a10a10a10); for ($i=0; $i<=2;$i++) {$F[$i]=~s/\s+//g;} ; printf "%s,%s,%.1f,%.3f,%.1f\n",$F[0],$F[1],$F[2],$F[3],$F[4],$F[5];' < $f.pri.node > $f.pri.node.0; done 
diff -u0iw S09I05.pri.node.0 full.pri.node.0

--- S09I05.pri.node.0   2016-02-18 18:08:52.807667767 -0800
+++ full.pri.node.0     2016-02-18 18:08:52.779667235 -0800
@@ -170 +169,0 @@
-NODE,CN1,0.0,0.000,0.0
@@ -172,3 +170,0 @@
-NODE,CN3,0.0,0.000,0.0
-NODE,CN4,0.0,0.000,0.0
-NODE,CN5,0.0,0.000,0.0
@@ -923 +919 @@
-NODE,SR_BLB,80.2,0.019,80.2
+NODE,SR_BLB,80.2,0.018,80.2
@@ -952 +948 @@
-NODE,SR_ML,2939.5,0.014,0.0
+NODE,SR_ML,2939.5,0.014,2939.5
@@ -956 +952 @@
-NODE,SR_ORO,2555.0,0.004,2633.4
+NODE,SR_ORO,2555.0,0.004,2633.3
@@ -969 +965 @@
-NODE,SR_SS,6941.0,0.017,0.0
+NODE,SR_SS,6941.0,0.017,6941.0
@@ -974 +970 @@
-NODE,SR_WHI,200.0,0.012,235.0
+NODE,SR_WHI,200.0,0.013,235.0

Regarding the Reserviors

select prmname,initialstorage,endingstorage from node_storage where prmname in ('SR_ML','SR_SS');
 prmname | initialstorage | endingstorage  
---------+----------------+----------------
 SR_ML   | 2939.541015625 | 2939.541015625
 SR_SS   |           6941 |           6941

or ...

select "PRMName","InitialStorage","EndingStorage" from "S09I05_in"."Nodes" where "PRMName" in ('SR_ML','SR_SS');
 PRMName | InitialStorage | EndingStorage 
---------+----------------+---------------
 SR_ML   |        2939.54 |       2939.54
 SR_SS   |           6941 |          6941

I am setting these to 0 and trying to run the model

Link issues.

There are two problems with the links. First, the precision of the data is off between the two. It appears as if LIAMP=precission to 0.XXX , LICOST=0.XX, and LILOWER=0.X00. More importantly, it seems as if negative LICOST are missing.
These are true for: C201-C71, D616-D42,D744-C92,U304-C79,SR_RLL_CMB-SR_RLL_CMB

Somehow link D884-N7 includes a positive LICOST no found in S09I05. It's because of a very big number in the cost function :(

calvin=# select "Name","Cost" from "Links" where "Name"='D884-N7';
   Name    |    Cost     
-----------+-------------
 D884-N7   | 3.40282e+38

Hmm, also for some reason the inflow for GW_MWD is not properly set. This is fixed.

{"type":"Feature",
 "geometry":{"type":"Point","coordinates":[-117.935524,33.981906]},
 "properties":{"prmname":"GW_MWD",
 "description":"Additional empty GW Capacity in MWD Service Area",
 "type":"Groundwater Storage",
 "initialstorage":750,
 "endingstorage":750,
 "inflows":null,
 "storage":{"$ref" : "storage.csv"},
 "costs":{"type":"Constant",
 "cost":0},
 "bounds":[{"type":"UBC","bound":1450}],
 "sinks":null,
 "flow":null}}

for f in full full_dbg S09I05; do perl -ne '@F=unpack(a10a10a10a10a10a10a10a10a10); for ($i=0; $i<=3;$i++) {$F[$i]=~s/\s+//g;} ; printf "%s,%s,%s,%s,%.3f,%.2f,%.1f,%.1f,%.3f\n",$F[0],$F[1],$F[2],$F[3],$F[4],($F[5]==0)?null:$F[5],$F[6],$F[7]+0.005,$F[8];' < $f.pri.link > $f.pri.link.0; done 
quinn@cws-quinn:~/HEC$ diff -u0iw S09I05.pri.link.0 full.pri.link.0--- S09I05.pri.link.0   2016-02-18 18:13:01.524397724 -0800
+++ full.pri.link.0     2016-02-18 18:13:01.464396584 -0800
@@ -314 +313,0 @@
-LINK,DIVR,CN1,SINK,1.000,0.00,0.0,0.0,0.000
@@ -317,3 +315,0 @@
-LINK,DIVR,CN3,SINK,1.000,0.00,0.0,0.0,0.000
-LINK,DIVR,CN4,SINK,1.000,0.00,0.0,0.0,0.000
-LINK,DIVR,CN5,SINK,1.000,0.00,0.0,0.0,0.000
@@ -1332 +1327,0 @@
-LINK,DIVR,SOURCE,SINK,1.000,0.00,0.0,0.0,0.000
@@ -1666 +1660,0 @@
-LINK,INFL,SOURCE,CN1,1.000,0.00,0.0,0.0,0.000
@@ -1668,3 +1661,0 @@
-LINK,INFL,SOURCE,CN3,1.000,0.00,0.0,0.0,0.000
-LINK,INFL,SOURCE,CN4,1.000,0.00,0.0,0.0,0.000
-LINK,INFL,SOURCE,CN5,1.000,0.00,0.0,0.0,0.000

Previious question re. Evaporation Rates

A lso, there are two items, that my program sez should be in there, and your's doesn't show. These two Evaporation rates are not in S09I05, but they have db inputs.

-EV        A= B=SR_BVLB C=EVAP_RATE(FT) D= E=1MON F=
-EV        A= B=SR_TL C=EVAP_RATE(FT) D= E=1MON F=

Questionable Monthly Costs

The two items with questionable monthly costs are: here

Differing methods of Month Blocking

./sacramento-river/northwest-valley/ud_reddin/hxi101-ext_reddin/costs/JAN.csv
./sacramento-river/northwest-valley/ud_reddin/hxi101-ext_reddin/costs/FEB.csv
./sacramento-river/central-basin-east/ud_yuba/hxi201-ext_yuba/costs/MAR.csv
./sacramento-river/central-basin-east/ud_yuba/hxi201-ext_yuba/costs/DEC.csv

capacity,cost 
0,0.01 
1e-05,0

./sacramento-river/central-basin-east/ud_yuba/hxi201-ext_yuba/costs/NOV.csv
./sacramento-river/central-basin-east/ud_yuba/hxi201-ext_yuba/costs/JAN.csv
./sacramento-river/central-basin-east/ud_yuba/hxi201-ext_yuba/costs/FEB.csv
capacity,cost 
0,0.01
1e-06,0 
}

./south-coast/santa-clara/ag_vent/hu501-ags_vent/costs/FEB.csv capacity,cost 0,0.05 0.01,0

./sacramento-river/central-basin-west/ag_cvpm06/hu203-cvpm06s/costs/DEC.csv

capacity,cost 0,0.06 0.01,0

55 files like

capacity,cost
0,0 
1000000,0

for f in `find . -name \*.csv | xargs wc -l | sort -nr | grep ' 3 \./' | sed -e 's/.*3 \././'`; do if (grep -q 1000000 $f) then echo $f; fi; done | wc -l

qjhart commented 8 years ago

@msdogan you can read above, but input issues are summarized here.

These first two are mostly just FYI, I can fix them in the conversion,

[ ] S09I05PD.dss Path: /SOUTH UPDT/T61_EXT: YUBA/Q(K$-KAF)-P//FEB/2050H POP 030911/ has second value 1e-7 which using HEC export comes out as 0.0000, making errorneous input
[ ] S09I05PD.dss : Path /SOUTH UPDT/T41_EXT: REDDING/Q(K$-KAF)-P//FEB/2050H POP 030911/ has second value 1e-7 which HEC exports as 0.00000 making erroneous input

This one, I don't know about.

[ ] Link D884-N7 has a Constant cost of 3.40282e+38 in the db. It's 0.00 in the S09I05.pri file. Don't know which is correct.

This I don't know about either.

[ ] Two reservoirs have evaporation pathnames that are OK in the database, but are not included in the S09I05.pri file.

  PRMName |                EvaporationRatePathname                
---------+-------------------------------------------------------
 SR_BVLB | /HEXT2014/SR-BVLB/EVAP_RATE(FT)//1MON/BUENA VISTA LB/
 SR_TL   | /HEXT2014/SR-TL/EVAP_RATE(FT)//1MON/TULARE LAKE BED/

msdogan commented 8 years ago

@qjhart Let's start with Calibration Nodes (CN).

There are 5 Calibration Nodes, one for each region. Each CN has one source and one sink. These are only to make mass balance work. For example, if there is a problem with mass balance in a node in Region 1 (Upper Sac. Valley), we use CN1. Currently, only CN2 is being used in the upper Bear River watershed in the Historical Base CALVIN run. CNs does not have any cost associated with. How we decide to use is that we run in Debug Mode, if we keep getting debug flow can't eliminate it, we use calibration flows to add or remove water to make mass balance work and get a feasible solution.

No end-of-period-storage values for some reservoirs.

As we discusses in the last meeting, reservoirs (mostly natural lakes) acting as sinks do not have ending storage value. Those reservoirs have only inlets and only outlet is evaporation. So, we don't know what would be ending storage is and let CALVIN decide.

Link issues:

Yes, we don't have a convention on precision of link amplitude and cost. It is generally two or three precision.

Negative Link Cost:

Since this is a cost minimization model, any benefits, such as hydropower (C201-C71), appear as negative cost. Another use negative cost is persuasion. Persuasion cost (D616-C42, D744-C92) is relatively small. It is a way to "manipulate" or "calibrate" the model. For U304-C79, I am copy and paste the metadata: Credit for downstream Hetch Hetchy water treatment cost on SR-ASF_T20 (-$5/af) plus for SCV local distribution/conveyance charge (-$95/af). Corrected in V42 from -125 to -100 (RR).

Right now, it is 148 instead of 100 because some years ago, all operating cost was multiplied with 1.48 to reflect inflation ratio.

D884-N7: I think that big number represents positive infinity in Access Database or HEC-PRM. Looks like the reason is we chose constant for the cost method but we didn't put any number. Let's update cost method as "none"

Reservoir Evaporation:

I have checked SR_TL and SR_BVLB. They both seem to have evaporation time-series data but their area-elevation-capacity conversion factors are set to zero. That could be the reason.

Urban Areas Penalty Precision (Redding & Yuba):

This could be as a result of an error in penalty generator macro for urban areas. I don't know why the model gives error when PRM sets them equal to zero. any idea @josue-medellin @mimijenkins1

mimijenkins1 commented 8 years ago

For the last point, is this about the differing methods of month blocking ?

i.e., copied here below (FEB exterior paired penalty data for yuba and redding)

ext_YUBA: capacity,cost 0,0.01 -0,0 +1e-06,0

ext_REDDING: capacity,cost -0,0 0,0.01 +1e-05,0

As Mustafa has noted these are probably round-off and precision formatting errors generated by the Macro, and were not cleaned up before uploading the Macro paired data into DSS

Some things that might cause PRM error problems:

the "negative zero" value ?
- cost Y value increasing from 0 to 0.01 and then decreasing back to 0
- quantity X value going from 0 to -0 to +1e-06

ucd-cws / calvin-network-data