Piecewise Cost Representation in Matrix Export

[msdogan]

@jrmerz @qjhart the piecewise cost representation seems to have some bugs in it. We have one extra link in the beginning (when k = 0), and it is constrained. Below, there is a link with piecewise cost HU507 to AGG_COACH. There are two months: July and August. I wonder why it constrains first pieces (k=0) with zero cost. This is true for all links with piecewise representation.

i   j   k   cost    amplitude   lower_bound upper_bound

HU507.2002-07-31    AGG_COACH.2002-07-31    0   0   1   18.49   18.49
HU507.2002-07-31    AGG_COACH.2002-07-31    1   -2450.118243    1   0   1.85
HU507.2002-07-31    AGG_COACH.2002-07-31    2   -2381.098208    1   0   1.84
HU507.2002-07-31    AGG_COACH.2002-07-31    3   -2243.863863    1   0   1.850001
HU507.2002-07-31    AGG_COACH.2002-07-31    4   -1961.753071    1   0   1.849998
HU507.2002-07-31    AGG_COACH.2002-07-31    5   -1641.658741    1   0   1.850001
HU507.2002-07-31    AGG_COACH.2002-07-31    6   -1423.632549    1   0   1.85
HU507.2002-07-31    AGG_COACH.2002-07-31    7   -1208.016126    1   0   1.85
HU507.2002-07-31    AGG_COACH.2002-07-31    8   -979.330264 1   0   1.849999
HU507.2002-07-31    AGG_COACH.2002-07-31    9   -768.2532673    1   0   1.850002
HU507.2002-07-31    AGG_COACH.2002-07-31    10  -337.3858576    1   0   1.84
HU507.2002-07-31    AGG_COACH.2002-07-31    11  0   1   0   0

HU507.2002-08-31    AGG_COACH.2002-08-31    0   0   1   15  15
HU507.2002-08-31    AGG_COACH.2002-08-31    1   -2453   1   0   1.5
HU507.2002-08-31    AGG_COACH.2002-08-31    2   -2369.833984    1   0   1.5
HU507.2002-08-31    AGG_COACH.2002-08-31    3   -2231.628984    1   0   1.51
HU507.2002-08-31    AGG_COACH.2002-08-31    4   -1963.080078    1   0   1.5
HU507.2002-08-31    AGG_COACH.2002-08-31    5   -1643.593099    1   0   1.5
HU507.2002-08-31    AGG_COACH.2002-08-31    6   -1424.586914    1   0   1.5
HU507.2002-08-31    AGG_COACH.2002-08-31    7   -1209.439941    1   0   1.5
HU507.2002-08-31    AGG_COACH.2002-08-31    8   -979.9932453    1   0   1.5
HU507.2002-08-31    AGG_COACH.2002-08-31    9   -769.160034 1   0   1.5
HU507.2002-08-31    AGG_COACH.2002-08-31    10  -335.9533287    1   0   1.5
HU507.2002-08-31    AGG_COACH.2002-08-31    11  0   1   0   0

Aha, I think I got the reason. For HU507 to AGG_COACH for July, our penalty data starts from 18.49. Below this number there is nothing. Because of that, your code constrains it. Below are figures from penalty data DSS.

I think the decision was to extend the beginning and ending penalties. As far as I know, HEC-PRM does that automatically. Am I right @qjhart @mimijenkins1?

[jdherman]

Ok -- I'm fine with making it general. But we still have the problem of identifying where this is happening, because we have to somehow separate out the links for which the upper bound value in the dataset is not representative of the real-world upper bound. How to do that? It must be based on a specific type of nodes/links.

[jrmerz]

Could this be a declarative flag on the data? So Any link/node can declaratively specify 'representative' v 'real-world' upper bound? And then code would look like:

if slope negative
  if UB exists and bound not representative
      enforce u = UB for last k

This would assume real world by default. So the flag would be representative = true.

[jdherman]

Something like this, yea. Our problem in this case is that the "upper bound" in the data for reservoirs actually represents the turbine capacity, not the flow capacity constraint. So it is representative of something, just not what we want it to be.

@msdogan can you think of other types of links where this might be happening? Or is it only reservoir release links?

[msdogan]

I can't think of any other types of links but @jrmerz could you do a formal check for this case: piecewise cost, slope negative and no physical upper bound?

[jrmerz]

Here is the current list meeting the above conditions:

PWP_Mojave-SR_SLW, SR_NML-D670, SR_LL_ENR-SR_DNP, SR_HTH-C44, SR_DNP-D662, SR_MCR-D642, D541-Surp_Delta, SR_SHA-D5, SR_ORO-C23, SR_ENG-C28, SR_BUL-C27, D5-D73, SR_TAB-C25, SR_FOL-D9, D9-D85, C23-C25, SR_CLE-D94, PWP_AAC-C151

[jdherman]

Current thought:

if slope negative:
  if UB exists:
    add new piece up to UB at cost 0 (flat slope)
  else:
    add new piece up to max_UB at cost 0 (flat slope)

[jdherman]

Copying this from Mustafa's email just so we have a record of it:

Okay, while checking the network, I found another bug in big "if statement" logic. https://cloud.githubusercontent.com/assets/1936620/18797140/505be90c-8181-11e6-84f6-d0ef0ae7e0eb.JPG

For left-hand side extension for slope negative and physical lb does not exist case, our statement says: enforce lb=0 for k=0 case. But does not say anything for the ub. Before the extension, ub on the k=0 case (sorry I forgot start from 0 index in the sketch below) is b - a. After extension it should be b, but it is still b - a in network matrix because we forgot to add this in the if statement.

Sorry Justin, but could you update your code again to include that. Hopefully this would be the last modification for cost extension.

[jrmerz]

Logic updated. But the cost of slope negative, Physical Lower Bound does not exist, we only modify lb if lb wasn't already equal to 0. Currently, no nodes/links meet this condition. In fact no nodes/links have slope negative and an undefined Physical Lower Bound.

[lzhangisu2011]

Just saw another example might be related to piecewise cost representation issue:

Infeasibility is for reservior node SR_MCR in the final month.

Here is all inflows to this node: i j k cost amplitude lower_bound upper_bound SR_MCR.2002-11-30 SR_MCR.2002-12-31 0 -2.309251768 0.999606565 115 126.214554 SR_MCR.2002-11-30 SR_MCR.2002-12-31 1 -1.473401553 0.999606565 0 182.383865 SR_MCR.2002-11-30 SR_MCR.2002-12-31 2 -1.009400653 0.999606565 0 254.401581 INFLOW.2002-12-31 SR_MCR.2002-12-31 0 0 1 33.66198 33.66198

The outflow of this node is: i j k cost amplitude lower_bound upper_bound SR_MCR.2002-11-30 SR_MCR.2002-12-31 2 -1.009400653 0.999606565 0 254.401581 SR_MCR.2002-12-31 FINAL 0 0 1 633.792908 633.792908

There is about 37.1309 difference between inflow and outflow.

[msdogan]

I found the bug, the upper bound on the first piece (k=0) of SR_MCR storage link should be 242.287 not 126.214554.

SR_MCR.2002-11-30 SR_MCR.2002-12-31 0 -2.309251768 0.999606565 115 126.214554

We probably missed this in the big if statement. I am going to the class now but I will try to find where to fix once I get a chance. Thanks @lzhangisu2011 for finding it.

[msdogan]

@jrmerz I am closing this issue. It is getting too large and hard to follow. We will open a new one if we get any problem with piecewise cost representation in the future.

[jrmerz]

agreed.