Bug: Pricepoint.getDemandAt() stuck in while loop

[alex254]

Hi,

With bjorn ive discovered another bug in Pricepoint.getDemandAt()...we are porting REX to PM2.0 whic has a fake agent that is sending out random bids.These are pricepoint bids which are never passed on by the concentrator to the auctioneer.We have started digging and found out that it hangs when converting the pointbid to an arraybid. Constructing an ArrayBid(Pointbid) calls Pricepoint.getDemandAt()

We noticed that cmp variable becomes 0 which causes it to get stuck in the while loop:

` @Override

public double getDemandAt(Price price) {

    if (pricePoints.length == 1) {
        // Flat bid, send any demand (they are all the same)
        return getMaximumDemand();

    } else if (price.compareTo(getFirst().getPrice()) < 0) {
        // If the price is lower than the lowest price, return the maximum
        // demand
        return getMaximumDemand();
    } else if (price.equals(getFirst().getPrice())) {

        // If the first matcher, it could be that the second is at the same price. If that is the case, use the // second, otherwise the first.

        PricePoint secondPricePoint = pricePoints[1];

        if (price.equals(secondPricePoint.getPrice())) {
            return secondPricePoint.getDemand();

        } else {
            return getMaximumDemand();
        }

    } else if (price.compareTo(getLast().getPrice()) >= 0) {
        // If the price is higher than the highest price, return the minimum
        // demand

        return getMinimumDemand();
    } else {

        // We have a normal case that is somewhere in between the lower and higher demands
       // First determine which 2 pricepoints it is in between
        int lowIx = 0, highIx = pricePoints.length;
        while (highIx - lowIx > 1) {
            int result = highIx - lowIx;

            LOGGER.info("Alex: " + Integer.toString(result));
            LOGGER.info("Alex price: " + price);
            LOGGER.info("Alex pricepoints: " + pricePoints);
            LOGGER.info("Alex high: " + highIx);
            LOGGER.info("Alex low: " + lowIx);

            int middleIx = (lowIx + highIx) / 2;
            PricePoint middle = pricePoints[middleIx];

            LOGGER.info("Alex pricepointMidden: " + middle);

            int cmp = middle.getPrice().compareTo(price);

            LOGGER.info("Alex cmp: " + cmp);

            if (cmp < 0) {
                lowIx = middleIx;
            } else if (cmp > 0) {
                highIx = middleIx;
            } else {
                // Found at least 1 point that is equal in price.
                // This is the special case with an open and closed node. Always the lower demand should be chosen.
                PricePoint nextPoint = pricePoints[middleIx + 1];
                if (price.equals(nextPoint.getPrice())) {
                    return nextPoint.getDemand();
                } else {
                    middle.getDemand();
                }
            }
        }
        PricePoint lower = pricePoints[lowIx];
        PricePoint higher = pricePoints[highIx];

        // Now calculate the demand between the 2 points
        // First the factor (between 0 and 1) of where the price is on the line
        double factor = (price.getPriceValue() - lower.getPrice().getPriceValue())
                        / (higher.getPrice().getPriceValue() - lower.getPrice().getPriceValue());
        // Now calculate the demand
        return (1 - factor) * lower.getDemand() + factor * higher.getDemand();
    }
}

`

Here you can see the output of the logging in the console:

[Lnet.powermatcher.api.data.PricePoint;@7665b1 12:10:12.206 [pool-2-thread-2] INFO net.powermatcher.api.data.ArrayBid - Alex high: 26 12:10:12.206 [pool-2-thread-2] INFO net.powermatcher.api.data.ArrayBid - Alex low: 13 12:10:12.206 [pool-2-thread-2] INFO net.powermatcher.api.data.ArrayBid - Alex pricepointMidden: {15 -> 3.15E5} 12:10:12.206 [pool-2-thread-2] INFO net.powermatcher.api.data.ArrayBid - Alex cmp: 0 12:10:12.206 [pool-2-thread-2] INFO net.powermatcher.api.data.ArrayBid - Alex: 13 12:10:12.206 [pool-2-thread-2] INFO net.powermatcher.api.data.ArrayBid - Alex price: Price{priceValue=15} 12:10:12.206 [pool-2-thread-2] INFO net.powermatcher.api.data.ArrayBid - Alex pricepoints: [Lnet.powermatcher.api.data.PricePoint;@7665b1 12:10:12.206 [pool-2-thread-2] INFO net.powermatcher.api.data.ArrayBid - Alex high: 26 12:10:12.206 [pool-2-thread-2] INFO net.powermatcher.api.data.ArrayBid - Alex low: 13 12:10:12.206 [pool-2-thread-2] INFO net.powermatcher.api.data.ArrayBid - Alex pricepointMidden: {15 -> 3.15E5} 12:10:12.206 [pool-2-thread-2] INFO net.powermatcher.api.data.ArrayBid - Alex cmp: 0 12:10:12.206 [pool-2-thread-2] INFO net.powermatcher.api.data.ArrayBid - Alex: 13 12:10:12.206 [pool-2-thread-2] INFO net.powermatcher.api.data.ArrayBid - Alex price: Price{priceValue=15} 12:10:12.206 [pool-2-thread-2] INFO net.powermatcher.api.data.ArrayBid - Alex pricepoints: [Lnet.powermatcher.api.data.PricePoint;@7665b1 12:10:12.206 [pool-2-thread-2] INFO net.powermatcher.api.data.ArrayBid - Alex high: 26 12:10:12.206 [pool-2-thread-2] INFO net.powermatcher.api.data.ArrayBid - Alex low: 13 12:10:12.206 [pool-2-thread-2] INFO net.powermatcher.api.data.ArrayBid - Alex pricepointMidden: {15 -> 3.15E5} 12:10:12.206 [pool-2-thread-2] INFO net.powermatcher.api.data.ArrayBid - Alex cmp: 0 12:10:12.206 [pool-2-thread-2] INFO net.powermatcher.api.data.ArrayBid - Alex: 13 12:10:12.206 [pool-2-thread-2] INFO net.powermatcher.api.data.ArrayBid - Alex price: Price{priceValue=15} 12:10:12.206 [pool-2-thread-2] INFO net.powermatcher.api.data.ArrayBid - Alex pricepoints: [Lnet.powermatcher.api.data.PricePoint;@7665b1 12:10:12.206 [pool-2-thread-2] INFO net.powermatcher.api.data.ArrayBid - Alex high: 26 12:10:12.206 [pool-2-thread-2] INFO net.powermatcher.api.data.ArrayBid - Alex low: 13 12:10:12.206 [pool-2-thread-2] INFO net.powermatcher.api.data.ArrayBid - Alex pricepointMidden: {15 -> 3.15E5} 12:10:12.206 [pool-2-thread-2] INFO net.powermatcher.api.data.ArrayBid - Alex cmp: 0 12:10:12.206 [pool-2-thread-2] INFO net.powermatcher.api.data.ArrayBid - Alex: 13 12:10:12.206 [pool-2-thread-2] INFO net.powermatcher.api.data.ArrayBid - Alex price: Price{priceValue=15} 12:10:12.206 [pool-2-thread-2] INFO net.powermatcher.api.data.ArrayBid - Alex pricepoints: [Lnet.powermatcher.api.data.PricePoint;@7665b1 12:10:12.206 [pool-2-thread-2] INFO net.powermatcher.api.data.ArrayBid - Alex high: 26 12:10:12.206 [pool-2-thread-2] INFO net.powermatcher.api.data.ArrayBid - Alex low: 13 12:10:12.206 [pool-2-thread-2] INFO net.powermatcher.api.data.ArrayBid - Alex pricepointMidden: {15 -> 3.15E5} 12:10:12.207 [pool-2-thread-2] INFO net.powermatcher.api.data.ArrayBid - Alex cmp: 0 12:10:12.207 [pool-2-thread-2] INFO net.powermatcher.api.data.ArrayBid - Alex: 13 12:10:12.207 [pool-2-thread-2] INFO net.powermatcher.api.data.ArrayBid - Alex price: Price{priceValue=15} 12:10:12.207 [pool-2-thr

I believe it gets stuck here:

} else { middle.getDemand(); }

But adding a return before caused another problem; the bidcurve suddenly became asccending and was rejected...

[wilcowijbrandi]

Ok, that is definitely a bug then. Do you by any change have the code that generates the bid and price that causes the problem? Would make debugging a lot easier. :)

[marcdejonge]

You are right Alex, looking at this code in that else statement it should return that middle.getDemand(). The question then is why the resulting bidcurve is ascending. If you can share the input PointBid, we can check why that happens.

[alex254]

Wilco I've sent you the bidGenerator..maybe the problem lies there i added a return statement before middle.getDemand()

in arrayBid a bid looks as follows:

[1578000.0, 1551000.0, 1551000.0, 1531000.0, 1531000.0, 1530999.9999999998, 1531000.0, 1531000.0,....

you can see there is probably a rounding error somewhere; have to check whether that's in the orignal bidgenerator or later on in the code...

[alex254]

ehm leave it to me for now; I think the problem is in the bidgenerator...it produces really strange pointbids.

however it might accidentily have uncovered a problem in the concentator where a single bid is not directly one on one the same as the aggregatedBid, even though it is a weird Bid.... if there is only one bid in the bidcache the aggregatedBid should turn out the same as the original bid.

[alex254]

it seems the bidgenerator is flawed because it generates multiple pricepoint on the same x coordinate:

[0 -> 10, 0 -> 8, 0-> 6…..

we will fix this...two pricepoints should be okay but more than that is not needed.

But apparantly the powermatcher starts doing weird things when there is multiple points on the same x coordinate. For robustness sake we should perform a check when constructing a bidladder that there is no more than 2 coordinates on the same x coordinate. For instance all points that are in between should be ignored before sending the pointbid onwards...

[alex254]

arghh i'm lost a bit..there does seem to be a bug in the powermatcher. if I follow the pointBid generated by the bidgenerator (even though there are multiple coordinates on the x axes which is fine for now) it arrives at the concentrator at some point the concentrator basematcher is scheduled to aggregate the bid from the bidcache it retrieves the pointbid correctly then goes into builder.addAgentbid using the correct pointbid it goes into addBid with the correct pointbid and then it goes wrong here:

    public Builder addBid(Bid bid) {
        if (bid.getMarketBasis().equals(marketBasis)) {
            double[] demand = bid.toArrayBid().getDemand();
            for (int ix = 0; ix < marketBasis.getPriceSteps(); ix++) {
                aggregatedBid[ix] += demand[ix];
            }
        }
        return this;
    }

at double[] demand = bid.toArrayBid().getDemand();

getDemand goes into the arrayBid and returns a demandArray which is NOT EQUAL to pointBid in that arrayBid. The PointBid is however the correct pointBid that I was following all the time...

I don't understand where this demandArray variable is created and why it is not equal pointBid variable...does this happen when the concentrator receives at the BaseAgentEndpoint?

[alex254]

okay I think I figured out where it goes wrong!

it is again here in getDemandAt():

`

ArrayBid(PointBid base) {

    super(base.getMarketBasis());

    int priceSteps = marketBasis.getPriceSteps();
    demandArray = new double[priceSteps];

    for (int ix = 0; ix < priceSteps; ix++) {
        demandArray[ix] = base.getDemandAt(new PriceStep(marketBasis, ix));
    }

    pointBid = base;
}

`

base.getDemandAt is creating the wrong demandArray from the pointBid. Probably for two reasons:

1) the pointBid has multiple coordinates on one x coordinate 2) the pointBid has 189 values >> 100 pricesteps

So either we need to check on these aspects and not allow them or finetune getDemandAt() so that it can handle these cases.

[wilcowijbrandi]

Thanks for looking into this problem @alex254. We've run into more troubles with the two different ways to represent a bid (ArrayBid and PointBId). Although both representations seem equivalent, they differ quite a lot in the details. Issue #199 is also causing problems at our side. My plan was to take some time next week to look into the details of these bugs.

We could try to fix all these individual bugs, but there maybe is an alternative. The PointBid representation is mainly used because it is easier to construct, which is nice if you have to develop a device agent. However, when bids are aggregated we always use the ArrayBid representation (because aggregating ArrayBids is more efficient). I think it would simplify things if we would switch to only using the ArrayBid representation, and offer the PointBid only as a convenience method to easily construct an ArrayBid. In other words: PointBids always get instantly converted into ArrayBids.

The downside of this change is that it would mean a significant API change.

Any thoughts?