Updates are not propagated from a PiecewiseYieldCurve after a failed bootstrap

[tomwhoiscontrary]

I don't know if this is a bug.

If you build a curve based on some quotes, and then observe that curve, you will get an update the first time any quote is changed following a bootstrap calculation of the curve. This makes sense - until a quote is changed, the curve stays the same; once a quote is changed, it needs to be re-calculated, and if several quotes are changed, it still needs to be re-calculated, so there is no need to send more updates.

However, if one or more of the quotes has a "bad" value (eg NaN, or absurdly huge), then the next attempt to calculate the curve fails. If the quotes are changed again, replacing the bad value with a good one, you do not get an update.

If the purpose of an update is to tell observers "discard your cached values", then this is fine, because observers of a failed curve cannot have any cached values. On the other hand, if the purpose of an update is to tell observers "come and get a new value", then this is not fine, because there is a new value which might never be observed.

I am, perhaps rashly, attempting to structure a program around updates.The idea is that i build a model, feed in updates to quotes as they arrive from market data sources, then use the observer mechanism to outputs which have changed. When this works, it's very nice, because i only have to read out actual changes, not every output every time, and i don't have to manually track dependencies between outputs and inputs. However, it falls apart in the case of failure - a failed curve becomes a black hole for updates!

I'm not sure what the alternative is here. A failed curve could set its calculated flag, so that subsequent updates would clear it and propagate themselves. But then what would happen when someone tries to read a value from this allegedly calculated curve? There is no curve, so the read would have to fail. That means the curve would have to somehow remember that it failed, and throw an exception when queried. That seems a bit odd.

Anyway, below is an example. It prints:

curve: registered
fra: registered
Initialising quotes ...
forwardRate: 1st iteration: failed at 3rd alive instrument, pillar April 26th, 2021, maturity April 26th, 2021, reference date January 23rd, 2018: root not bracketed: f[-1,1] -> [nan,nan]
Updating quotes ...
1.000028 % Actual/360 simple compounding

For me, ideally it would print:

curve: registered
fra: registered
Initialising quotes ...
forwardRate: 1st iteration: failed at 3rd alive instrument, pillar April 26th, 2021, maturity April 26th, 2021, reference date January 23rd, 2018: root not bracketed: f[-1,1] -> [nan,nan]
Updating quotes ...
curve: updated
fra: updated
1.000028 % Actual/360 simple compounding

The code:

#include <iostream>

#include <ql/handle.hpp>
#include <ql/indexes/ibor/usdlibor.hpp>
#include <ql/instruments/forwardrateagreement.hpp>
#include <ql/quotes/simplequote.hpp>
#include <ql/termstructures/yield/piecewiseyieldcurve.hpp>
#include <ql/termstructures/yield/ratehelpers.hpp>

#include <boost/make_shared.hpp>

using QuantLib::Cubic;
using QuantLib::Date;
using QuantLib::ForwardRate;
using QuantLib::ForwardRateAgreement;
using QuantLib::FraRateHelper;
using QuantLib::Handle;
using QuantLib::IborIndex;
using QuantLib::Observable;
using QuantLib::Observer;
using QuantLib::Period;
using QuantLib::PiecewiseYieldCurve;
using QuantLib::Position;
using QuantLib::Quote;
using QuantLib::RateHelper;
using QuantLib::RelinkableHandle;
using QuantLib::SimpleQuote;
using QuantLib::TimeUnit;
using QuantLib::USDLibor;
using QuantLib::YieldTermStructure;

// this is a very simple observer that just prints a message when updated
class Reporter : public Observer {

std::string label;

public:

Reporter(std::string label, boost::shared_ptr<Observable> observable) : label(label){
        registerWith(observable);
        std::cout << label << ": registered" << std::endl << std::flush;
}

void update() {
        std::cout << label << ": updated" << std::endl << std::flush;
}

};

int main(int argc, char **argv) {
        // set up the index
        RelinkableHandle<YieldTermStructure> curveHandle;
        boost::shared_ptr<IborIndex> index = boost::make_shared<USDLibor>(Period(3, TimeUnit::Months), curveHandle);

        // set up quotes
        std::vector<boost::shared_ptr<SimpleQuote> > quotes = {
                boost::make_shared<SimpleQuote>(),
                boost::make_shared<SimpleQuote>(),
                boost::make_shared<SimpleQuote>()
        };

        // set up the curve
        std::vector<boost::shared_ptr<RateHelper> > helpers = {
                boost::make_shared<FraRateHelper>(Handle<Quote>(quotes[0]), Period(1, TimeUnit::Years), index),
                boost::make_shared<FraRateHelper>(Handle<Quote>(quotes[1]), Period(2, TimeUnit::Years), index),
                boost::make_shared<FraRateHelper>(Handle<Quote>(quotes[2]), Period(3, TimeUnit::Years), index)
        };

        boost::shared_ptr<PiecewiseYieldCurve<ForwardRate, Cubic> > curve = boost::make_shared<PiecewiseYieldCurve<ForwardRate, Cubic> >(Date::todaysDate(),
                                                                                                                                         helpers,
                                                                                                                                         index->dayCounter());

        curveHandle.linkTo(curve);

        Reporter curveReporter("curve", curve);

        // set up the instrument to price
        boost::shared_ptr<ForwardRateAgreement> fra = boost::make_shared<ForwardRateAgreement>(Date::todaysDate() + Period(6, TimeUnit::Months),
                                                                                               Date::todaysDate() + Period(18, TimeUnit::Months),
                                                                                               Position::Type::Long,
                                                                                               0,
                                                                                               1,
                                                                                               index,
                                                                                               curveHandle);

        Reporter fraReporter("fra", fra);

        // first round of updates - set an incomplete set of rates, and try in vain to price the instrument
        std::cout << "Initialising quotes ..." << std::endl;
        quotes[0]->setValue(0.01);
        quotes[1]->setValue(0.02);
        quotes[2]->setValue(std::numeric_limits<double>::quiet_NaN());

        try {
                std::cout << fra->forwardRate() << std::endl;
        } catch(std::exception& e) {
                std::cerr << "forwardRate: " << e.what() << std::endl;
        }

        // second round of updates - complete the set of rates, and succeed in pricing the instrument
        std::cout << "Updating quotes ..." << std::endl;
        quotes[2]->setValue(0.03);

        try {
                std::cout << fra->forwardRate() << std::endl;
        } catch(std::exception& e) {
                std::cerr << "forwardRate: " << e.what() << std::endl;
        }
}

[lballabio]

You're right, that's a problem. The idea was that, after forwarding a notification, the LazyObject machinery wouldn't forward further ones until someone requested for it to be calculated; that's because we supposed that this would mean that the observers still hadn't updated. We missed that case in which they tried to update but failed.

To treat this properly, we probably have to store another boolean in LazyObject. For the time being, you can force all updates to be forwarded by calling the alwaysForwardNotifications on the term structure. However, (a) this will decrease performance in the successful cases and (b) be aware of the problems I outline in https://www.youtube.com/watch?v=RUZyg3-YdtM.

[tomwhoiscontrary]

I've run into this again. This time, it's a VanillaSwap that is priced using forecast fixings from a term structure in its IborIndex, and discount factors from a term structure in a DiscountingSwapEngine. Initially, the handles for both those term structures are empty. When one handle gets populated, the VanillaSwap updates, but it cannot actually be priced. When the other handle gets populated, the VanillaSwap does not update, even though it can now be priced.

If i was starting from scratch, then rather than a boolean, i would use a three-state enum here - something like NOT_CALCULATED, CALCULATED, CALCULATIONFAILED. But since the calculated boolean is protected, i suppose it's too late to do this.

There are currently three boolean fields in LazyObject. Alignment rules mean those will take up four bytes even on 32- or 16-bit machines (!), so there should be no space overhead to adding a boolean to track failure.

I can work out what logic is required in LazyObject to handle this. A class extending LazyObject and just implementing performCalculations should work correctly. But could there be classes in QuantLib that would be subtly broken by that change? Classes with their own logic around calculated_?

If this is not definitely going to be a nightmare, i am happy to have a go at a PR for this!

[lballabio]

I don't think other classes built around calculated_. You might have a try and see what happens to the test suite...

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.