Memory Leak Observed with BlackVarianceSurface

[aradhya-variational-io]

Using BlackVarianceSurface with python lists, noticing a steep increase in memory usage when taking heap snapshots. Confirmed this is the case by removing the following call and confirming that no memory was leaking afterwards.

If it helps, expiration_dates and strikes are of type list and implied_vols is a matrix. Memory grows consistently with each call to the below function. Currently on version 1.30. Any insight would be greatly appreciated!

  local_vol_surface = ql.BlackVarianceSurface(
            today, CALENDAR, expiration_dates, strikes, implied_vols, DAY_COUNTER
        )

[boring-cyborg[bot]]

Thanks for posting! It might take a while before we look at your issue, so don't worry if there seems to be no feedback. We'll get to it.

[lballabio]

Thanks for the report—when you say that implied_vols is a matrix, do you mean a list of lists or an instance of ql.Matrix?

[aradhya-variational-io]

Hi, thanks for the response! implied_vols is a ql.Matrix

implied_vols = ql.Matrix(np.matrix(implied_vols).transpose().tolist())

expiration_dates is a list, and strikes is a numpy.ndarray

[lballabio]

Sorry, I'm afraid I'm not able to reproduce your problem. Can you attach a sample script that does?

[lballabio]

Also, are you using the usual python or pypy? What version?

[aradhya-variational-io]

Sorry, I'm afraid I'm not able to reproduce your problem. Can you attach a sample script that does?

Sure I can extract this code out and try to reproduce with some static inputs.

Also, are you using the usual python or pypy? What version?

Regular python, 3.10

[aradhya-variational-io]

Hi! The below example reproduces the issue fairly quickly, im running this locally on python 3.11 (i believe the result is the same with 3.10 but happy to verify that). For monitoring the memory usage, I am using the activity monitor on the mac, but I also confirmed the same with the leaks command.

import numpy as np
import pandas as pd
import QuantLib as ql

DAY_COUNTER = ql.Actual365Fixed()
CALENDAR = ql.NullCalendar()

def timestamp_to_ql_date(ts: pd.Timestamp) -> ql.Date:
    """Convert a pandas timestamp to a QuantLib date."""
    return ql.Date(ts.day, ts.month, ts.year)

def create_vol_surface() -> ql.BlackVarianceSurface:
    spot_data = {
        "mark_price": 1849.6966453526368,
        "last_updated": pd.Timestamp("2023-08-12 15:56:04+0000", tz="UTC"),
    }
    today = timestamp_to_ql_date(spot_data["last_updated"])
    params = {
        0.004573970983113478: {
            "v0": 15.170950808,
            "beta": 0.410832031,
            "alpha": 12.432745001,
            "rho": 0.15529925,
            "fwd": 1850.4114056129265,
        },
        0.007313697079674025: {
            "v0": 14.906051139,
            "beta": 0.4,
            "alpha": 11.942237828,
            "rho": 0.101944605,
            "fwd": 1850.8396663723117,
        },
        0.015532874906817004: {
            "v0": 13.372962753,
            "beta": 0.470465007,
            "alpha": 6.654669694,
            "rho": 0.019834329,
            "fwd": 1852.1250433720359,
        },
        0.0347109517674721: {
            "v0": 13.116499508,
            "beta": 0.469523096,
            "alpha": 4.917684581,
            "rho": 0.054054555,
            "fwd": 1854.3184321399792,
        },
        0.053889039445200786: {
            "v0": 13.029888358,
            "beta": 0.462742212,
            "alpha": 4.021565548,
            "rho": 0.095596906,
            "fwd": 1856.514619792689,
        },
        0.13060136812880413: {
            "v0": 12.926640797,
            "beta": 0.490574797,
            "alpha": 2.438944596,
            "rho": 0.029542936,
            "fwd": 1865.3075692280117,
        },
        0.2073136910406006: {
            "v0": 12.889959062,
            "beta": 0.486859434,
            "alpha": 2.01669713,
            "rho": 0.041829942,
            "fwd": 1872.2264136680571,
        },
        0.37991643328852437: {
            "v0": 12.861615412,
            "beta": 0.503213714,
            "alpha": 1.498654785,
            "rho": 0.101399503,
            "fwd": 1887.8728029010442,
        },
        0.6292315050984425: {
            "v0": 12.843606661,
            "beta": 0.522501885,
            "alpha": 1.109829483,
            "rho": 0.218575889,
            "fwd": 1909.7955416367697,
        },
        0.8785465738214364: {
            "v0": 12.842446006,
            "beta": 0.54255465,
            "alpha": 0.865249919,
            "rho": 0.314711437,
            "fwd": 1930.943147645677,
        },
    }
    params = pd.DataFrame(params).transpose()
    strikes = np.linspace(
        spot_data["mark_price"] / 20, spot_data["mark_price"] * 5, 200
    )
    expiration_dates = [today + ql.Period(int(365 * x), ql.Days) for x in params.index]
    implied_vols = []
    for tte, row in params.iterrows():
        fwd, v0, beta, alpha, rho = (
            row["fwd"],
            row["v0"],
            row["beta"],
            row["alpha"],
            row["rho"],
        )
        vols = [
            ql.sabrVolatility(strike, fwd, tte, v0, beta, alpha, rho)
            for strike in strikes
        ]
        implied_vols.append(vols)
    implied_vols = ql.Matrix(np.matrix(implied_vols).transpose().tolist())
    print(expiration_dates, strikes, implied_vols)
    local_vol_surface = ql.BlackVarianceSurface(
        today, CALENDAR, expiration_dates, strikes, implied_vols, DAY_COUNTER
    )
    local_vol_surface.setInterpolation("bicubic")
    local_vol_surface.enableExtrapolation()
    return local_vol_surface

if __name__ == "__main__":
    import time

    while True:
        local_vol_surface = create_vol_surface()
        time.sleep(0.1)

[aradhya-variational-io]

One interesting observation, using a ql.Matrix(np.matrix(implied_vols).transpose().tolist()) seems to cause the leak. Using np.matrix(implied_vols).transpose().tolist() seems to be working fine. Basically passing in an instance of a ql.Matrix() seems to be causing the leak 🤔

[lballabio]

Thanks, this helps a lot.

[lballabio]

579 fixes this, it will be in next release. In the meantime, pass a list instead of a ql.Matrix, as I guess you're doing already. Thanks!