DollarNeutral portfolios are not fully invested

[thayes75]

Specifications

• Problem: Dollar Neutral constraint does not produce a fully invested portfolio
• OS: Linux
• Cvxportfolio version: 1. 3.2
• Python version: 3.9
• Pandas version: 2.2.0
• Data: Online

Description

The main problem I'm trying to solve for is a fully invested Long/Short, or 130/30 portfolio. I've tried several iterations employing the User Provided Forecasters example as a playground. Some potential implementations strategies I've tried include:

• Setting a Long Notional Target (see example constraint below) coupled with a Leverage Constraint
• Setting a Cash Target Constraint (e.g., max 50bps) which is infeasible (see example constraint below)
• NoCash and DollarNeutral combined are infeasible.
• MarketNeutral gets closer but still not ideal

Indeed, most of the example plots always seem to have significant USDOLLAR holdings. In reading section 2.1 of the paper, the last paragraph mentions what I'd be trying to accomplish but I seem to be missing something on how the Cash Account is being used/managed.

How to reproduce

import matplotlib.pyplot as plt

if __name__ == '__main__':

    import cvxportfolio as cvx

    # Here we define a class to forecast expected returns
    # There is no need to inherit from a base class, in this simple case
    class WindowMeanReturns: # pylint: disable=too-few-public-methods
        """Expected return as mean of recent window of past returns.

        This is only meant as an example of how to define a custom forecaster;
        it is not very interesting. Since version ``1.2.0`` a similar
        functionality has been included in the default forecasters classes.

        :param window: Window used for the mean returns.
        :type window: int
        """

        def __init__(self, window=20):
            self.window = window

        def values_in_time(self, past_returns, **kwargs):
            """This method computes the quantity of interest.

            It has many arguments, we only need to use ``past_returns`` in this
            case.

            :param past_returns: Historical market returns for all assets in
                the current trading universe, up to each time at which the
                policy is evaluated.
            :type past_returns: pd.DataFrame
            :param kwargs: Other, unused, arguments to :meth:`values_in_time`.
            :type kwargs: dict

            :returns: Estimated mean returns.
            :rtype: pd.Series

            .. note::

                The last column of ``past_returns`` are the cash returns.
                You need to explicitely skip them otherwise Cvxportfolio will
                throw an error.
            """
            return past_returns.iloc[-self.window:, :-1].mean()

    # Here we define a class to forecast covariances
    # There is no need to inherit from a base class, in this simple case
    class WindowCovariance: # pylint: disable=too-few-public-methods
        """Covariance computed on recent window of past returns.

        This is only meant as an example of how to define a custom forecaster;
        it is not very interesting. Since version ``1.2.0`` a similar
        functionality has been included in the default forecasters classes.

        :param window: Window used for the covariance computation.
        :type window: int
        """

        def __init__(self, window=20):
            self.window = window

        def values_in_time(self, past_returns, **kwargs):
            """This method computes the quantity of interest.

            It has many arguments, we only need to use ``past_returns`` in this
            case.

            :param past_returns: Historical market returns for all assets in
                the current trading universe, up to each time at which the
                policy is evaluated.
            :type past_returns: pd.DataFrame
            :param kwargs: Other, unused, arguments to :meth:`values_in_time`.
            :type kwargs: dict

            :returns: Estimated covariance.
            :rtype: pd.DataFrame

            .. note::

                The last column of ``past_returns`` are the cash returns.
                You need to explicitely skip them otherwise Cvxportfolio will
                throw an error.
            """
            return past_returns.iloc[-self.window:, :-1].cov()

    # define the hyper-parameters
    WINDOWMU = 252
    WINDOWSIGMA = 252
    GAMMA_RISK = 5
    GAMMA_TRADE = 3

    # define the forecasters
    mean_return_forecaster = WindowMeanReturns(WINDOWMU)
    covariance_forecaster = WindowCovariance(WINDOWSIGMA)

    # define the policy
    policy = cvx.SinglePeriodOptimization(
        objective = cvx.ReturnsForecast(r_hat = mean_return_forecaster)
            - GAMMA_RISK * cvx.FullCovariance(Sigma = covariance_forecaster)
            - GAMMA_TRADE * cvx.StocksTransactionCost(),
            constraints = [cvx.DollarNeutral(), cvx.LeverageLimit(1)]
        )

    # define the simulator
    simulator = cvx.StockMarketSimulator(['AAPL', 'GOOG', 'MSFT', 'AMZN'])

    # back-test
    result = simulator.backtest(policy, start_time='2020-01-01',end_time='2024-09-04')

    # Check to see if fully invested 
    df_hld = result.h.tail(n=1)
    df_csh = df_hld[df_hld.columns.values[-1]]
    df_hld = df_hld[df_hld.columns.values[:-1].tolist()]

    print( 'Cash Value at end period : ' + str(int(df_csh.values[0]))  )    
    print( 'Value of Short holdings  : ' + str(int(df_hld[df_hld<=0].sum(axis=1).values[0])) )
    print( 'Value of Long holdings   :  ' + str(int(df_hld[df_hld>0].sum(axis=1).values[0])) )    

    # show the result
    print(result)
    figure = result.plot()
    plt.show()

Additional information

Some example constraints I've tried to custom build (but don't work) include:

LongNotionalTarget Custom Constraint This one is getting the sign wrong but I provide it to help show my thought process.

class LongNotionalTarget(InequalityConstraint):
    r"""LongNotionalTarget allows user to specify the notional value
    of the long side of the portfolio.

    This is useful to provide the ability to specify the investment 
    of dollars to ensure it is deployed:

    .. math::
        {(h_plus_t)}_{1:n} \leq target_long_ntl.

    :param target_ln: We require that the Long Side (ex cash) notional value 
    (in dollars) is less than or equal to this. This is either 
    constant, expressed as :class:`float`, or changing in time, expressed as a
        :class:`pd.Series` with datetime index.
    :type target_ln: float or pd.Series
    """

    def __init__(self, target_ln ):
        self.target_ln = DataEstimator(target_ln, compile_parameter=True)

    def initialize_estimator( # pylint: disable=arguments-differ
            self, universe, **kwargs):
        """Initialize parameter with size of universe.

        :param universe: Trading universe, including cash.
        :type universe: pandas.Index
        :param kwargs: Other unused arguments to :meth:`initialize_estimator`.
        :type kwargs: dict
        """
        self._stock_holdings = cp.Variable(len(universe)-1)

    def values_in_time( # pylint: disable=arguments-differ
            self, current_weights, current_prices, **kwargs):
        """Update parameter with current market weights and covariance.

        :param past_volumes: Past market volumes, in units of value.
        :type past_volumes: pandas.DataFrame
        :param kwargs: Unused arguments passed to :meth:`values_in_time`.
        :type kwargs: dict
        """
        self._stock_holdings = cp.multiply(current_weights[:-1],current_prices) 

    def _compile_constr_to_cvxpy( # pylint: disable=arguments-differ
            self, **kwargs):
        """Compile left hand side of the constraint expression."""
        return cp.sum(cp.pos(self._stock_holdings))

    def _rhs(self):
        """Compile right hand side of the constraint expression."""
        return self.target_ln.parameter    

CashTarget Custom Constraint As you probably guessed already, this is infeasible.

class CashTarget(InequalityConstraint):
    r"""CashTarget allows user to specify the max cash target in weight space.

    This is useful to provide the ability to specify the investment 
    of dollars to ensure it is deployed:

    .. math::
        {(w_plus)}_{n+1} \leq ct.

    :param cash_target: We require that the cash target weight value 
    to beless than or equal to this. This is either 
    constant, expressed as :class:`float`, or changing in time, expressed as a
        :class:`pd.Series` with datetime index.
    :type cash_target: float or pd.Series
    """

    def __init__(self, cash_target ):
        self.cash_target = DataEstimator(cash_target, compile_parameter=True)

    def _compile_constr_to_cvxpy( # pylint: disable=arguments-differ
            self, w_plus, **kwargs):
        """Compile left hand side of the constraint expression."""
        return w_plus[-1]

    def _rhs(self):
        """Compile right hand side of the constraint expression."""
        return self.cash_target.parameter    

Output

Cash Value at end period : 896851
Value of Short holdings  : -85425
Value of Long holdings   :  86049

#################################################################
Universe size                                                   5
Initial timestamp                       2020-01-02 14:30:00+00:00
Final timestamp                         2024-09-03 13:30:00+00:00
Number of periods                                            1175
Initial value (USDOLLAR)                                1.000e+06
Final value (USDOLLAR)                                  8.975e+05
Profit (USDOLLAR)                                      -1.025e+05

Avg. return (annualized)                                    -1.5%
Volatility (annualized)                                     12.5%
Avg. excess return (annualized)                             -3.8%
Avg. active return (annualized)                             -3.8%
Excess volatility (annualized)                              12.5%
Active volatility (annualized)                              12.5%

Avg. growth rate (annualized)                               -2.3%
Avg. excess growth rate (annualized)                        -4.6%
Avg. active growth rate (annualized)                        -4.6%

Avg. StocksTransactionCost                                    0bp
Max. StocksTransactionCost                                    1bp
Avg. StocksHoldingCost                                        1bp
Max. StocksHoldingCost                                        3bp

Sharpe ratio                                                -0.30
Information ratio                                           -0.30

Avg. drawdown                                              -12.2%
Min. drawdown                                              -18.3%
Avg. leverage                                               91.6%
Max. leverage                                              110.7%
Avg. turnover                                                1.0%
Max. turnover                                               26.4%

Avg. policy time                                           0.015s
Avg. simulator time                                        0.012s
    Of which: market data                                  0.003s
    Of which: result                                       0.001s
Total time                                                31.882s
#################################################################

[enzbus]

Just a note, before I read in full. A dollar neutral portfolio has always cash position equal to 1. It is fully invested if its leverage is greater or equal than one. I know that cash accounting in Cvxportfolio is different than other portfolio optimization tool, we have developed a more accurate way to account for it in my opinion (read the self-financing condition in the paper).

[thayes75]

Understood. I will look more into the accounting to see if I can get a better handle on it. The main problem is ensuring I've deployed, for example, 1MM long and 1MM short as a target outcome from a practical perspective.

[enzbus]

So, if you want a 130/30 portfolio: cash should be 0 (use NoCash, not DollarNeutral). Leverage should be 1.6, so you fix that in LeverageLimit. I guess that's enough? That will give you a portfolio with 1.3 of capital employed in long holdings, and 0.3 in short holdings (assuming the leverage limit is hit, but you can make sure you do with appropriate risk penalization).

[enzbus]

Other comments on your code: CashTarget looks correct, you can use EqualityConstraint if you want, but in any case it conflicts with DollarNeutral, which requires the cash weight to be always equal to 1. The long notional limit is correct conceptually, there's a mistake (you'd need to use a CVXPY parameter and set its value attribute at each iteration). I think you can use direct CVXPY multiply function so no need to create internal parameter. But see my comment above, I don't think you need that if I understand what you're trying to do.

[thayes75]

Appreciate the reply. Let me try it out and I'll revert back later today with the results

[thayes75]

Just about to reply with little success. #181 looks like it might help. Happy to close this issue while the other one is open.

[thayes75]

Forgot to close