OpenInterestFutureUniverseSelectionModel - Failed to get historical open interest

DerekMelchin commented 8 months ago

Expected Behavior

OpenInterestFutureUniverseSelectionModel works w/o error

Actual Behavior

self.AddUniverseSelection(
    OpenInterestFutureUniverseSelectionModel(
        self, 
        lambda utc_time: [Symbol.Create(Futures.Indices.SP500EMini, SecurityType.Future, Market.CME)]
    )
)

works in Python https://www.quantconnect.com/terminal/processCache?request=embedded_backtest_be173aeb8eab0ab765cd6c13b5e6ebb8.html

But the C# equivalent

AddUniverseSelection(
    new OpenInterestFutureUniverseSelectionModel(
        this, 
        utcTime => new[] { QuantConnect.Symbol.Create(Futures.Indices.SP500EMini, SecurityType.Future, Market.CME) }
    )
);

throws

Backtest Handled Error: OpenInterestFutureUniverseSelectionModel.FilterByOpenInterest: Failed to get historical open interest, no symbol will be selected.

Potential Solution

N/A

Reproducing the Problem

namespace QuantConnect.Algorithm.CSharp
{
    public class LogicalRedOrangeDog : QCAlgorithm
    {

        public override void Initialize()
        {
            SetStartDate(2022, 7, 18);
            SetCash(100000);

            AddUniverseSelection(
                new OpenInterestFutureUniverseSelectionModel(
                    this, 
                    utcTime => new[] { QuantConnect.Symbol.Create(Futures.Indices.SP500EMini, SecurityType.Future, Market.CME) }
                )
            );
        }

        /// OnData event is the primary entry point for your algorithm. Each new data point will be pumped in here.
        /// Slice object keyed by symbol containing the stock data
        public override void OnData(Slice data)
        {
        }

    }
}

System Information

QC Cloud

Checklist

[x] I have completely filled out this template
[x] I have confirmed that this issue exists on the current master branch
[x] I have confirmed that this is not a duplicate issue by searching issues
[x] I have provided detailed steps to reproduce the issue

DerekMelchin commented 8 months ago

The OpenInterestFutureUniverseSelectionModel constructor also doesn't accept a UniverseSettings object like all the other universe selection models do

Marinovsky commented 8 months ago

After debugging the C# version of this algo in the cloud, I found it fails when the current time is February 27th of 2023.

When the OpenInterestFutureUniverseSelectionModel.GetOpenInterest() is executed, the date 2023/2/27 is assigned to the variable current as well as endTime. To compute the value of the variable previousDay, the method Time.GetStartTimeForTradeBars() is called. This method "rounds down" the value of the variable endTime by first converting it to the UTC timezone (adding 5 hours), then rounding down that date (removes the five hours added) and finally converting it again to the New York Timezone (removes 5 hours). Therefore, since the initial value of endTime was 2023/2/27 00:00:00, the extension method RoundDownInTimeZone() returns 2023/2/26 19:00:00. Then, this date is subtracted by barSize (1 day) so the final date returned by the Time.GetStartTimeForTradeBars() method is 2023/2/25 19:00:00.

Thus, the history request for open interest data is from the 25th to the 27th and since we are missing open interest data for the 26th, we obtain no open interest data and the algo fails.

I've run both versions (Python and C#) of the algo attached in this issue and seen them fail (throw the error mentioned in this algo).

QuantConnect / Lean