fix(polygon-option-data): add new endpoint for polygon options

[endk17]

Description

• This PR has been opened to suppor the specification and reqiuirements as described in https://github.com/QuantConnect/Lean.DataSource.Polygon/issues/12
  • update polygon endpoint to get the snapshot of all options contracts for an underlying ticker.
  • update data model to support updated objects in LEAN syntax with the following data object: open interest, greeks, Implied volatility, last pice + volume

Related Issue

https://github.com/QuantConnect/Lean.DataSource.Polygon/issues/12

Motivation and Context

The original implementation of Polygon Options endpoint did not support parsing of data objects: open interest, greeks, Implied volatility, last pice + volume

Types of changes

• [x] Bug fix (non-breaking change which fixes an issue)
• [x] New feature (non-breaking change which adds functionality)

Checklist:

• [x] My code follows the code style of this project.
• [x] I have read the CONTRIBUTING document.
• [ ] I have added tests to cover my changes.
• [ ] All new and existing tests passed.
• [x] My branch follows the naming convention bug-<issue#>-<description> or feature-<issue#>-<description>

[jhonabreul]

Additional comments:

• The OptionContract model should be completly changed in case this path works, right? They seem to be different. The new endpoint response has the details in the details property, but your models doesn't have that. Same for the OHLC, etc. values.
  • It would be nice to have at least one unit test to make sure the result has all of this new data
• Perhaps the original endpoint can be used and later fetch somehow the open interest for each? I took a quick look and the only one I found was Option Contract Snapshot but after a quick test it seems to only have data for contracts available at the time of the request (real time).

[endk17]

Hi @jhonabreul

Thanks for getting back to me and appreciate the detailed feedback.

Apologies, my initial described technical specification / requirements as noted in https://github.com/QuantConnect/Lean.DataSource.Polygon/issues/12#issuecomment-2088865996 didn't mention that using the new endpoint /v3/snapshot/options/{underlyingAsset} would be for live trading only. Currently, IBKR option datasets (US Equity and Options Add-On Streaming Bundle) are being used for historical and back testing purposes.

Would the below PR changes make sense, from your end of things, for this requirement:

Description:

• Create a new and separate data model for polygon endpoint: /v3/snapshot/options/{underlyingAsset}
• Update the current data model OptionContract model in file OptionContract.cs to a different naming convention
  • Example: OptionContractQuery
• Map and merge the api responses from /v3/reference/options/contracts and /v3/snapshot/options/{underlyingAsset} into a combined data model
  • This would be an updated version of OptionContract model in file OptionContract.cs as to adhere to the current LEAN syntax

Technical (sudo code)

• Create a new data model
  • Example naming convention: OptionContractSnapshot
• Update naming convention in current data model
  • OptionContract ==> OptionContractQuery
• Use PolygonOptionChainProvider : IOptionChainProvider in file PolygonOptionChainProvider.cs to perform two API requests
  • Request 1: /v3/reference/options/contracts
  • Request 2: /v3/snapshot/options/{underlyingAsset}
• When retrieving data from both endpoints:
  • Parse the JSON responses into their respective models.
  • Create an instance of OptionContract and populate it using data from both sources i.e. OptionContractQuery + OptionContractSnapshot
  • Use Ticker and ExpirationDate as keys to ensure data integrity

Implementation / solution [ROUGH DRAFT]

    var request1 = new RestRequest("/v3/reference/options/contracts", Method.GET);
    var request2 = new RestRequest($"/v3/snapshot/options/{underlying.Value}", Method.GET);

    Dictionary<string, CombinedOptionData> combinedOptions = new Dictionary<string, CombinedOptionData>();

    foreach (var option in request1)
    {
        var key = $"{option.Ticker}-{option.ExpirationDate.ToString("yyyy-MM-dd")}";
        combinedOptions[key] = new CombinedOptionData
        {
            Ticker = option.Ticker,
            ContractType = option.Right,
            ExerciseStyle = option.Style,
            ExpirationDate = option.ExpirationDate,
            StrikePrice = (double)option.StrikePrice

        };
    }

    foreach (var snapshot in request2)
    {
        foreach (var result in snapshot.Results)
        {
            var key = $"{result.Details.Ticker}-{result.Details.ExpirationDate.ToString("yyyy-MM-dd")}";
            if (!combinedOptions.TryGetValue(key, out var combinedOption))
            {
                combinedOption = new CombinedOptionData { Ticker = result.Details.Ticker };
                combinedOptions[key] = combinedOption;
            }

            combinedOption.Greeks = result.Greeks;
            combinedOption.LastQuote = result.LastQuote;
            combinedOption.LastTrade = result.LastTrade;
        }
    }

Interested to get your thoughts on this approach

Many thanks Enda

[jhonabreul]

Hi @endk17

The difference between those two endpoints is not just the data model they return. The one being currently used gets the option chain at a given date, that is, all the contracts available for a given underlying at a given date:

• If you get the option chain for SPX for 2023/11/20 you might receive 20k contracts.
• If you request the option chan for SPX again 2023/11/21 or any other date, you will get potentially:
  • +/- contracts
  • A different set of contracts: some might have expired/delisted.

The endpoint you propose to use also gets an option chain (with more data like greeks and open interest) but in real time, that is, all contracts available literally today. The IOptionChainProvider.GetOptionContractList method should be able to get the option chain for a given date, not only for today. And also, that date argument is not the contracts expiration date; it is the point in time at which you want to get the option chain, which might be today or some past date.

There might be a way to keep using the original endpoint and then get the wanted data for each contract, maybe with another endpoint, but again, I did not find one that would allow us to do this.

[ArthurAsenheimer]

Hi @jhonabreul,

I'm working with @endk17 and would like to make a few additions.

Firstly, we don't actually use the method OptionChainProvider.GetOptionContractList in our algorithm. We obtain the options chain data through QCAlgorithm.CurrentSlice.OptionChains.Get(<canonicalOptionSymbol>).

I'm not sure if this makes a difference, or if OptionChainProvider.GetOptionContractList with the parameter time=utcNow is then invoked in the background, but it does show that we only really need it for live trading.

Additionally, Polygon is not officially supported by QC as a data provider for backtesting yet. However, for live trading, Polygon is officially listed as a supported data provider (which is why I think the related GitHub issue should be classified as a bug). Therefore, I believe it makes sense to first ensure that this part of the API (live trading) is functioning flawlessly. For our use case, we only require live trading. For backtesting, we use the default data provider for index options which is Algoseek.

I think PolygonOptionChainProvider.cs can remain unchanged. However, the data on open interest, Greeks, and IV still need to be feeded into the Slice objects, and this is not happening at the moment. Hence the warning

Warning: USA IndexOption OpenInterest data not supported. Please consider reviewing the data providers selection.

As far as I can see, this can only be achieved through the API endpoint /v3/snapshot/options/{underlyingAsset}.

I'm also wondering, when real-time data from Polygon comes in, how are fields/attributes like OptionContract.OpenInterest updated? It seems that this is not currently being done.

I'm not sufficiently familiar with C# and the DataSource plugins of LEAN to implement these changes myself. I believe these adjustments would be better handled by experts like you or other developers from the QC team.

[ArthurAsenheimer]

@jhonabreul Here is a simple algorithm that reproduces the bug:

# region imports
from AlgorithmImports import *
# endregion

class IndexOptionAlgorithm(QCAlgorithm):
    def initialize(self):
        self.set_start_date(2024, 1, 1) 
        self.set_end_date(2024, 5, 1)
        self.set_cash(100000)
        self.spx = self.add_index('SPX') 
        self.spx_option = self.add_index_option(self.spx.symbol) 
        self.spx_option.set_filter(-1, 1, 0, 100)  

    def on_data(self, data: Slice):
        if self.algorithm_mode is not AlgorithmMode.LIVE:
            return 
        chain = data.option_chains.get(self.spx_option.symbol)
        if not chain:
            return 
        for contract in chain: 
            self.log(f"{contract.symbol}| open interest: {contract.open_interest}; delta: {contract.greeks.delta}; implied volatility: {contract.implied_volatility}.") 

When deploying this algorithm for live trading with QC + Polygon as data providers, the following logs are generated:

2024-05-10 17:37:48 Launching analysis for L-3ffd0d19687922fd9e89e48e6bdf8fea with LEAN Engine v2.5.0.0.16423
2024-05-10 17:37:54 Paper Brokerage account base currency: USD
2024-05-10 17:38:08 Warning: usa IndexOption OpenInterest data not supported. Please consider reviewing the data providers selection.
2024-05-10 17:39:00 SPX   240517C05215000| open interest: 0.0; delta: 0.532253383735248; implied volatility: 0.0920031928428969.
2024-05-10 17:39:00 SPX   240517C05220000| open interest: 0.0; delta: 0.505825425148433; implied volatility: 0.0918866720162672.
2024-05-10 17:39:00 SPX   240517P05215000| open interest: 0.0; delta: -0.470339109238199; implied volatility: 0.102137279946327.
2024-05-10 17:39:00 SPX   240517P05220000| open interest: 0.0; delta: -0.49415161757284; implied volatility: 0.101699571257173.
2024-05-10 17:39:00 SPX   240621C05215000| open interest: 0.0; delta: 0.558908042608909; implied volatility: 0.109208822620727.
2024-05-10 17:39:00 SPX   240621C05220000| open interest: 0.0; delta: 0.549011714694276; implied volatility: 0.108904994834179.
2024-05-10 17:39:00 SPX   240621P05215000| open interest: 0.0; delta: -0.442198760403517; implied volatility: 0.112071579867093.
2024-05-10 17:39:00 SPX   240621P05220000| open interest: 0.0; delta: -0.451815241524863; implied volatility: 0.111652211549492.
2024-05-10 17:39:00 SPX   240719C05215000| open interest: 0.0; delta: 0.571491234138189; implied volatility: 0.116239701209309.
2024-05-10 17:39:00 SPX   240719C05220000| open interest: 0.0; delta: 0.564330031525248; implied volatility: 0.115874306439842.
2024-05-10 17:39:00 SPX   240719P05215000| open interest: 0.0; delta: -0.427852489705179; implied volatility: 0.114750769139942.
2024-05-10 17:39:00 SPX   240719P05220000| open interest: 0.0; delta: -0.435078075633236; implied volatility: 0.114323583761875.
2024-05-10 17:39:00 SPX   240816C05220000| open interest: 0.0; delta: 0.575357028689254; implied volatility: 0.121746578619667.
2024-05-10 17:39:00 SPX   240816P05220000| open interest: 0.0; delta: -0.423570866189567; implied volatility: 0.1191776470517.

Here, I notice that it seems to work with delta and IV, and the problem is limited to the open interest.