SIP TradeConditions for Tick behavior appear to not be mapped into any logic within the code.

TightLinesOfCode commented 3 years ago

Expected Behavior

Trade Conditions should be implemented at the system level to ensure proper behavior for Algorithm APIs

Actual Behavior

No logic implemented yet

Potential Solution

A solution level parse behavior mechanism for SIP equity data that is used so that each algorithm doesn't need to implement the logic too.

Reproducing the Problem

Any tick API that includes multiple sale conditions as an array on the request response

System Information

C#, Basic Polygon.IO API

https://polygon.io/glossary/us/stocks/conditions-indicators

Checklist

[x] I have completely filled out this template
[x] I have confirmed that this issue exists on the current master branch
[ ] I have confirmed that this is not a duplicate issue by searching issues
[ ] I have provided detailed steps to reproduce the issue

TightLinesOfCode commented 3 years ago

Updated Template

Martin-Molinero commented 3 years ago

Hey @TightLinesOfCode! Tick type have a SaleCondition field that should have this information, in the case of the PolygonDataQueueHandler it seems it's not being set (TODO: bug). Lean already implements an IDataFilter abstraction, please see TickDataFilteringAlgorithm as an example. This filter could look at tick sale conditions and ignore any undesired tick. TODO: we should have a regression algorithm for this too & live unit test. Is this what you were looking for?

TightLinesOfCode commented 3 years ago

Hi Martin, I am working on rewriting the PolygonData Downloaders so they work. But the SaleConditions from the PolygonData are more than one (i.e. int[14,22,3] and the Tick : BaseData class assumes a single SaleCondition. So I am somewhat concerned about the overall data structure and architecture for your other data sources too.

During backtesting on the cloud instance, which sale conditions are you using for other data sets ?

I am fixing the PolygonDataQueueHandler, but it might be nice to make this fix in the overall API to give logic that will help the other quant developers.

Do you have any set logic that you are already using ?

Should this raw sale condition data from Polygon be mapped to the QuantConnect Data Model for Ticks ?

Is there code anywhere that deals with the sale conditions ?

Another question about the tick data model ... I am looking at the readme.md file and it shows a data model that uses an exchange code. (i.e. "P") However, when I look at the data from one of the examples, the exchange is recorded in the tick data by name ( i.e. NASDAQ ).

Is there a documentation anywhere which specifies the exact data models for everything ?

Martin-Molinero commented 3 years ago

I am working on rewriting the PolygonData Downloaders so they work. But the SaleConditions from the PolygonData are more than one (i.e. int[14,22,3] and the Tick : BaseData class assumes a single SaleCondition. So I am somewhat concerned about the overall data structure and architecture for your other data sources too.

Thanks for digging into it! 💯 The sale conditions could be encoded, you can see the Tick.ParsedSaleCondition that supposes SaleCondition to be a hex format number.

Should this raw sale condition data from Polygon be mapped to the QuantConnect Data Model for Ticks ?

I'd believe it would be Polygon specific

Another question about the tick data model ... I am looking at the readme.md file and it shows a data model that uses an exchange code. (i.e. "P") However, when I look at the data from one of the examples, the exchange is recorded in the tick data by name ( i.e. NASDAQ ).

hm good point 👍 they should follow the same normalized, maybe https://github.com/QuantConnect/Lean/blob/master/Common/Global.cs#L654 TODO

TightLinesOfCode commented 3 years ago

@Martin-Molinero I have already written 90% of the code needed, but really need to get an understanding from you regarding how your cloud system currently handles tick data. Are you filtering out the required sale conditions ? Or is the vendor doing that for you.

Most ticks have multiple sale conditions, but I don't see any code in the open source solution regarding logic relative to the sale condition.

Also, is your vendor flagging the tick as suspicious ? Or is that logic in your code.

I plan on writing a lot of code for this open source project, so really want to align my efforts with how you guys are already handling things. I'm first going to equity data for Polygon, but when I see things that aren't consistent with the overall architecture, I want to fix that too.

TightLinesOfCode commented 3 years ago

@Martin-Molinero Please give me guidance. Can you help me decide where to add the code.

The quant quote specification specifies multiple sale condition codes in their string. https://quantquote.com/docs/TickView_Historical_Trades.pdf

So what logic are you currently using in the cloud back testing to map to the SIP logic for updating high/low, trade bars, etc.

I can write the code for the logic, but wanted to make sure that I using the same logic that you are already using. I don't see this in the open source project. But maybe I missed it.

Martin-Molinero commented 3 years ago

Hey @TightLinesOfCode,

The quant quote specification specifies multiple sale condition codes in their string. https://quantquote.com/docs/TickView_Historical_Trades.pdf

The sale conditions could be encoded, you can see the Tick.ParsedSaleCondition that supposes SaleCondition to be a hex format number. Multiple sale conditions can live together.

So what logic are you currently using in the cloud back testing to map to the SIP logic for updating high/low, trade bars, etc. I can write the code for the logic, but wanted to make sure that I using the same logic that you are already using. I don't see this in the open source project. But maybe I missed it.

The IDataQueueHandler implementations are internally using the AggregationManager, which will filter out suspicious ticks out of aggregation https://github.com/QuantConnect/Lean/blob/master/Engine/DataFeeds/AggregationManager.cs#L144 for bigger resolution bars. As JB mentioned here https://github.com/QuantConnect/Lean/issues/4954#issuecomment-740082992, ticks don't/shouldn't get filtered. So the implementation can internally flag as suspicious the ticks accordingly and they won't be used for aggregation, no other change should be required. Hope it helps answer your concerns!

TightLinesOfCode commented 3 years ago

Hi Martin,

Not really. My challenge is that the tick class is using an integer and assumes that each Trade Tick for Equity has a single sale condition.

But in actuality, there are multiple sale conditions for each trade. Please advise on how you want to me go about fixing this bug system wide.

I am concerned about whether this is intentional or not ? So for example, your quant quote data example also does not seem to implement correct selection using the SIP standard specification.

Please advise on you want me to proceed with fixing this issue.

Thanks,

John

From: Martin-Molinero [mailto:notifications@github.com] Sent: Wednesday, December 16, 2020 9:25 AM To: QuantConnect/Lean Lean@noreply.github.com Cc: TightLinesOfCode jdalmon316@gmail.com; Mention mention@noreply.github.com Subject: Re: [QuantConnect/Lean] SIP TradeConditions for Tick behavior appear to not be mapped into any logic within the code. (#4965)

Hey @TightLinesOfCode https://github.com/TightLinesOfCode ,

The quant quote specification specifies multiple sale condition codes in their string. https://quantquote.com/docs/TickView_Historical_Trades.pdf

The sale conditions could be encoded, you can see the Tick.ParsedSaleCondition that supposes SaleCondition to be a hex format number. Multiple sale conditions can live together.

So what logic are you currently using in the cloud back testing to map to the SIP logic for updating high/low, trade bars, etc. I can write the code for the logic, but wanted to make sure that I using the same logic that you are already using. I don't see this in the open source project. But maybe I missed it.

The IDataQueueHandler implementations are internally using the AggregationManager, which will filter out suspicious ticks out of aggregation https://github.com/QuantConnect/Lean/blob/master/Engine/DataFeeds/AggregationManager.cs#L144 for bigger resolution bars. As JB mentioned here #4954 (comment) https://github.com/QuantConnect/Lean/issues/4954#issuecomment-740082992 , ticks don't/shouldn't get filtered. So the implementation can internally flag as suspicious the ticks accordingly and they won't be used for aggregation, no other change should be required. Hope it helps answer your concerns!

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/QuantConnect/Lean/issues/4965#issuecomment-746677886 , or unsubscribe https://github.com/notifications/unsubscribe-auth/ABMVZWXTZAPMAVFMS37YP4TSVDUQJANCNFSM4UCTQXXQ . https://github.com/notifications/beacon/ABMVZWRA5YXHFVRIHLUH6VDSVDUQJA5CNFSM4UCTQXX2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOFSAWM7Q.gif

-- This email has been checked for viruses by AVG. https://www.avg.com

TightLinesOfCode commented 3 years ago

Martin,

Thanks for helping, sorry for so many questions. Where is the logic that handles aggregate rules to follow the SIP specifications for updating high/low, etc ?

See. https://polygon.io/glossary/us/stocks/conditions-indicators

It’s NOT just suspicious ticks that need to be filtered for aggregation.

Also, is there any code for replicating the suspicious ticks ? Or will I need to create a new algorithm for use with any tick data provider ?

Thanks for guidance.

John

From: Martin-Molinero [mailto:notifications@github.com] Sent: Wednesday, December 16, 2020 9:25 AM To: QuantConnect/Lean Lean@noreply.github.com Cc: TightLinesOfCode jdalmon316@gmail.com; Mention mention@noreply.github.com Subject: Re: [QuantConnect/Lean] SIP TradeConditions for Tick behavior appear to not be mapped into any logic within the code. (#4965)

Hey @TightLinesOfCode https://github.com/TightLinesOfCode ,

The quant quote specification specifies multiple sale condition codes in their string. https://quantquote.com/docs/TickView_Historical_Trades.pdf

The sale conditions could be encoded, you can see the Tick.ParsedSaleCondition that supposes SaleCondition to be a hex format number. Multiple sale conditions can live together.

So what logic are you currently using in the cloud back testing to map to the SIP logic for updating high/low, trade bars, etc. I can write the code for the logic, but wanted to make sure that I using the same logic that you are already using. I don't see this in the open source project. But maybe I missed it.

The IDataQueueHandler implementations are internally using the AggregationManager, which will filter out suspicious ticks out of aggregation https://github.com/QuantConnect/Lean/blob/master/Engine/DataFeeds/AggregationManager.cs#L144 for bigger resolution bars. As JB mentioned here #4954 (comment) https://github.com/QuantConnect/Lean/issues/4954#issuecomment-740082992 , ticks don't/shouldn't get filtered. So the implementation can internally flag as suspicious the ticks accordingly and they won't be used for aggregation, no other change should be required. Hope it helps answer your concerns!

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/QuantConnect/Lean/issues/4965#issuecomment-746677886 , or unsubscribe https://github.com/notifications/unsubscribe-auth/ABMVZWXTZAPMAVFMS37YP4TSVDUQJANCNFSM4UCTQXXQ . https://github.com/notifications/beacon/ABMVZWRA5YXHFVRIHLUH6VDSVDUQJA5CNFSM4UCTQXX2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOFSAWM7Q.gif

-- This email has been checked for viruses by AVG. https://www.avg.com

QuantConnect / Lean