Closed stucash closed 1 year ago
Hi stucash, I don't know your data provider, so I can't give detailed feedback. Depending on your problem at hand, it might be better to focus on only one exchange anyway. But your mileage may vary depending on your application.
@kboudt: I think you can close this issue because this particular question of data cleaning is more related to how @stucash's data is constructed.
@onnokleen Thanks for reverting back to me; are you referring to Polygon.io, as the data provider you don't know? Securities Information Processor (SIP) is the data distributor used across the US markets (or exchanges). A list of exchanges that are using SIP can be found here: https://www.ctaplan.com/index
And Polygon.io gives SIP data directly to the user.
I understand that eventually, I'll have to settle on one exchange but the problem for me is, that I don't know which exchange sequence to pick (AskExchange, or BidExchange).
I read the paper dedicated to highfrequency
as well and I can see the data used was from Wharton Research Data Service(WRDS); are we saying that the tick data from WRDS should resemble the raw tick data from exchange (and it is SIP data as well)?
Could you share some reliable data sources that have an identical data structure with WRDS tick data? Or is there a heuristic I could use to just pick the exchange for quotes data? At the end of day, what I wanted to do was to use the package to run analysis in a reliable way and if I can't clean the data properly, it's probably a no-go :(.
Besides the exchange question, I realized that the sampleTDataRaw
has a column called CORR
, which, according to your paper, is the correlation indicator (which looks to have a value of 0/1). Correct me if I am wrong, but this column is fairly unlikely to show up in many data feeds' raw data; my gut feeling is that we have to generate this column ourselves?
Thanks a lot!
CORR stands for corrected trades. Hence, if you want to include all trades, then just add a CORR column with ones in it.
@kboudt Can you close this issue?
I was trying to clean up the quotes data requested from a SIP source (e.g., Polygon.io: https://polygon.io/docs/stocks/get_v3_quotes__stockticker), which is directly from the exchange.
I found that the package took in EX (exchange) and SIP source gave me AskExchange and BidExchange; this is going to cause trouble during clean-up.
sampleQDataRaw:
My sample SIP data:
Please ignore the irrelevant columns, the whole purpose is to demonstrate this exchange thing.
Am I missing something before approaching the clean-up? Or it actually doesn't really matter which one to pick, be it AskExchange, or BidExchange, in the case of SIP Data? The question here, for me, is that the AskExchange and BidExchange are not identical, it changes what remains after the clean-up.
Thanks for your help Team!