search

crypto-chassis / ccapi

A header-only C++ library for interacting with crypto exchanges. Bindings for Python, Java, C#, Go, and Javascript are provided.
https://discord.gg/b5EKcp9s8T
MIT License
568 stars 196 forks source link

Enhancement idea: streaming historical data in ccapi_cpp #54

Closed woonsangcho closed 3 years ago

woonsangcho commented 3 years ago

Currently, historical data *(1-second book snapshots + trades + OHLC) are provided from REST API https://github.com/crypto-chassis/cryptochassis-api-docs. This ccapi_cpp repo is for streaming live data and execution.

Enhancement: streaming coarse historical data through the same ccapi_cpp repo

This enhancement benefits the training pipeline:

Easier features generation for training models, through three events * streamed chronologically.

Without much changes in code, the user simply 'switch on' live mode and stream the same custom features in real-time.

The user can have confidence that it's the same set of features that the model was trained upon.

From a design perspective, it could make auto-download daily * event files to user's local tmp folder and read from there. It also requires implementing subscriptions to OHLC events https://github.com/crypto-chassis/cryptochassis-api-docs#ohlc in the current ccapi_cpp repo.

On your roadmap, do you have plans for such an enhancement? Please let me know how I can contribute if this is something you have in mind.

cryptochassis commented 3 years ago

Hey, @woonsangcho, thank you very much for the suggestions. First of all, we've opened a pull request (https://github.com/crypto-chassis/ccapi_cpp/pull/55) to address the OHLC data collection problem that you mentioned above. Because quite some exchanges don't provide OHLC streams, we've taken a slightly different approach: calculate OHLC from the trade streams. Back to the central question about "streaming coarse historical data through the same ccapi_cpp repo" (which in fact has been one of the existing tickets on our internal project management board for while), I've performed extensive research in the past week and would like to propose the following: it seems to be better to establish a new repository for that purpose. The reasons are: a. We try to make code and data to be decoupled which would be more flexible to satisfy a plethora of different end user needs (e.g. machine learning gurus, technical indicator veterans, market making tycoons...). b. The enhancement itself resembles a backtesting engine (or replay engine, or trade simulation engine...), the scope of which deserves and is better suited to a standalone project. c. We've always strived to be the best data provider (https://github.com/crypto-chassis/cryptochassis-api-docs), there are certainly other good data providers on the market (e.g. Kaiko) which covers more exchanges and trading pairs. We think multi-approach (and leaving that choice to end users) creates a healthy environment and would like to give the end users the flexibility to "plug in" other data providers' data. Therefore a related but independent repository seems to be a perfect candidate. If you are interested in starting such a project, that's something awesome: in fact your "simulator" can be a completely independent project with different streaming / data providers as "connectors", and we'd be happy to code our "connect" in your project. In fact, looking at the open source community, we indeed haven't found any good simulator yet, thus the project can be significant.

woonsangcho commented 3 years ago

Hello @cryptochassis, thank you for the response. I agree with what you have written above. What I proposed goes beyond the scope of the nature of this open source repo for its longevity, and this idea deserves a standalone project. Speaking of which, as much as I would like to start such a open source project, in the long term unfortunately it would be difficult for me to commit to maintaining it for future conflict of interest. I appreciate your willingness to help - I think we can table this aside in the roadmap. Thank you for the great contribution to the open source community and playing with the repo I really like what you have built!

cryptochassis commented 3 years ago

@woonsangcho If you or your firm/institution has any more feature request, feel free to let us know. We are very open-minded and try to be as thorough as possible to satisfy the need of end users. Thank you for your interest in us. 👍