Can you put the example data somewhere for download, so one can run the example more easily and quickly?

nkaz001 / hftbacktest

A high-frequency trading and market-making backtesting tool in Python and Rust, which accounts for limit orders, queue positions, and latencies, utilizing full tick data for trades and order books, with real-world crypto market-making examples for Binance Futures

MIT License

1.75k stars 347 forks source link

Can you put the example data somewhere for download, so one can run the example more easily and quickly? #69

Closed cutebomb closed 4 weeks ago

cutebomb commented 6 months ago

the data in the examples like 'data/ethusdt_20221003.npz', 'data/ethusdt_20221004.npz', 'data/ethusdt_20221005.npz', 'data/ethusdt_20221006.npz', 'data/ethusdt_20221007.npz'

nkaz001 commented 6 months ago

Do you know a good platform for uploading large amounts of data for free? Initially, I uploaded the sample data on GitHub LFS, but it quickly reached the bandwidth limit.

cutebomb commented 5 months ago

kaggle datasets may be one option. or other cloud filesystem like google file

nkaz001 commented 5 months ago

i'll check it out. since i need to collect data newly, it will take time.

SUSHANTH009 commented 5 months ago

@nkaz001 can you please please upload this data 'data/ethusdt_20221003.npz', 'data/ethusdt_20221004.npz', 'data/ethusdt_20221005.npz', 'data/ethusdt_20221006.npz', 'data/ethusdt_20221007.npz' like maye be on kaggle

nkaz001 commented 5 months ago

i'll set up a new data collection to avoid any licensing issues and provide the sample data publicly. But, it also costs me to run a server in the cloud, so please wait for a moment. or you can directly collect the data yourself using https://github.com/nkaz001/collect-binancefutures.

nkaz001 commented 5 months ago

I began collecting the feed and may provide sample data which will be about for a week. Once the collection and preparation of the sample data are complete, I will attempt to upload it and update the demo accordingly.

qtbgo commented 4 months ago

Great, are you done?

nkaz001 commented 4 months ago

Still collecting it. need to select pairs for upload. already now it's ~hundreds of gb. maybe I'll try to upload it next week?

qtbgo commented 4 months ago

wow

nkaz001 commented 4 months ago

i couldn't create a public dataset.

nkaz001 commented 4 months ago

Meanwhile, I've added a Rust version of the data collector for Binance Futures USDM. It's a quick way to gather the data and test it. https://github.com/nkaz001/collect-binancefutures/tree/master/rust

huxianguang commented 2 months ago

can you please please upload this data 'data/ethusdt_20221003.npz', 'data/ethusdt_20221004.npz', 'data/ethusdt_20221005.npz', 'data/ethusdt_20221006.npz', 'data/ethusdt_20221007.npz'

nkaz001 commented 2 months ago

ue to the size of the data files, it is difficult to find a place to upload them. Instead, I have added an example showing the complete process of backtesting, including downloading the data from Tardis and converting it. This would be the quickest way unless you collect the data yourself using collect-data.

kronael commented 2 months ago

I'm impressed with the work you're doing @nkaz001 . I'd be happy to host some data for you if it would help the community. I guess I'll be able to find a few TB for it. Also, would possibly be able to thin it without impacting the backtests much. Please, feel free to reach out with the details if you're interested, on Telegram @kronael .

nkaz001 commented 2 months ago

Good to hear that. Thanks for the suggestion. By the way, do you have Discord, or can you let me know your email? Or, please get in touch with me through Discord nkaz001 or email nkaz001@protonmail.com

nkaz001 commented 1 month ago

@kronael provides a huge resource. You can freely download the data from the link below. There are certain bandwidth limits and please do not abuse it. I appreciate this and it will be added to the official documentation soon.

https://reach.stratosphere.capital/data/usdm/