search

Celeborn2BeAlive / bitmex-xbtusd-history

OHLCV history of XBTUSD bitmex contract, with python scripts to update it.
20 stars 9 forks source link

Why This Way of working? #2

Closed Jiraiyah closed 5 years ago

Jiraiyah commented 5 years ago

@Celeborn2BeAlive Hi First of all, you don't know how much you saved me on time and effort by this repository. I really appreciate your job. But, looking at the json file the very first time, I was thinking like, WHY !? I mean, if you would simply export candle info to csv files, that would be more than enough ! honestly, you didn't save that much space by serializing the data this way ! Another question I asked myself : WHY hard code the pair in between the lines? I mean, if I want to, I can easily look into the code and change XBTUSD to ETHUSD, but, wouldn't it be better to some how produce a way for use to identify this? then we could use the same code for another market.

Now, to my next question and actually it is a feature request. Expend the code to work with Binance side by side bitmex please.

Celeborn2BeAlive commented 5 years ago

Hi @Jiraiyah , I'm glad it helped you :) There is no logical reason for me to have exported data to json rather than CSV, just the fact that I like json more haha. Space is not an issue anyway, there is not so much data in these files. To get CSV you can use the script "csv_convert.py". Also for processing and backtesting I think the best is to first convert the json files to big binary files, it's faster to load after.

I know I could have made the trading pair configurable from command line but I was only interested in XBTUSD at the time.

I will soon work on a portofolio rebalance project for Binance so I will do the same kind of repository with binance pairs (and make them configurable from command line).

Thank you for your feed back :)

Jiraiyah commented 5 years ago

oh didn't noticed the csv converter. nice. well, if you are doing python, chances are you will think about stuff like tensor flow ;) in that case, csv will be a big winner with python, specialy with ta lib for calculating some really useful indicator/oscillator values. I may fork that repo from github and push it forward as much as i can because the base is really there. although his obv calculation is wrong but that whouldn't be hard to fix. that is why i said you don't know how much you save people like on time and effort.

honestly i'm new in python, one of the real down falls on bitmex is not having 15 minute charts, if you are familiar enough with python and would have time for this, keep this as a possible feature request, 15 minute and 2 and 3 and 4 and 6 hour charts are really nice to have specially if you want to push things toward algorithmic trading ;) and one more thing, not sure if it's possible or not, but having a script register itself as a service on the os (as i said not sure if it's possible on python) and continuously get the info, then we could put such a thing on a server and always have a fresh info. specially if you can provide the extra candle sticks, that would be something i call complete (considering that weekly data are small to be really useful other than looking at them for this week related to the previous one, else, add weekly and monthly to future todo list ;) )

again, your job helps many people i think. sadly, most people are lazy enough to put this little time and give feedback and thank you for what you do, just remember they exist and really appreciate it.

Celeborn2BeAlive commented 5 years ago

@Jiraiyah Well I was not aware it helps so much ^^' I'm just scrapping bitmex data and putting it on a repository. It was mainly for me that's why the scripts are not really polished and configurable.

About other timeframes: you can easily compute them by taking 1min timeframe (for example for 15min tf: you regroup 1m candles 15 by 15, then the first candle of a group gives you the open price, the last gives you the close price, the high can be obtained by taking the max() among all 15 high prices, and the low can be obtained by taking the min() among all 15 low prices). That is why it is not useful to store these timesframes on disk, you can recompute them. M y script gets all timeframes that bitmex gives but I could have drop 1d, 1h and 5m and just reconstruct them from 1m data. However, 1m data does not go as far as others. 1m data starts at 2016-12-31 while daily data starts at 2015-09-01.

I already have a server running the script periodically. Its an archlinux with a systemd timer script. I can put the script in the repository if you want it. All commits you see on https://github.com/Celeborn2BeAlive/bitmex-xbtusd-history/commits/master are made by this script every hour.

Celeborn2BeAlive commented 5 years ago

@Jiraiyah I've started a new repository https://github.com/Celeborn2BeAlive/cryptobigbro with a script for fetching OHLCV from multiple exchanges and multiple instruments. Also it saves data in CSV files instead of json. You were right, CSV is more suited for this kind of data.

For now the script only fetches from bitmex, but is can handle all instruments (XBTUSD, ETHUSD, etc.). I plan to add binance and coinbasepro soon (by the end of this week, or the next one), you can subscribe to notifications of the repository to watch my changes.

I'm closing this issue for now. I might also delete this repository in a near future to create a new one registering data for multiple exchanges.