Error when trying to get future history data

soaked-reflection commented 2 years ago

I want to download btc-usdt future 1h history data. Gate.io launched it since 2019-11-01. Until now it's around 2x365x12=8760 rows of data. Since gate-api does not allow get more than 2000 rows at a time. I have to find other ways. The function list_futures_candlesticks(settle, contract, _from, to, limit, interval) doesn't take from and limit at the same time, which makes the logic "download from 2019-11-01 for 2000 rows then set from to new and download anther 2000" couldn't work. Also you can't do it backwards, like get recent 2000 and get anther 2000 before. The only way seems to be calculating accurate from and to

# download from 2019-11-01 to 2021-11-01
settle = 'usdt'
contract = 'BTC_USDT'
_from = 1572537600.0 # 2019-11-01 00:00:00
to = 1635696000.0 # 2021-11-01 00:00:00
limit = 2000
interval = '1h'

candlesDF = pd.DataFrame(columns=['Time', 'Open', 'High', 'Low', 'Close', 'Vol'])
while True:
    to = _from + 3600*(limit-1)
    if to > 1635696000:
        to = 1635696000
    cs_response = api_instance.list_futures_candlesticks(settle, contract, _from=_from, to=to, interval=interval)
    for cs in cs_response:
        cs = cs.to_dict()
        for key, value in cs.items():
            cs[key] = float(value)
        candlesDF.loc[len(candlesDF)] = [cs['t'], cs['o'], cs['h'], cs['l'], cs['c'], cs['v']] 
    if to == 1635696000:
        break
    _from = to + 3600

I tried this to download and store data in pandas DataFrame. It returned this error

ApiException: (504)
Reason: Gateway Time-out
HTTP response headers: HTTPHeaderDict({'Server': 'awselb/2.0', 'Date': 'Thu, 18 Nov 2021 21:49:40 GMT', 'Content-Type': 'text/html', 'Content-Length': '132', 'Connection': 'keep-alive'})
HTTP response body: <html>
<head><title>504 Gateway Time-out</title></head>
<body>
<center><h1>504 Gateway Time-out</h1></center>
</body>
</html>

Really need some help, please

revilwang commented 2 years ago

Yes, calculating the exact from and to is the way to retrieve historical data. I don't see any problem with your code snippet. The 504 Gateway should be one-time error, meaning the server cannot handle your requests in time for now, which you can retry at a later time(All non-2xx response will be raised as an ApiException).

But we're moving towards archiving old data. Trading data long time ago now can be retrieved from here (you can also find the entrance from Developers at the website's main page down below). For historical data, I advise you to try it. I think your requirements can be mostly satisfied from it.

soaked-reflection commented 2 years ago

Thanks, this exactly meets my needs

gateio / gateapi-python

Error when trying to get future history data #81