Historic rates time interval not working

[kevinkatzke]

Hey,

I try to request historic rates for BTC-USD for a specific time interval:

servertime = cbpro.get_time()
endtime = datetime.datetime.strptime(servertime['iso'],'%Y-%m-%dT%H:%M:%S.%fZ')
granularity = 300
count = 3
delta = granularity * count 
starttime = endtime - datetime.timedelta(seconds=delta)
starttime = starttime.isoformat()
endtime = endtime.isoformat()

data = cbpro.get_product_historic_rates(str(instrument), start=str(starttime), end=str(endtime), granularity=int(granularity))

The requested time interval is:

Requested starttime :    2018-08-26T12:00:31
Requested endtime :      2018-08-26T12:15:31

The problem is that this returns anything but what is expected. Insted of 4 results I get 26. None of those values lies within the defined time interval. The most recent value is indeed older than my starttime.

When I take my time strings and I past them in this form: https://gdaxurl.surge.sh/ I correctly get the result as expected. Here is the HTTP request and its output:

https://api.gdax.com/products/BTC-USD/candles?start=2018-08-26T12:00:31.00000Z&end=2018-08-26T12:15:31.00000Z&granularity=300

How come the HTTP request delivers the correct values while the cbpro request does not?

---

I've installed the most recent version of cbpro 1.1.1 from github via pip.

[danpaquin]

Please read through #321 and see if the UTC issue is at fault here.

If not, please let us know if you are still experiencing this issue after updating your package.

[kevinkatzke]

@danpaquin I've updated my cbpro package to the latest git master and I've now version 1.1.2 installed. I'm still facing the same issue. Reponse way larger than expected and more or less random values.

[danpaquin]

I looked into this and getting correct results. The request url that the library is producing from the params is below. Clearly, this is not the exact request in your example, but returns the same data. Please let me know if you continue having issues here and I can dig deeper if you provide more information.

Request: public.get_product_historic_rates(product_id, start='2018-08-26T12:00:31', end='2018-08-26T12:15:31', granularity=300) https://api.pro.coinbase.com/products/BTC-USD/candles?start=2018-08-26T12%3A00%3A31&end=2018-08-26T12%3A15%3A31&granularity=300

Response: [[1535285700, 6700, 6700.01, 6700, 6700.01, 0.48444962999999996], [1535285400, 6700, 6704.75, 6704.41, 6700, 17.78793604], [1535285100, 6699.64, 6710.73, 6699.65, 6704.39, 43.122801550000005], [1535284800, 6696.9, 6702.98, 6696.98, 6699.64, 8.263433840000001]]

[kevinkatzke]

@danpaquin The Error still exists. Ist seems to be a bug indeed. Here's the response from Coinbase Support.

---

Hi Kevin,

Thank you for contacting Coinbase Pro Support.

I looked through your code and it seems to be fine. I was still able to reproduce the error using the same code on the cbpro endpoint.

I tested it on our still-active GDAX endpoints with the same code and it worked fine. I will submit this for our team for review, but in the meantime, we suggest using the unofficial GDAX python library. The GDAX endpoints will be operational until the end of the year.

I hope this helps. If you have any questions, please let me know!

Robert Coinbase Pro Support Team

---

I found out that the bug only appears when I'm using the authentificatedClient. The public client works fine:

# Produces an Internal server error currently!
auth_client = gdax.AuthenticatedClient(key, b64secret, passphrase, api_url)

# Works perfectly fine!
public_client = cbpro.PublicClient()

data = public_client.get_product_historic_rates(product_id="BTC-USD", start='2018-08-26T12:00:31', end='2018-08-26T12:15:31', granularity=300)

print(data)

[danpaquin]

Obviously, if you run the same code you can recreate almost any error. Please let me know if you cannot get the hard-coded strings to work on your end. The current authenticated client traces through the public client to make this call, so I highly doubt the issue with the endpoint differs between the two.

We are no longer supporting the GDAX library, but before we debug anymore, we must verify that the following expression is true before we debug any more. If not, the issue is with our datetime implementation and not the api wrapper.

str(starttime.isoformat()) == '2018-08-26T12:00:31'
and
str(endtime.isoformat()) == '2018-08-26T12:15:31'

The following code works for me:

private = cbpro.AuthenticatedClient(
    "key",
    "secret",
    "pass")

product_id = 'BTC-USD'
print(private.get_product_historic_rates(product_id, start='2018-08-26T12:00:31', end='2018-08-26T12:15:31', granularity=300))

Output:

[[1535285700, 6700, 6700.01, 6700, 6700.01, 0.48444962999999996],
 [1535285400, 6700, 6704.75, 6704.41, 6700, 17.78793604],
 [1535285100, 6699.64, 6710.73, 6699.65, 6704.39, 43.122801550000005],
 [1535284800, 6696.9, 6702.98, 6696.98, 6699.64, 8.263433840000001]]

[kevinkatzke]

Hey @danpaquin,

thanks for your response and sorry for my late response. Here is my current status. I've implemented everything as discussed in a public repl document so that we have a common test environment.

In the first script I've formatted the time strings as discussed. The call for historicRates is then made via the public_client. The datetime implementation tests are good and the cbpro results are just as expected:

Test our time formatting:
True
True

CBpro API response:
[[1535285700, 6700, 6700.01, 6700, 6700.01, 0.48444962999999996], [1535285400, 6700, 6704.75, 6704.41, 6700, 17.78793604], [1535285100, 6699.64, 6710.73, 6699.65, 6704.39, 43.122801550000005], [1535284800, 6696.9, 6702.98, 6696.98, 6699.64, 8.263433840000001]]

Here is the link to the working example:

https://repl.it/@k3vk4/cbprohistoricRatespublicClienttest

I've copied the script to my local machine , execute it in terminal and again everything looks good.

Now let's look at the follwoing example. Here everthing is the same as before but this time I'm not setting a fixed (faked) server time. This time I'm getting the current server time from coinbase API. Now the error shows up. Instead of the requested 3 datapoints we are getting 301, hence undefined behavior:

https://repl.it/@k3vk4/cbprohistoricRatespublicClientservertimetest

To fix this one has to substract ~5 seconds from the servertime that is retrieved from coinbase API. Now the code is working again and we're only getting the historic rates within out time interval:

https://repl.it/@k3vk4/cbprohistoricRatespublicClientservertimewithdelaytest

There is another issue. If one now takes the working code example and uses the coinbase pro sandbox instead of the regular environment the code produces an internal server error:

https://repl.it/@k3vk4/cbprohistoricRatessandboxservertimewithdelaytest

---

Please remove the duplicate tag from this issue if you agree that these issues are not the same as in #321

[danpaquin]

I hope this link works, but I could not solve this issue with the following either: https://repl.it/repls/CraftyKnownProject

My guess is the API is treating anything with an endtime ~= servertime to be a default request rather than a specific date range, but this is far fetched.

Leaving this open for someone else to consider.