API request issues with the new /roles route

[john-robert]

In agreement with @madninja, I'll summarize some API related issues I observe with the new /roles route. I also point out that according to the docs, the old /activity route will be deprecated on May 1st so this issue may perhaps benefit from some upranking in the task queue – up to the devs.

1) Missing information Request: https://api.helium.io/v1/accounts/14Z2cyJuhLKk2rtUiR64Zzptze9BYD8Jktr4cSG5hZNf5hNffWM/roles?min_time=2021-02-20T00:00:00&max_time=2021-03-05T00:00:00 Response: list of key-value pairs exclusively with types "rewards_v1" Issue: I added a gateway and asserted its location within that timeframe undoubtedly, however, none is listed although I loop through all cursors. The hotspot in question is: 11DxBy2UTSoHx2E1ShRf35cV3pC4hpHZ9zs93CPgPQDbcrLtmVD

Note this issue seems to depend on the wallet address, e.g., this request (https://api.helium.io/v1/accounts/13aYRgiwZCVqEGeWPwUPrvDa41NgLjKoHjNm4UBNP63kxnyFoJS/roles?min_time=2021-02-20T00:00:00&max_time=2021-03-05T00:00:00) works just as expected – "add_gateway_v1" and "assert_location_v1" transactions are listed accordingly.

2) Missing time ranges Request (one year time range!): https://api.helium.io/v1/accounts/14Z2cyJuhLKk2rtUiR64Zzptze9BYD8Jktr4cSG5hZNf5hNffWM/roles?filter_types=payment_v1,payment_v2&min_time=2021-04-01T00:00:00&max_time=2022-04-15T00:00:00 Response: list of key-value pairs as expected Issue: Only payments for April 2022 are returned although I request one year of data (I do payments every other month, so they should be there). This indicates to me not all information are retrieved. I did observe this behaviour not only for filter_types "payment_v*" but also for more general /roles calls although this is somewhat harder to pinpoint.

3) For payments, MEMO in response is not as expected Request: https://api.helium.io/v1/accounts/14Z2cyJuhLKk2rtUiR64Zzptze9BYD8Jktr4cSG5hZNf5hNffWM/roles?filter_types=payment_v1,payment_v2&min_time=2022-04-01T00:00:00&max_time=2022-04-15T00:00:00 Request2 (based on one transaction hash of above request): https://api.helium.io/v1/transactions/JLQltkLan7-nkv6bOTy82pTxlpL9qjYeZLXnNje4gKQ?actor=14Z2cyJuhLKk2rtUiR64Zzptze9BYD8Jktr4cSG5hZNf5hNffWM Response: nested key-value pairs as indicated in the docs. Issue: When handling my network and paying out location owners, I always use the memo field in the Helium app to indicate what months the payments are for. MEMOs from the API may read as "RGVjLU1hcgA=" or "SmFuLU1hcgA=", though they should read "Jan-Mar" or similar. Could it be an encoding issue?

4) API weight limit inconsistencies When sending my requests, I more often than not get a 429 response. What is weird is that the weight limits seem to be retrogressiv, i.e., with each time I send the request I get more data out before being 429ed. After 3x or so my request gets finally fully through. Also, this 429 response seems to trigger early. It is not like I abuse the API with thousands of calls, it already happens when simply testing some small time ranges. My server location is Germany. For my Heroku IP, this issues seems to be less true.

If interested what for I use the API, you could have a look at the hnt-tracker that is listed as community tool in the Helium explorer.

[madninja]

1. This PR https://github.com/helium/blockchain-etl/pull/318 will address number 1 once we run deploy it and run the associated migration.
2. Could you file a new issue with these details. I haven't been able to look at that in detail yet
3. A payment memo is a base64 encoded 8 bytes.. it doesn't attempt to interpret them as strings for display. They are not considered as strings
4. The rate limiter is primarily based on use of the api from a single source ip and is intentionally aggressive because there has been tremendous abuse. Please contact the fine folks at BlockJoy if you would like a dedicated access controlled api (cc @densone )