search

w3f / staking-rewards-collector

Apache License 2.0
77 stars 33 forks source link

Subscan API rate limit issue #31

Closed bjweaver closed 3 years ago

bjweaver commented 3 years ago

Subscan implemented an API rate limit, causing rewards calculation to fail.. I added a basic 100ms sleep to the API calls.

https://github.com/w3f/staking-rewards-collector/pull/30

jonasW3F commented 3 years ago

I encountered the issue also. Will take a look.

jonasW3F commented 3 years ago

Something weird is going on here... With the current version of the collector, I run into an error "TypeError: Cannot read property 'count' of undefined" (without a mention of the API limit). With your fix, it seems to work for Polkadot Addresses but for Kusama addresses, I still get a "TypeError: Cannot read property 'block_timestamp' of undefined".

Could you try to replicate this? Use the version of your PR and change the start date to ""start": "2020-01-01". Now, as long as the Kusama address is in the userConfig, it will produce the abovementioned error. Having only the Polkadot Address in will produce a valid output. I tried it with a few DOT addresses (always works) and few KSM addresses (never works, if the time-span is long enough).

bjweaver commented 3 years ago

yes, seeing the same error, seems unrelated to the rate limit issue. Looks like the loopIndex it over running the amount of records returned on the last page. Something weird with the subscan data probably. If you change line 41 to this, it works, but this might not be the right fix.

loopIndex = min(stakingObject.data.count - page*100 - 1,100);

jonasW3F commented 3 years ago

yes, I also saw that reducing the loopindex would help, but a quick check showed that not all rewards would be parsed then. I first need to understand if the API changed.

jonasW3F commented 3 years ago

Subscan confirmed that it is an issue on their side. They have too many requests on their Kusama API and their service times out. They are trying to solve this. Once it is resolved, I will check your PR #30 and merge it.

Thank you!

jonasW3F commented 3 years ago

The issue seems to be fixed on Subscan's side. Once you do the minor formatting changes to #30, I merge it and it should be operational again. Thank you

I'll close this issue.