retry mechanism / better error handling

[kmudrick]

Not sure if yahoo is just being flakey tonight, but I was running into an issue where some calls were randomly failing with a 500 error (no response body). Successive calls would succeed, however.

When yahoo returns a 500 and there is no response body, then nothing gets parsed as data and this line fails actually

Locally, I solved my issue by dropping in requestretry, which by default will retry on 500 errors up to 5 times, with a delay in between.

Any thoughts on adopting that approach as well? Or at least inspecting the response statusCode and invoking the api callback with an (appropriate) error when data is undefined (and then users could simply implement their own retry logic.)

[whatadewitt]

I'm willing to bet this is as a result of the change to OAuth2 and something I've done has buggered up the error handling. I'll fix it as soon as I can...

[kmudrick]

Hmm. Not sure if this is what you were hinting at, but these definitely were not auth-related failures that I was seeing. I got more debug logging from the request module via NODE_DEBUG=request to see what was going on and yahoo was literally returning 500s and an empty body. I did not check to see if all the failures were coming from the same server, but I'm guessing either they have an unhealthy machine somewhere responding to requests or a scaling issue.

I also would have expected for request itself to see a 500 response status and pass an appropriate error to the callback, but I'm used to using superagent which considers 4xx/5xx to be errors by default.

[whatadewitt]

I'll play around a bit with superagent to see if I can duplicate what you're seeing... for what it's worth I am able to easily refresh my token with request right now.

Based on issue #26 I am removing the actual "token expired" bit out of the API, as it's causing issues with other errors and it's pretty easy to handle within apps... I doubt this will really help out here, but I figured I should let you know in case you're looking for that "token_expired" error reason...

[kmudrick]

It is definitely not a token expired or other oath issue (continuing to use the same token after failed requests from 500 errors works fine) - the reality is that any API call can fail (for a non auth related reason - servers down etc - statuses in the 5xx range not 4xxx) and it is not handled correctly, so there's no way to even retry.

[bflopez]

Any progress on the issue? I am making over 800 requests against this API, specifically collecting all the players in the players/leagues. I ran into this issue somewhat while testing it manually but it was very minimal.

I decided to automate the API requests using async.mapLimit to avoid duplicate callbacks but I get an error every time before completion. I am wondering if it has something to do with hitting the API too quickly/often? Similar to @kmudrick but in a different file, I am getting an error on this line because data.fantasy_content is undefined. After logging the requests it looks like Yahoo sends an HTML file as data and no error.

[whatadewitt]

Can you let me know what the call is that is causing this for you? I can investigate it, apologies for it falling off my radar, it's been busy times at the 9 to 5...

[whatadewitt]

Sweet shit... reviewing the Yahoo! API, almost none of the subresource calls for the "players" collection "leagues" call are only applied in the league context... I guess I missed that when I built this with the old docs... I'll try to make this change tonight as I am now curious about some stuff I'd been trying to figure out a few weeks ago.

[bflopez]

@whatadewitt I deleted a previous comment but I was hammering away at the Players.leagues call. What I found was that Yahoo sometimes returns no error and an HTML response body telling you about an internal 500 error or a 999 (rate limit exceeded) error.

[whatadewitt]

I've had 999 when I first started building this thing...

I will play around and see if I can get the 500, but I'm going to try to fix the players.leagues call now, then I'll probably push a v3.0.0 that will change that to players.league, as all of the subresource calls only (really) matter within the league context..

After that maybe we can try to figure out if we can hookup the 500 error, but I am confident that simply having this thing work better according to the docs, that it could help resolve the problem (but maybe that's just because I'm a wishful thinker)

[whatadewitt]

Do you have the code that I could use to debug this? I cannot replicate the issue...

[coolsoftwaretyler]

Hey @whatadewitt - thanks for all your work on this. I used your documentation to write some custom libraries focused on fantasy football.

I've been seeing 999 errors which seem to be rate limits. You mentioned seeing them during development as well. Did you ever find any documentation or guidance about Yahoo sports API rate limits specifically? Any mention of "rate limit" in their docs specifically say 100,000 requests/hour, but I'm seeing 999s waaaaaay sooner than that.

Any tips or tricks you know of? Thanks in advance if you get around to reading this.

[whatadewitt]

@coolsoftwaretyler I'll reach out to the guy at Yahoo! who helps with these sorts of things. I'll let you know!

[Liam-OShea]

Hey @coolsoftwaretyler ,

I scoured the web a while ago looking for more info on the rate limit boundaries but there is practically no info, some figures I found did not match my experience at all and I think may have applied to older versions of Yahoo's API.

I did a bit of research back then and was able to determine a rough figure for the number of batched calls I could make at once, you can check out that info and the resulting convo here https://github.com/whatadewitt/yahoo-fantasy-sports-api/issues/81#issuecomment-913756634

Hope this helps a bit!

[coolsoftwaretyler]

Thanks, everyone!

@whatadewitt - that would be a huge help to me. I want to play by the rules, but they just haven't published them...

@Liam-OShea - your comment there is really helpful. I'm going to respond in that thread.

[whatadewitt]

@coolsoftwaretyler -- this was the response I got asking about rate limits:

Likely not. That error more generally applies to abuse detection, and we don't document abuse prevention details (because the abusers would love that). That being said, it does seem fair and reasonable to document what rate you're allowed to hit a public API, and if our docs were updated, I'd expect that to be included. But they aren't :-(

[coolsoftwaretyler]

Thanks so much @whatadewitt. That's really helpful. I owe you one!