search

Bungie-net / api

Resources for the Bungie.net API
Other
1.22k stars 92 forks source link

DestinyUnhandledException for various Stats endpoint calls on some accounts/activity modes #638

Closed ghost closed 6 years ago

ghost commented 6 years ago

For some people, it doesnt matter how many times you retry, it will never work. Alex is an example of this:

http://destinydevs.github.io/BungieNetPlatform/docs/API-Test#/Destiny2/GetHistoricalStatsForAccount/p:membershipType=4,destinyMembershipId=4611686018467250482/q:groups=0

I believe he was banned from comp, but thats the only thing that i can think of that makes him unique.

ghost commented 6 years ago

I havent been able to get my database to update in over 24 hours now. Do you know when things will be back to normal?

vthornheart-bng commented 6 years ago

We are currently experiencing increased latency to the game servers as a result of the massive increase in traffic both to the game and the API, as well as latency around the one-time upgrade to the current game version. To protect the integrity of the game servers, we are throttled as a result, which any users of the API will experience. Generally, the upgrade-related latency and timeouts tend to subside within a day or two from a major release, which should relieve the main source of this issue.

floatingatoll commented 6 years ago

Alternate, shorter, unofficial answer:

If it's a launch day (D2 ahoy), a patch day (Forsaken today), a beta-test day (such as Gambit this weekend), or if login queues are active (such as today), then all API consumers should plan on the API being down for 24-48 hours. On Tue, Sep 4, 2018 at 2:32 PM Vendal Thornheart notifications@github.com wrote:

We are currently experiencing increased latency to the game servers as a result of the massive increase in traffic both to the game and the API, as well as latency around the one-time upgrade to the current game version. To protect the integrity of the game servers, we are throttled as a result, which any users of the API will experience. Generally, the upgrade-related latency and timeouts tend to subside within a day or two from a major release, which should relieve the main source of this issue.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

ghost commented 6 years ago

But why do specific users just not work? You'd think it'd work eventually, but it doesnt seem to matter how many times I request this specific persons information. It just fails every single time!

It's not just me. Ive had others try to make the exact same call and it fails for them too.

floatingatoll commented 6 years ago

For your specific outage example, database sharding of 50 million active players at peak capacity would absolutely lead to a scenario where some users are inaccessible and others are fine.

On Tue, Sep 4, 2018 at 2:37 PM migit128 notifications@github.com wrote:

But why do specific users just not work? You'd think it'd work eventually, but it doesnt seem to matter how many times I request this specific persons information. It just fails every single time!

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/Bungie-net/api/issues/638#issuecomment-418526451, or mute the thread https://github.com/notifications/unsubscribe-auth/AAFqDMMabqd7DMjEvNCL_hWs4Hyv7BR6ks5uXvKggaJpZM4WZzW8 .

vthornheart-bng commented 6 years ago

Aye, and that is - for all intents and purposes - what's occurring right now unfortunately.

ghost commented 6 years ago

sigh... My stuff is designed for an all or nothing update, so I'm completely screwed for now.

floatingatoll commented 6 years ago

It’ll be okay! Everyone gets used to it after a while and it’s just part of launch days: Play Destiny Actual, rather than Helper Apps Anything :)

On Sep 4, 2018, at 14:47, migit128 notifications@github.com wrote:

sigh... My stuff is designed for an all or nothing update, so I'm completely screwed for now.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

lax20attack commented 6 years ago

Another failing request if it helps.

https://www.bungie.net/Platform/Destiny2/1/Account/4611686018429660408/Character/2305843009261349838/Stats/?periodType=AllTime&modes=7,5&groups=General

ghost commented 6 years ago

I have been playing destiny, but the thing is, the automation automates things that I now need to do instead of playing destiny.

Things still don't work.

vthornheart-bng commented 6 years ago

I'll give it a look, thanks for the link and info!

vthornheart-bng commented 6 years ago

Logged as TFS 706827 to investigate, I need to determine whether this is launch server latency/throttling related, or if there's something else happening.

ghost commented 6 years ago

Thanks!

ghost commented 6 years ago

Any updates?

I got some data on this.... The "GetHistoricalStatsForAccount" call succeeded for 160 people in my clan and failed for the other 373 people. That's almost exactly 30% of users that it works for.

For now I've got to disable the "get deleted characters" method. Hopefully I've been a good little programmer and designed things properly so the scripts that use the data from my db work with > 3 characters per account... Hopefully the loader will recover itself once these 1618 errors are resolved...

My discord is an absolute mess atm and I need my db to update...

floatingatoll commented 6 years ago

Could you select a random member of the 373 group for Bungie to inspect more closely — and/or, Can you provide the response header Set-Cookie value bungled=... for one of the 373 failures?

On Thu, Sep 6, 2018 at 11:02 AM migit128 notifications@github.com wrote:

Any updates?

I got some data on this.... The "GetHistoricalStatsForAccount" call succeeded for 160 people in my clan and failed for the other 373 people. That's almost exactly 30% of users that it works for.

For now I've got to disable the "get deleted characters" method. Hopefully I've been a good little programmer and designed things properly so the scripts that use the data from my db work with > 3 characters per account... Hopefully the loader will recover itself once these 1618 errors are resolved...

My discord is an absolute mess atm and I need my db to update...

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/Bungie-net/api/issues/638#issuecomment-419187808, or mute the thread https://github.com/notifications/unsubscribe-auth/AAFqDEvLyxq5GnCUgq9r06p4i06gmbQ7ks5uYWNEgaJpZM4WZzW8 .

vthornheart-bng commented 6 years ago

We will not be able to deploy a fix for this until 9/12, which is our next scheduled hotfix - but I will let you know once that's ready to roll!

ghost commented 6 years ago

ok. i've patched my stuff enough so it works for people in my clan so this isn't quite as bad anymore. The 1000 people in my discord that aren't in my clan wont have their roles though... Not as big of a deal.

floatingatoll - there are no cookie values for this. sounds like they understand the problem so i wont be reporting which users work and which dont.

floatingatoll commented 6 years ago

"Roles"? On Thu, Sep 6, 2018 at 12:14 PM migit128 notifications@github.com wrote:

ok. i've patched my stuff enough so it works for people in my clan so this isn't quite as bad anymore. The 1000 people in my discord that aren't in my clan wont have their roles though... Not as big of a deal.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

ghost commented 6 years ago

discord roles. part of the reason i'm hitting the api and loading the data in my own db as a cache is so i can have roles in my discord that show how many raid clears people have, what their KD is, and what division they are in. there are other scripts that act on the data in the db as well to report on players activity in the game, hand out high activity roles, create a list of people who do not meet our activity requirements, and more!

vthornheart-bng commented 6 years ago

After further investigation, I found out that this and #633 unfortunately have different root causes. I was able to fix that (and thus, to fix Activity History that requests Gambit as a subset) for 9/12, but the root cause of this issue you're seeing looks to unfortunately be happening on a deeper level than Bungie.net itself: it's going to require a fix for the game servers. I've reported it up to the folks who work on that system, but I don't have an ETA yet of when they'll be able to deploy it: their deployment cadence and deadlines for when a fix must be in and verified before it can ship are much more stringent than the Bungie.Net side, so it may be a bit longer before we have a real fix for this.

I'll give you an ETA when I can. I apologize for the outages.

ghost commented 6 years ago

Yikes.... guess I need to stop polling for deleted characters everywhere for now.

vthornheart-bng commented 6 years ago

Aye. Sorry for the interruption this is causing you - at first it sounded like the symptoms of game server throttling that we were seeing in other systems, but it's definitely not the case. It appears to be fairly intermittent in terms of who actually is affected: potentially due to some specific activities that they've played: but I'm not certain of the specifics that can cause a character's aggregate history to end up in this state. I hope to get more info when they're able to do further analysis.

vthornheart-bng commented 6 years ago

Okay, looks like they've been given a release window of September 18th for the game-side fixes for this. Once we get those, we should be good to go. Sorry about that - looks like it'll be a little longer of a wait, but not as long as I was fearing.