Bungie-net / api

Resources for the Bungie.net API
Other
1.22k stars 92 forks source link

Plugs are kind of a nightmare #735

Open xxbiohazrdxx opened 6 years ago

xxbiohazrdxx commented 6 years ago

This is gonna be a long one, so bear with me please.

Let's say I want to parse all of the plugs for a particular instance of an item. Specifically, I would like to find the Weapon Mods for that item. Let's take a look at a table of the Weapon Mods in the manifest.

displayProperties.name itemTypeDisplayName hash plug.plugCategoryIdentifier plug.plugCategoryHash itemType itemSubType
Kinetic Damage Mod Weapon Mod 4160547565 v300.weapon.damage_type.kinetic 405287501 19 0
Targeting Adjuster Weapon Mod 3228611386 v400.weapon.mod_guns 510594033 19 0
Radar Tuner Weapon Mod 371216963 v400.weapon.mod_guns 510594033 19 0
Freehand Grip Weapon Mod 736000386 v400.weapon.mod_guns 510594033 19 0
Icarus Grip Weapon Mod 941997506 v400.weapon.mod_guns 510594033 19 0
Counterbalance Stock Weapon Mod 1588595445 v400.weapon.mod_guns 510594033 19 0
Weapon Attack Mod Weapon Mod 4207478320 v300.weapon.damage_type.attack 674045876 19 0
Boss Spec Weapon Mod 2788909693 v400.weapon.mod_damage 974142739 19 0
Minor Spec Weapon Mod 4091000557 v400.weapon.mod_damage 974142739 19 0
Major Spec Weapon Mod 984527513 v400.weapon.mod_damage 974142739 19 0
Taken Spec Weapon Mod 1513326571 v400.weapon.mod_damage 974142739 19 0
Solar Damage Mod Weapon Mod 2273483223 v300.weapon.damage_type.energy 1466776700 19 0
Random Damage Type Weapon Mod 2510923716 v300.weapon.damage_type.energy 1466776700 20 0
Void Damage Mod Weapon Mod 3728733956 v300.weapon.damage_type.energy 1466776700 19 0
Arc Damage Mod Weapon Mod 3994397859 v300.weapon.damage_type.energy 1466776700 19 0
Arc Damage Mod Weapon Mod 4126105782 v300.weapon.damage_type.energy 1466776700 19 0
Solar Damage Mod Weapon Mod 344032858 v300.weapon.damage_type.energy 1466776700 19 0
Void Damage Mod Weapon Mod 1837294881 v300.weapon.damage_type.energy 1466776700 19 0
Empty Mod Socket Weapon Mod 2323986101 v400.weapon.mod_empty 3945646995 19 0
Backup Mag Weapon Mod 3336648220 v400.weapon.mod_magazine 4087004818 19 0

How would I do that? At first glance it looks like the itemType and itemSubType might work, but the combination of

itemType == 19 and itemSubType == 0

is actually valid for many, many other plugs and not just Weapon Mods (not to mention there's a single 20 in there for itemType).

I can't use the plug.plugCategoryHash even though it looks promising. Pre 4.x/Forsaken mods have a value of 1466776700....except when they don't. Kinetic Damage Mod and Weapon Attack Mod are different. Post 4.x/Forsaken mods have a plug.plugCategoryHash of either 510594033 or 974142739 unless there is no mod, in which case it is 3945646995.

It looks like the easiest way to get only Weapon Mods is itemTypeDisplayName == "Weapon Mod". That's not ideal, but it works.

However, this doesn't carry over to other categories. Let's take Armor Mods (Note that I have significantly truncated this and the proceeding tables because we don't need to see every plug to get the point)

displayProperties.name itemTypeDisplayName hash plug.plugCategoryIdentifier plug.plugCategoryHash itemType itemSubType
Impact Absorption Mod Arms Armor Mod 2361721165 enhancements.arms 640682011 19 0
Solar Ordnance Mod Arms Armor Mod 2499197808 enhancements.arms 640682011 19 0
Void Ordnance Mod Arms Armor Mod 3014957206 enhancements.arms 640682011 19 0
Kinetic Counterbalance Mod Chest Armor Mod 2565377288 enhancements.chest 383756333 19 0
Arc Paragon Mod Chest Armor Mod 2782922115 enhancements.chest 383756333 19 0
Kinetic Counterbalance Mod Chest Armor Mod 2855491837 enhancements.chest 383756333 19 0
Emperor's Blaze Class Item Mod 3018857481 enhancements.class 1955304674 19 0
Emperor's Balance Class Item Mod 3511308612 enhancements.class 1955304674 19 0
Emperor's Shock Class Item Mod 2044767186 enhancements.class 1955304674 19 0
Kinetic Counterbalance Mod Hunter Cloak Mod 2352344841 enhancements.class_hunter 1425760675 19 0
Solar Impact Mod Hunter Cloak Mod 2407848812 enhancements.class_hunter 1425760675 19 0
Acceleration Mod Hunter Cloak Mod 3076431036 enhancements.class_hunter 1425760675 19 0
Solar Ordnance Mod Titan Mark Mod 2291847593 enhancements.class_titan 397607061 19 0
Power Munition Loader Titan Mark Mod 3131975519 enhancements.class_titan 397607061 19 0
Kinetic Munition Loader Titan Mark Mod 3217779897 enhancements.class_titan 397607061 19 0
Energy Bracket Mod Warlock Bond Mod 2575146041 enhancements.class_warlock 1216445120 19 0
Arc Impact Mod Warlock Bond Mod 3094848290 enhancements.class_warlock 1216445120 19 0
Void Ordnance Mod Warlock Bond Mod 3145634685 enhancements.class_warlock 1216445120 19 0
Solar Paragon Mod Helmet Armor Mod 2365108107 enhancements.head 744326128 19 0
Solar Ordnance Mod Helmet Armor Mod 2450367421 enhancements.head 744326128 19 0
Impact Absorption Mod Helmet Armor Mod 2451142299 enhancements.head 744326128 19 0
Energized Leg Armor Mod 2426323900 enhancements.legs 1175552225 19 0
Void Impact Mod Leg Armor Mod 2988871238 enhancements.legs 1175552225 19 0
Self-Repairing Mod Leg Armor Mod 3034612620 enhancements.legs 1175552225 19 0
Empty Mod Socket Armor Mod 2600899007 enhancements.universal 3347429529 19 0
Transcendent Blessing Armor Mod 3255036626 enhancements.universal 3347429529 19 0
Riven's Curse Armor Mod 3300281544 enhancements.universal 3347429529 19 0
Paragon Mod Armor Mod 3649276740 enhancements.universal 3347429529 19 0
Resilience Mod Armor Mod 3790655795 enhancements.universal 3347429529 19 0
Empty Mod Socket Armor Mod 3851138800 enhancements.universal 3347429529 19 0
Recovery Mod Armor Mod 126924337 enhancements.universal 3347429529 19 0
Mobility Mod Armor Mod 275671171 enhancements.universal 3347429529 19 0
Super Mod Armor Mod 444600262 enhancements.universal 3347429529 19 0
Ordnance Mod Armor Mod 817770433 enhancements.universal 3347429529 19 0
Impact Mod Armor Mod 1512056643 enhancements.universal 3347429529 19 0
4072982370 enhancements.universal 3347429529 19 0
Empty Mod Socket 1835369552 enhancements.universal 3347429529 19 0

The hashes are a bit better here, 3.x Armor Mods have a plug.plugCategoryHash that definitively mark this Plug as being an Armor Mod and for a particular item/bucket. This isn't super helpful if we want to find all armor mods for all slots, for that we can instead do a itemTypeDisplayName.contains("Armor Mod") or perhaps plug.plugCategoryIdentifier.startsWith("enhancements").

Masterworks?

displayProperties.name itemTypeDisplayName hash plug.plugCategoryIdentifier plug.plugCategoryHash itemType itemSubType
Sturm Catalyst 1824496860 v300_new_hand_cannon0_masterwork 746309217 19 0
Sunshot Catalyst 3617228241 v300_new_hand_cannon1_masterwork 4249350918 20 0
Upgrade Masterwork 223776600 v300_new_hand_cannon1_masterwork 4249350918 19 0
Sunshot Catalyst 354293076 v300_new_hand_cannon1_masterwork 4249350918 20 0
Sunshot Catalyst 1824496861 v300_new_hand_cannon1_masterwork 4249350918 19 0
Upgrade Masterwork 3547298846 v300_new_pulse_rifle0_masterwork 520351532 19 0
Vigilance Wing Catalyst 615063267 v300_new_pulse_rifle0_masterwork 520351532 19 0
Vigilance Wing Catalyst 1620506138 v300_new_pulse_rifle0_masterwork 520351532 20 0
Vigilance Wing Catalyst 1772168355 v300_new_pulse_rifle0_masterwork 520351532 20 0
Tier 4 Armor 2572873168 v400.plugs.armor.masterworks.stat.resistance_1 1514141502 19 0
Masterwork 2572873169 v400.plugs.armor.masterworks.stat.resistance_1 1514141502 19 0
Tier 1 Armor 2572873173 v400.plugs.armor.masterworks.stat.resistance_1 1514141502 19 0
Tier 2 Armor 2572873174 v400.plugs.armor.masterworks.stat.resistance_1 1514141502 19 0
Tier 3 Armor 2572873175 v400.plugs.armor.masterworks.stat.resistance_1 1514141502 19 0
Tier 1 Armor 3130025792 v400.plugs.armor.masterworks.stat.resistance_2 1514141501 19 0
Tier 3 Armor 3130025794 v400.plugs.armor.masterworks.stat.resistance_2 1514141501 19 0
Tier 2 Armor 3130025795 v400.plugs.armor.masterworks.stat.resistance_2 1514141501 19 0
Tier 2 Weapon 2203506851 v400.plugs.weapons.masterworks.stat.draw_time 482070447 19 0
Tier 5 Weapon 2203506852 v400.plugs.weapons.masterworks.stat.draw_time 482070447 19 0
Tier 4 Weapon 2203506853 v400.plugs.weapons.masterworks.stat.draw_time 482070447 19 0
Tier 7 Weapon 2203506854 v400.plugs.weapons.masterworks.stat.draw_time 482070447 19 0
Tier 6 Weapon 2203506855 v400.plugs.weapons.masterworks.stat.draw_time 482070447 19 0
Tier 9 Weapon 2203506856 v400.plugs.weapons.masterworks.stat.draw_time 482070447 19 0
Tier 8 Weapon 2203506857 v400.plugs.weapons.masterworks.stat.draw_time 482070447 19 0
Masterwork 3689550782 v400.plugs.weapons.masterworks.stat.draw_time 482070447 19 0
Masterwork 1639384016 v400.plugs.weapons.masterworks.stat.draw_time 482070447 19 0
Masterwork 2357520979 v400.plugs.weapons.masterworks.stat.handling 199786516 19 0
Masterwork 186337601 v400.plugs.weapons.masterworks.stat.handling 199786516 19 0
Tier 8 Weapon 518224738 v400.plugs.weapons.masterworks.stat.handling 199786516 19 0
Tier 9 Weapon 518224739 v400.plugs.weapons.masterworks.stat.handling 199786516 19 0
Tier 2 Weapon 518224744 v400.plugs.weapons.masterworks.stat.handling 199786516 19 0
Tier 3 Weapon 518224745 v400.plugs.weapons.masterworks.stat.handling 199786516 19 0

Check and see if plug.plugCategoryIdentifier.contains("masterwork") then also see if plug.plugCategoryIdentifier.startsWith("v4") so I know to query the manifest for investmentStats[0].statTypeHash to get the localized name of the Masterwork stat.

How about Shaders?

displayProperties.name itemTypeDisplayName hash plug.plugCategoryIdentifier plug.plugCategoryHash itemType itemSubType
Default Shader Restore Defaults 4248210736 shader 2973005342 19 20
War Cult Scheme Shader 2296172971 shader 2973005342 19 20
SUROS Tone Shader 2307426896 shader 2973005342 19 20
SUROS Modular Shader 2307426898 shader 2973005342 19 20
Crucible Triumph Shader 2361935691 shader 2973005342 19 20
Metallic Sunrise Shader 2395477994 shader 2973005342 19 20
Precursor Vex Chrome Shader 2395477996 shader 2973005342 19 20
Desert of Gold Shader 2395477997 shader 2973005342 19 20
Descendant Vex Chrome Shader 2395477998 shader 2973005342 19 20
Mercury Vex Chrome Shader 2395477999 shader 2973005342 19 20
Rusted Iron Shader 2448092902 shader 2973005342 19 20
Praetorian Visage Shader 2543722796 shader 2973005342 19 20
Calus's Shadow Shader 2543722797 shader 2973005342 19 20
Crucible Solemnity Shader 2588739576 shader 2973005342 19 20

This one is easy! Every shader in the game has a plug.plugCategoryHash of 2973005342! I wish all the plugs were like this!

I realize that the actual data that the API is retrieving may very well may be beyond your control, but if there was some kind of massaging that could be done before it is returned would make Plugs a lot easier to use. An enumerator for the types of Plugs and honestly just some consistency of which field I should be looking at to determine exactly what the Plug is.

floatingatoll commented 6 years ago

(Cancel that, sorry, I misread something.)

xxbiohazrdxx commented 6 years ago

There's also some inconsistency in the way Plugs/sockets are presented from the instance item. For example

{ "sockets": [ { "plugHash": 2874284214, "isEnabled": true, "isVisible": true, "reusablePlugHashes": [ 2874284214 ], "reusablePlugs": [ { "plugItemHash": 2874284214, "canInsert": true, "enabled": true } ] }, { "plugHash": 1288081798, "isEnabled": true, "isVisible": true, "reusablePlugHashes": [ 1288081798, 1452368632 ], "reusablePlugs": [ { "plugItemHash": 1288081798, "canInsert": true, "enabled": true }, { "plugItemHash": 1452368632, "canInsert": true, "enabled": true } ] }, { "plugHash": 1588595445, "isEnabled": true, "isVisible": true }, { "plugHash": 518224751, "isEnabled": true, "isVisible": true, "reusablePlugHashes": [ 518224748 ], "reusablePlugs": [ { "plugItemHash": 518224748, "canInsert": true, "enabled": true } ] } ] }

This is an excerpt from an actual item.

The vast majority of the time, reusablePlugs is an array of plugHashes representing all of the possible Plugs for this particular 'section' (or whatever terminology you would like to use), including the currently active Plug.

However, this is not always true.

In the case of Masterworks, reusablePlugs contains an array of the next Masterwork upgrade tier but NOT the currently active tier.

Similarly, in the case of Shaders, reusablePlugs contains the 'Default Shader' option but NOT the currently active Shader.

In the case of Mods, reusablePlugs is simply absent.

This is a minor gripe, but honestly the wildly varying/inconsistent nature makes for a lot of boilerplate edge case checking code.

vthornheart-bng commented 6 years ago

Yes indeed, the complexity of the sockets code reflects the complexity of the potential states in which an item can have socket data combined with a desire to minimize the potential amount of data that we have to return. I deeply empathize with your frustration, as we've felt the same working on our own features as well as in creating the API itself.

Sockets were originally intended to be a fairly simple system - a reduction in complexity from the extremely complex Talent Grid system of Destiny 1 - but quickly expanded into something that is - for both you and we - pretty tedious to use. (it is a real textbook example of feature creep: and unfortunately, we have few options for hiding this increase in complexity on the game side without either being less comprehensive or creating a great deal of redundancy in the data we return from requests, which I will try to go into in the write-up so as to give a better understanding for why we don't abstract the game's complexity for these features as well as I wish we would)

I'll try to give a more comprehensive write-up of this information about how Sockets work, and how you can get at the information you're looking for: but indeed, there isn't an easy to use view of sockets. I will do my best to write an article to use as a guide, however: I think it's reached a level of complexity where it's downright frustrating to use without one, and that sucks... and particularly sucks given that we don't have a guide for it.

Just to make sure I'm covering all the bases of what you're interested in when I make this write up, you're wondering about:



Does anyone else (yourself included) have any other questions/topics on the subject that you'd like me to cover in this writeup? I'm eager to share what I can!

vthornheart-bng commented 6 years ago

Also, on a completely unrelated side topic, I love your Github icon. Tecmo Bowl for life.

floatingatoll commented 6 years ago

It would be invaluable to have a complete walkthrough for two or three wildly different items that, between them, make use of all known sockets 'features'.

On Fri, Oct 19, 2018 at 3:21 PM Alex Tennant notifications@github.com wrote:

What do the “empty” sockets represent? i.e. not initialHash, reuseable and random plugs.

How are curated rolls represented? Is it always the initialHashes? What’s the best way of identifying something with a curated roll?

How can you tell which masterworks an item can actually roll with? It seems the item definitions contain a lot of “impossible” masterworks.

I’m sure have more, these are what come to mind right now.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/Bungie-net/api/issues/735#issuecomment-431516072, or mute the thread https://github.com/notifications/unsubscribe-auth/AAFqDEIU_QeBU0T96qldN9Y7ejdzMzGNks5umlBggaJpZM4Xxapn .

adtennant commented 6 years ago

What do the “empty” sockets represent? i.e. no initialHash, reuseable and random plugs.

How are curated rolls represented? Is it always the initialHashes? What’s the best way of identifying something with a curated roll?

How can you tell which masterworks an item can actually roll with? It seems the item definitions contain a lot of “impossible” masterworks.

I’m sure have more, these are what come to mind right now.

vthornheart-bng commented 6 years ago

Yes, thank you both for your input! Anyone with more, please bring them forward here!

ArkahnX commented 6 years ago

Tacking on to adtennant here, when an item is a curated roll ( no changeable perks, fully masterworked) it appears to be that the curated roll is represented by the initial plug hash values. Are these values also capable of being random rolls? Or is it limited exclusively to the random list of plugs provided (forgot the property name, can't look it up right now)

This was asked earlier on GitHub, but there isn't a way to have in the api how many random rolls a given column can contain correct?

vthornheart-bng commented 6 years ago

Thank you, good questions as well!

vthornheart-bng commented 6 years ago

Okay, this is my outline so far for what needs to be covered. It's going to take me some time to fill this all out, but where I could I tried to include info in this summary that might help with some of the more direct questions:

vthornheart-bng commented 6 years ago

Ah, there's one more thing in here too not relevant to the documentation: it sounds like we're having issues with item category hashes. Those ought to be a reliable way to find out what items are mods, and at least in some vague level what type of mods they are. If you're finding miscategorized ones, something's broken there that we can fix! We won't support the old/obsoleted mods anymore, but we can fix any miscategorized 4.x plugs!

ckhicks commented 6 years ago

This. Is. Epic. Dropping a comment here to cheer everyone on and make sure I get notifications about this one in case I miss something later. 👍

floatingatoll commented 6 years ago

The object “Five of Swords” is a stellar example of maximum plaid, in case it comes in handy later on.

On Oct 19, 2018, at 20:52, CK Hicks notifications@github.com wrote:

This. Is. Epic. Dropping a comment here to cheer everyone on and make sure I get notifications about this one in case I miss something later. 👍

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

xxbiohazrdxx commented 6 years ago

On another note, is there a particular reason why plugs are in the DestinyInventoryItemDefinition table in the manifest instead of having their own table?

vthornheart-bng commented 6 years ago

@floatingatoll Good call, that's another whole way that sockets are being used! I'll add that to the list.

@xxbiohazrdxx That's a very good question with a very unfortunate answer. The short answer is that, very early on, the game designers decided that almost anything they wanted in the game could be represented as either an Item or a Vendor. The implication of this original decision made before Destiny 1 went public lives on today in features that are more-or-less bolted onto both concepts.

The long answer is that Plugs reuse a lot of properties shared between the various types of things that the game considers to be items: enough that they didn't want to build separate systems to handle the very similar functionality, so Plugs became a "feature" that an item can have rather than its own entity. Though in practice, any item that is a Plug doesn't get treated in any context other than as a plug (even though plugs themselves get treated in very different ways, as eluded to in the outline above). Theoretically, some cruel designer could someday make a weapon that could be plugged into another weapon, or a ship that could be plugged into a sparrow. Thankfully, they haven't done that yet.

As to why we don't hide that in the API, there's a couple of reasons: most of them having to do with our own time and maintenance constraints, that also make it easier for us to treat them the same way as the game's content treats them.

If I had my druthers, or a larger staff working on the API, or even a promise that the concept of Plugs won't expand further than it already has (I have no such promise, and thus can't make that promise), I'd love to look into making our own "derivative definitions" that treat plugs as separate entities to external users.

However, we have two problems with that: (1) breaking backward compatibility, and (2) derivative definitions have often bitten us in the butt in the API: and I'm very hesitant to do so unless I get solid assurances that the given feature is likely to not change in such a way that it breaks our derivative definitions.

Milestones are an interesting example of where we attempted to create a derivative definition because the source items were basically unusable for the stated purposes from an API perspective, and the fact that we've done so has bitten us in the butt over and over again as the concept of what a milestone is and the data it holds changes, leaving us scrambling to try and maintain backwards compatibility while exposing whatever new functionality or variability has been introduced. Users of the Destiny 1 API will remember that we made the same mistake with "Advisors," as well as when we tried to hide some of the complexity of the original talent grids only to have core assumptions change out from under us at the whim of game design: changes that required us to make sudden backwards-compatibility breaking changes in order to fix the data we were returning.

vthornheart-bng commented 6 years ago

If you want a good laugh, I go into the Vendor side of this bolting-on here: https://bungie-net.github.io/multi/schema_Destiny-Definitions-DestinyVendorDefinition.html

xxbiohazrdxx commented 6 years ago

@vthornheart-bng

Hey since you brought up potentially miscategorized plugs

displayProperties.name hash itemTypeDisplayName plug.plugCategoryIdentifier plug.plugCategoryHash
Boss Spec 2788909693 Weapon Mod v400.weapon.mod_damage 974142739
Minor Spec 4091000557 Weapon Mod v400.weapon.mod_damage 974142739
Major Spec 984527513 Weapon Mod v400.weapon.mod_damage 974142739
Taken Spec 1513326571 Weapon Mod v400.weapon.mod_damage 974142739
Empty Mod Socket 2323986101 Weapon Mod v400.weapon.mod_empty 3945646995
Targeting Adjuster 3228611386 Weapon Mod v400.weapon.mod_guns 510594033
Radar Tuner 371216963 Weapon Mod v400.weapon.mod_guns 510594033
Freehand Grip 736000386 Weapon Mod v400.weapon.mod_guns 510594033
Icarus Grip 941997506 Weapon Mod v400.weapon.mod_guns 510594033
Counterbalance Stock 1588595445 Weapon Mod v400.weapon.mod_guns 510594033
Backup Mag 3336648220 Weapon Mod v400.weapon.mod_magazine 4087004818

I think it's a little silly that an empty mod and damage mod have a different category hash, but I can kind of understand why. However, it makes no sense why Backup Mag is completely different and in it's own category.

floatingatoll commented 6 years ago

I would guess it has to do with magazine scripting being especially special (since D1).

vthornheart-bng commented 6 years ago

Just as an update on this, I've had 0 time to work on the aforementioned article. I'm hoping to loop back to this as work begins to settle down for the holiday season. My plan is to bring back the winter wishlist and make the creation of this document a priority on that list.

thepoofy commented 4 years ago

What's the current best documentation or example of Plugs? Trying to build a tool that relies heavily on this functionality and the caveats with the far simpler Perks are giving me migraines.