Unique item data - Githubissues

Eruyome commented 7 years ago

Hey, TradeMacro dev here.

Inspired by your little project here I thought maybe I could get the unique item data from the ggpk instead of scraping the poe wiki. I even thought about asking you if you could also export that data since I don't know any python and don't like it either and would probably write something from scratch with nodejs. I thought is wasn't much work. After taking a quick look at the ggpk file and dat files I have not the slightest idea how to get a complete unique item, even something as basic like this:

{
    "name" : "Araku Tiki",
    "mods" : [{
            "name_orig" : "You gain 100 Evasion Rating when on Low Life",
            "name" : "You gain # Evasion Rating when on Low Life",
            "values" : [100],
            "isVariable" : false
        }, {
            "name_orig" : "+(30-50) to maximum Life",
            "name" : "+# to maximum Life",
            "ranges" : [[30, 50]],
            "isVariable" : true
        }, {
            "name_orig" : "+(20-30)% to Fire Resistance",
            "name" : "+#% to Fire Resistance",
            "ranges" : [[20, 30]],
            "isVariable" : true
        }, {
            "name_orig" : "1% of Life Regenerated per second while on Low Life",
            "name" : "#% of Life Regenerated per second while on Low Life",
            "values" : [1],
            "isVariable" : false
        }
    ]
},

Is it really that complicated or am I missing something?

brather1ng commented 7 years ago

Hey!

Unique data isn't in the ggpk (at least not client side). They removed it at some point because people datamined upcoming uniques.

However, the wiki has an API, so you don't need to scrape it. I recently got unique data from there, see the result xml file and the UpdateDB folder in PoESkillTree.

Eruyome commented 7 years ago

Ok didn't know of that API, have to take a look at that.

Thx :-)

Btw, I'd also like to replace the data that ItemInfo uses for the mods and their tiers. I opened a discussion about it here https://github.com/aRTy42/POE-ItemInfo/issues/63. Is parsing that data to something like this https://github.com/aRTy42/POE-ItemInfo/tree/master/data as complicated as aRTy42 makes it out to be? I believe you're a bit more familiar with all of this, but I can get this probably also from the wiki API, right?

brather1ng commented 7 years ago

It doesn't have much documentation, but it has a nice sandbox. You can see the possible predicates by clicking "Browse properties" in the sidebar of a item page, e.g. browse properties for Abberath's Hooves.

Getting that to an identical format is probably not easy. An idea on how you might be able to work with the RePoE files directly (I'm calling the stat line that is on the item and for which you want to know the tier "translation"):

find all stats in stat_translation.json that match the translation (replace values with placeholders)
find all mods in mods.json that contain the stat's ids (as stats[i].id)
select the mods where the value lies in stats[i].min and stats[i].max (note that some values are modified while being translated, see stat_translations.index_handlers)
ignore mods with a generation_type that is not prefix or suffix
ignore mods where domain doesn't match the item
ignore mods that can't spawn on the item (using spawn_tags, you need to know the tags of the item)
you are hopefully left with exactly one mod now
get the tier by selecting all mods with the same generation_type, domain and group and ordering them by their stats values

(this will not catch essence and master crafted mods)

Getting the tags of an item: query the wiki API with Has name::typeLine of your item as condition and let it return Has tags or map the item's properties manually to tags.

I am planning to get the mods into a preprocessed format that might make this easier, but I can't tell you when I'll do that.

I don't think you can get that data from the wiki API.

Eruyome commented 7 years ago

Thx for the detailed explanation. Fortunately there's no reason to rush the affix data parsing/export. Those files are being manually compiled/maintained since the first version of ItemInfo and don't change much, so no need to hurry. I'll just take my time to think about solutions and keep an eye on your updates.

I've noticed that you guys over at the offline SkillTree use exilemods js-data to get the affixes, are there any plans to change this to for example RePoEs data (similiar to what you exlpained)?

I believe I could use your affix and unique data instead of getting that myself from the wiki or somewhere else, though there would be a few questions:

Are there any plans to change the xml structure in the near future?
How often is this data being updated? Client-side (for example when starting the program) or manually by you for developing purposes and releases?
Are there any reasons you would not recommend depending on your data?

Using those xml files and parsing them to json with a few changes would be by far the easiest and fastest solution for now, so I may be interested in that.

brather1ng commented 7 years ago

I've noticed that you guys over at the offline SkillTree use exilemods js-data to get the affixes, are there any plans to change this to for example RePoEs data (similiar to what you exlpained)?

Yes, that's what I'd do the more preprocessed format for..

Are there any plans to change the xml structure in the near future?

Affixes: If I do above, it will change. But not before that (except should Path of Exile updates bring significant changes).

Uniques: The file is only a few days old since I just started implementing unique item crafting. I don't plan to change it atm, but it might be a good idea to wait until I'm done and my branch is merged.

There might be changes at a later point so that it will also include mod ids, but that'd be an extension of the current structure and no replacement.

How often is this data being updated? Client-side (for example when starting the program) or manually by you for developing purposes and releases?

The xml file is included in releases, it's not updated client-side at all. Optimally it would be updated after every Path of Exile update, but we did forget that in the past.

Are there any reasons you would not recommend depending on your data?

Affixes: exilemods is not really maintained anymore, so game changes mostly have to be hardcoded. So that isn't much better than what you have atm. Since your situation is fine, I'd not use our affix file until its based on ggpk data/RePoE.

PoESkillTree's game data files in general: There might be changes to the file structure at any time as the files are meant for internal usage in PoESkillTree. As the files are not a part of public releases, you'd have to rely on commit messages to see changes. But I don't think that will happen often.

Eruyome commented 7 years ago

Ok, thanks for the information. I'll just take my time and keep an eye on further developments. The data I have is sufficient and while scraping the wiki is somewhat unstable (html changes could easily break it), it works for now. But I'm always open for better solutions.

brather1ng commented 7 years ago

I just noticed PoESkillTree's affix file is sligthly outdated. So no, don't use that one.

brather1ng commented 7 years ago

The unique stuff in PoESkillTree was merged, so you can find the file at https://github.com/EmmittJ/PoESkillTree/blob/master/WPFSKillTree/Data/Equipment/Uniques.xml now.

If you don't want to use it because it is potentially unstable, loading uniques from the Wiki API is really no big deal.

Eruyome commented 7 years ago

Thanks for the heads up, I already rewrote my wiki scraper to get the data from the API instead of scraping the html. Probably would've been a bit less work if I had used the PoESkillTree data but it works.

brather1ng commented 7 years ago

Alright, it's also much more flexible if you don't rely on PoESkillTree's data.

I'll keep this open, maybe GGG will give us access to unique data at some point in the future.

brather1ng / RePoE

Unique item data #1