search

South-Paw / warframe-item-list

👾 A list of all Warframe items that contribute to player mastery rank
GNU General Public License v3.0
13 stars 5 forks source link

Add API Update Scripts #40

Closed South-Paw closed 6 years ago

South-Paw commented 6 years ago

@Mulchman, @will-janz and @andersbrandt if anyone has a chance to look over these changes and provide feedback that'd be awesome.

Changes in v5.0.0

Remaining Work

Fixes #40, fixes #39, fixes #38, fixes #37 and fixes #35

South-Paw commented 6 years ago

I also have a question around whether I should implement fetching and resizing of the weapon images - however this will of course add heaps of megabytes to the file size of the package... thoughts, suggestions or ideas on this would be much appreciated.

andersbrandt commented 6 years ago

Nice touch with the missing items system, but the Venari Kavat does not contribute towards mastery rank. Keep the system though, might be needed in the future.

Mulchman commented 6 years ago

As far as I'm aware (and some others on Reddit, Venari does give Mastery): https://www.reddit.com/r/Warframe/comments/8dorwn/does_venari_have_mastery_associated_with_her/

On Sat, Sep 1, 2018, 11:45 andersbrandt notifications@github.com wrote:

Nice touch with the missing items system, but the Venari Kavat does not count towards mastery rank. Keep the system though, might be needed in the future.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/South-Paw/warframe-item-list/pull/40#issuecomment-417879465, or mute the thread https://github.com/notifications/unsubscribe-auth/ALNpqZiR_ZE-EoDGvrtxLUlWVS18mRp6ks5uWtXHgaJpZM4WWH90 .

South-Paw commented 6 years ago

Yeah, I was working from the wiki which says it does but isn't listed on player profiles: http://warframe.wikia.com/wiki/Kavat#Venari

https://i.imgur.com/fRIyjXh.png

South-Paw commented 6 years ago

@andersbrandt Regarding the acquisition data - is there any known endpoints or pages that provide a list of that information? I don't want to update things manually as it requires time and results in update delays

andersbrandt commented 6 years ago

I stand corrected :)

Regarding acquisition data - I changed my mind, that data are important, but this repo will be better off without manual steps for updates. I will add the data for acquisition in my own end instead.

Regarding images: I assume we are talking about adding a URL to an image, and not the actual image-file. If that's the case it is an most excellent idea :)

South-Paw commented 6 years ago

@andersbrandt I was thinking of adding the actual images to the /data/img folder.

Providing an endpoint/url for them is easy but not all the images that the mobile endpoints provide are sized correctly. They have to be run through a resizer for them to appear correctly so I was thinking of doing that and providing the resized images.

I'll look into exporting them in alternative means (such as data urls rather than pngs) and see what the bundle size comes out like?

Mulchman commented 6 years ago

I don't think the images themselves should be a part of this. A URL to the image, sure, but the actual data, no.

If you've ever messed with Destiny 1/2 data (and their APIs) they just provide the URL to weapons/armor/gear/etc.

South-Paw commented 6 years ago

I've been messing around with the best way to do images for the repo the past week and I've got a few options available...

No images/urls

Without any image information being added to the data/json/... files, the base file size is 105KB (136KB on disk).

URLs only

Its super easy and lightweight to do this as I'm basically only adding a URL to each item object. The file size of the data/json files becomes 156KB (180KB on disk).

Implementing applications would need to fetch these items themselves from the endpoint as well as do their own resizing on the images as they are all 512x512 images but need to be resized to 512x342 to actually display correctly.

The item objects would look like the following;

{
    "id": "/Lotus/Weapons/Corpus/LongGuns/CrpShotgun/CrpShotgun",
    "name": "Arca Plasmor",
    "type": "Primary",
    "category": "Shotgun",
    "masteryRank": 10,
    "imageUrl": "http://content.warframe.com/MobileExport/Lotus/Interface/Icons/Store/CrpShotgun.png",
    "wiki": "http://warframe.wikia.com/wiki/Arca_Plasmor"
},

Image Data URLs

This is a heavy update operation and results in data urls being available in each item object. The file size of the data/json files becomes 75.6MB (75.6MB on disk).

However every one of these data urls has an image that is already resized and prepared for use in an application. The downside is there's no file for them in the repo, only a data url and the files become really really really long so diffs might be an issue.

{
    "id": "/Lotus/Weapons/Tenno/Archwing/Melee/ArchSwordHook/ArchHookSwordWeapon",
    "name": "Agkuza",
    "type": "ArchwingMelee",
    "category": "Archwing Melee",
    "masteryRank": 0,
    "imageUrl": "http://content.warframe.com/MobileExport/Lotus/Interface/Icons/Store/ArchHookSword.png",
    "imageData": "data:image/png;base64,iVBORw0KGgoAAA....",
    "wiki": "http://warframe.wikia.com/wiki/Agkuza"
  }

Saving image files to data/img folder (after resizing)

Another heavy process and results in resized image files being available in a folder (and referenced by each image object). The file size of the data/... folders files becomes 61.5MB (62.5MB on disk).

And if I run the images through a imagemin after saving them, the file size becomes 16.2MB (17.2MB on disk) which in the end seems pretty reasonable.

South-Paw commented 6 years ago
File Size on Disk Resized?
No Images 105KB 136KB n/a
URLs Only 156KB 180KB no
Data URLs 75.6MB 75.6MB yes
Image Files 61.5MB 62.5MB yes
Image Files (+ minification) 16.2MB 17.2MB yes
Data URLs from minified images 20.0MB 20.1MB yes

So @Mulchman and @andersbrandt given the above, what are your thoughts?

Bear in mind, if I only provide urls the images are not sized correctly so the url is as good as useless (in my opinion only though).

I'm very tempted to add the resized and minified images to the repo at this stage.

andersbrandt commented 6 years ago

Image Files (+ minification) sound about right. There is the option of scraping Wikia for images, i do that in my repo. That procedure is rather time-consuming though.

Have you tested to convert to image data after minification?

South-Paw commented 6 years ago

@andersbrandt just made some quick changes and tested data urls from the minified images; 20.0MB json files (20.1MB on disk) - have updated table too.

You're also talking ~71,000 or more characters per image which is insane.

I'm keen to stick with the files at this stage.

South-Paw commented 6 years ago

@andersbrandt @Mulchman I could do with some help on the travis ci front if either of you have time or suggestions on how to get it running again.

My hunch is simply that the ci doesn't have lua installed or on the path correctly - however I don't know much about linux or how to solve this particular issue.

andersbrandt commented 6 years ago

@South-Paw Aight, i will get some help to look into that

andersbrandt commented 6 years ago

https://github.com/victorliu/S4/issues/45

https://github.com/victorliu/S4/issues/46

https://github.com/kikito/middleclass/blob/master/.travis.yml

https://github.com/moteus/lua-travis-example

South-Paw commented 6 years ago

Right well I'm at an absolute loss on this and nothing I've been able to pull up via google has helped.

I've raised a stack overflow question (https://stackoverflow.com/questions/52366146/node-js-and-travis-ci-cant-find-lua-h-when-installing-node-lua-package) and hopefuly someone who knows what the heck is wrong can come save this PR.

resir014 commented 6 years ago

CI fixed. #41

Edit: Sorry it took so long btw @South-Paw. Work and all that.

South-Paw commented 6 years ago

Shoulda, coulda, woulda squashed the commits when I merged that ... but not really worried 👍

Thanks so much for your help @resir014.

I'll go about raising test coverage now that CI is working again - glad to see that getting it building on CI ended up being so easy 😆

South-Paw commented 6 years ago

@Mulchman and @andersbrandt do you have any comments on this or are we all happy if I go ahead and merge this PR? 😄

hylte-brandt commented 6 years ago

Just go man, incredible work, this project is now so much better :)

hylte-brandt commented 6 years ago

My only objection is Venari, the companion to Khora. The actual number of ranked items will not correspond to the number displayed in-game if Venari are still among the items. However, it's a small task to remove that entry for projects that use this repo as a base.