search

RePod / psdle

Improving everyone's favorite online download list, one loop at a time.
https://repod.github.io/psdle
MIT License
194 stars 14 forks source link

Download list omits some items since v2.048 #20

Closed ghost closed 9 years ago

ghost commented 9 years ago

Hi, i currently have 3398 entries in my download list (3392 are gaming-related stuff, 6 are movie trailers) however the list generated by PSDLE tells me that i have 3396 (3395 without "Catalog" enabled, and the third missing item is "DYNASTY WARRIORS 8 Empires Free Alliances Version" for PS3) out of 3398 items.

I don't know about the other missing items. I'm using the IT store.

RePod commented 9 years ago

Here's a list of commits that update the Userscript.

Is there a difference between:

I'm not seeing anything between updates that would result in this happening.

RePod commented 9 years ago

While viewing the store page: 

var a=gEntitlementManager.getAllEntitlements(), bad=0, valid=0, w=0;
$.each(a, function(i,o){
    if ((o.VUData) || (o.drm_def && o.drm_def.contentType == "TV")) { bad++ }
    else if (o.drm_def || o.entitlement_attributes) { valid++ }
    else { w++ }
})
console.log(a.length,valid,w,bad)

This bit of code loops through the Entitlements list (which holds content you're entitled to including download list content, including non-download list items) similar to how PSDLE does and uses the same content checks (but logs the amounts).

RePod commented 9 years ago

Any news?

Been busy, will get to it eventually™.

Also why sometimes when i start the script, all i get is 0/0 results?

This is an intermittent issue. If I had to guess the Entitlements are not loaded for whatever reason (store side), I can't do much about this. Refreshing the page seems to work.

ps: i swear the issue with the omitted items already happened in the past but was fixed...

In my other reply I gave a list of Userscript updates. The most recent ones (within the past 3 months) haven't done anything to the table or download list generation so I don't have a good starting place to look.

You said the problem only occurs with the Catalog enabled. If you could export the list for both and narrow down the results further (more than "DYNASTY WARRIORS 8 Empires Free Alliances Version") that'd also be great. If it's an issue with the data retrieved by the Catalog that would cause them to pass the first step (download list creation) then fail the second (Catalog data).

RePod commented 9 years ago

No, the problems happens with Catalog enabled AND disabled.

Whoops.

Are you sure the results i posted 2 days ago are enough? It's just a bunch of random numbers for me.

In order, the numbers 3569 3398 171 0 are:

You originally posted you were getting 3398 results (including omissions) and that snippet returned 3569 3398 171 0. The 3398 here is the amount of valid items PSDLE sees, add that to the 171 other items to get the total: 3569. This means the snippet (and by extension, PSDLE) is seeing and processing everything on the Entitlements list as it should. Also for future reference is 3398 the number shown on the original download list page?

After the relevant items on the Entitlements list are cached by PSDLE it then generates the table with the available filters (system, category if Catalog is enabled) which are all enabled by default. The primary reason an item wouldn't display is if for some reason it failed to match the enabled systems or categories enabled.

RePod commented 9 years ago

I made a small modification to assist in debugging this issue.

Please execute this version with the Userscript disabled.

To use (same as the previous snippet):

This modification first checks when content is being marked as invalid (which is logged, so keep the console open), then forces them to be marked as valid which should help identify the problem.

RePod commented 9 years ago

Anything in the console?

RePod commented 9 years ago

SHA-1 and XHR requests are store-related. I went ahead and made an actual debug command to compare the data PSDLE is using.

Please execute this version, as before with the Userscript disabled.

When viewing the list, in console execute: repod.psdle.debug.difference(true)

This will compare the two lists PSDLE uses ("everything" and "the currently displayed items"), find the differences, and force those to be displayed.

RePod commented 9 years ago

The first one is getting back this information from the Catalog:

repod.psdle.debug.checkParse("EP2005-PCSB00596_00-SOLDNERX2LBCROSS")
Object { deep_type: "downloadable_game", rating: "4.93", images: Array[10], videos: Array[0], metadata: Object, long_desc: "Divertiti ancora di più con la vers…" }

The second one:

repod.psdle.debug.checkParse("EP2005-PCSB00596_00-SOLX2LASTCHAPTER")
Object { deep_type: "add_on", rating: "4.70", metadata: Object, long_desc: "Preparati a giocare a uno sparatutt…" }

Notably, the second is getting a lot less information. Currently PSDLE only overwrites the determined system if it's a PS1 or PS2 classic, so if it doesn't already have that information from the Entitlements that's probably the cause (having no system associated).

Please execute the following code (PSDLE doesn't matter):

$.each(gEntitlementManager.getAllEntitlements(),function(i,obj) {
    var c = (obj.drm_def||obj.entitlement_attributes)?!0:!1;
    if (c) {
        if (obj.id.indexOf("EP2005-PCSB00596_00") > -1) {
            var platform, pids = 0;
            if (obj.game_meta) { platform = ["PS4"] }
            else { 
                platform = []; pids = obj.drm_def.drmContents[0].platformIds;
                $.each({"1":KamajiPlatformFlags.PS3,"3":KamajiPlatformFlags.PSP,"8":KamajiPlatformFlags.VITA}, function (t,u) { 0 !== (pids >>> 1 & u >>> 1) && platform.push(KamajiPlatforms[Number(t)]); });
            }
            //Remove personal information (such as dates) and extraneous URLs.
            var safe = JSON.stringify(obj, function(k,v) { if(/[\d\-]+T.+Z$/.test(v) || /^http/.test(v)) { return "~" } return v; });
            console.log(i,platform,pids,safe)
        }
    }
})

This goes through your Entitlements, finds the problematic items (in this case "EP2005-PCSB00596_00") and logs their Entitlement data. At minimum you should get two results. Don't worry if the results look like a mess.

A sample of the output would be:

200 Array [ "PS3" ] 2147483648 "{"active_date":"~","active_flag":false,"drm_def":{"active_flag":false,"autoDownload":false,"availableDate":"~","contentName":"GRID 2","contentType":"GAME","downloadableStatus":"ready","drmContents":[{"bitrate":0,"contentId":"UP4001-NPUB31075_00-00000000000GRID2","contentName":"GRID 2","contentSize":6207017184,"contentUrl":"~","downloadType":0,"drmContentType":1,"drmType":2,"gracePeriod":1200,"platformIds":2147483648,"position":0,"spName":"Codemasters Inc.","titleName":"GRID 2"}],"duration":0,"entitlementId":"UP4001-NPUB31075_00-00000000000GRID2","episodeNumber":0,"expiration":"~","firstPlayExpiration":0,"firstPlayExpirationHours":0,"image_url":"~","media_type":0,"productId":"UP4001-NPUB31075_00-00000000000GRID2","release_date":0,"rewards":[{"available_count":0,"count_until_expiration":0,"expiration_date":"~","price_after_discount":0,"promo_sku_id":"IP9101-NPIA90005_01-PLUS1MONTHPACKAG-UX01","rewardType":2,"reward_id":"IP9101-NPIA90005_01-RWD-974","reward_status":1,"time_until_expiration":120}],"runtime":0,"salesType":"own","seasonOrder":0,"startedStreaming":false,"year_release":0},"entitlement_type":2,"feature_type":0,"id":"UP4001-NPUB31075_00-00000000000GRID2","inactive_date":"~","is_consumable":false,"license":{"entitlement_id":"UP4001-NPUB31075_00-00000000000GRID2","expiration":"~","feature_type":0,"infinite_duration":false,"start_date":"~"},"preorder_flag":false,"preorder_placeholder_flag":false,"product_id":"UP4001-NPUB31075_00-00000000000GRID2","revision_id":1341187614729,"sku_id":"UP4001-NPUB31075_00-00000000000GRID2-U001","subs_flag":false,"use_count":0,"use_limit":0}"
RePod commented 9 years ago

As expected.

Comparing with my example: Array [ "PS3" ] 2147483648

To your results:

The contents of the array being the determined usable systems, the number being used to determine that.

In short, technically PSDLE is doing the right thing by not showing them (as they have no associated systems). Enabling the Catalog could potentially overwrite (or fill in) the missing system data.

There are a few options:

Keep in mind I have to think in the grand scale, I can't just solve this specific problem without considering what else will be affected.

What I want to know is how the official Download List handles this situation. I would assume it hides them as they have no available platforms.

If it's not too much to ask for, could I task you with checking for their appearances on the webstore's Download List and from a console? If I had to guess based on the information you provided, they should be between pages 114 and 116 (if at all) on the webstore.

Also, does the game have some sort of DLC option in-game? Where it may open a store front for purchase, but you also download said purchases from within the game (thereby not requiring a valid system to be associated).

RePod commented 9 years ago

I assume when you say Account Managment you mean the one off the store and on account.sonyentertainmentnetwork.com? The page numbers I provided were for the store's Download List, it's fine though.

When checking under Account Management, in the second column, "Type", what does it say for those items? Looking at mine (list in general, not similar items) I'm seeing only Game and Game Service. If those items are Game Service that's probably where the issue lies.

RePod commented 9 years ago

Anything new?

I've been looking into how the official Download List determines what to show, and how to show it. The main issue is according to your response the SOLDNER items are showing up in the official Download List. It's somehow determining the playable system despite its Entitlement data not containing any.

Also i wonder why certain PS3 content has no size in the PSDLE list?

Size is obtained from the Entitlements in raw bytes then converted (1KB, 1MB, 1GB...). If it's not there, or is 0, PSDLE can't guess. Accurate Download List behavior returns a size of "N/A" when it is unable to determine the size, I may modify PSDLE to do something similar.

Do they still count to the total?

Most likely. It'd be like adding 0.

If you want to help with the size issue, please use the latest version (run in console) which contains newer debug tools.

When ran, execute repod.psdle.debug.entitlement("Lords of Shadow") (replacing "Lords of Shadow" with any other problematic content).

RePod commented 9 years ago

Looking at those two entries they have a valid platformIds value (2147483648, PS3) but a contentSize value of 0. As far as PSDLE is concerned, the contentSize given is valid and cannot be determined otherwise (except for PS4 content which store their sizes elsewhere).

I went ahead and pushed this minor fix so when they have a value of 0 they're presented as "N/A" but still treated as 0. 1 2

ps: is the issue with that other game (Dynasty Warriors Free Alliances blah blah) the same as SOLDNER? Or is it different because of that "Game Service" thingy?

repod.psdle.debug.entitlement("Dynasty Warriors") would help with that. The issue SOLDNER is having is lacking platform data (which at the moment appears to be being pulled out of thin air by the Download List).

However, I am curious on a few things:

If there's a discrepancy between the Entitlements data and what is being shown on the Download List I can only believe it to be the third point. You did mention "[Cross-Buy]" was added to a title.

RePod commented 9 years ago

It seems the PS3 version of Dynasty Warriors has the same issue as the SOLDNER Vita items: no platformIds value. The PS3 version appears when using Catalog since it overwrites one of the stored platform values, but leaves the original untouched. This is best noticed if you try to open the window by clicking the item, which will fail since it still lacks proper system data.

Regardless, I added a few more debugging commands to inject Entitlements so I can work on it verbatim from my side: (Catalog enabled)

screenshot_1

As soon as I figure out how the Download List determines their systems I can implement the changes required in PSDLE.

I thought PSDLE includes more entries than the Download List because it includes the "Movies"?

You mentioned these earlier but I didn't ask about them. Are these actual video items (episodes, etc.) or things like game trailers and shows up as a category at the top: (Categories are only available with Catalog enabled)

screenshot_2

If they're regular videos (from the "TV" tab on the Download List), those should not be displayed.

RePod commented 9 years ago

Given the Dynasty Warriors game had the same issue as the Soldner games (no platform data) it seems the easiest fix is to wait for it to update on Sony's side.

I'm still figuring out how it gets the system for the Download List when it lacks this information, though. The PS3 version of Dynasty Warriors shows up on the webstore's (not Account Management) Download List, correct?

RePod commented 9 years ago

Using the latest version again (run in console), please execute repod.psdle.debug.entitlement("EP2005-","id") which should log all the SOLDNER items' Entitlement data.

I have a feeling the Entitlements were updated (aside from the now existing platform data), and should at least reflect the change and maybe other differences.

RePod commented 9 years ago

As I suspected. At some point the Vita entries for SOLDER were updated, as shown by the them getting platform data, but also from their _revisionid attribute changing:

Assuming the dates are irrelevant, during the course of finding out the issue their data was modified from Sony's side (and probably somewhere around the point you viewed them on the Download List or possibly Account Management). I want to say the PS3 version of Dynasty Warriors is in the same situation, but I'll continue to look into it.

RePod commented 9 years ago

The problem with Dynasty Warriors has been fixed, and you didn't have to do anything, lol.

Welp! I can finally push all these recent changes live I guess.

do you think it's possible to implement an item counter for the Download Queue just like official one?

If you mean viewable from the list (the Queue button), it's definitely possible. PSDLE used to have this before I rewrote the Queue support by replacing the "Queue" label with "# / # / #" (PS3/4/V). However, PSDLE's Download Queue support works a bit differently than the official one. The webstore polls the Download Queue status every few seconds while PSDLE only polls it when it is accessed. Unless I give that same behavior to PSDLE I'd have to track it internally (which isn't the hard part) but it would be less accurate.

PSDLE's Download Queue needs a lot of improvement to begin with, this is definitely on the list.

And a way to add multiple items at once?

Fun fact: originally PSDLE sent item in batches (when I first implemented Queue support).

However, at that time I was only sending items to PS3 so I didn't have to worry about the platforms. As soon as I find out a decent solution to handle multi-platform items (auto handle, send to one, ignore, etc.) as well as selecting all the desired items, batch support will be available.