search

IsThereAnyDeal / IsThereAnyDeal-Issues

16 stars 0 forks source link

GOG unbundling is interfering with import #158

Closed tfedor closed 3 years ago

tfedor commented 3 years ago

GOG.com is currently in the process of breaking the 1:1 relationship between catalog entries and library entries so users can install single games from packs like "King's Quest 4+5+6".

This is, in theory, a good thing, since it means that the "packages -> games" arrangement will more closely match the records you scrape from places like Steam, but it has resulted in quite a few titles in the import list being ignored as "unknown".

Because they haven't done any internal reverse-referencing, they don't provide any kind of catalogue link I could scrape.

However, the creator of the MaGog search engine has expressed interest in making a mapping table and, given that he offers a dump of the data he already scrapes, I think he'd probably be receptive if you asked to work with him to avoid duplicating effort.

[Issue created by ssokolow: 2015-03-22] [Last updated on bitbucket: 2015-03-29]

[Comment created by ssokolow: 2015-03-29] Names which appear identical but ITAD insists that the list-view version is in the collection but the shelf-view isn't:

Mismatches between shelf and list view which need to be aliased if we can't determine a proper primary key to export: (importing one does not import the other and neither is unknown)

Listed as "unknown" in shelf view because converting the slug-style ID doesn't yield the title:

[Comment created by tfedor: 2015-03-29] Okay, I relaxed conditions a little bit. Until now, game was "unknown" not only if it wasn't in my database, but also if it had no deal. Now I have started separately checking for unknown and for sale, so you may get both "unknown" or "no deal" notes.

Behaviour for unknown is left unchanged, so when there is unknown game it will not be checked by default. "No deal" will be checked if you are importing Collection but won't be checked if you are importing wishlist.

[Comment created by ssokolow: 2015-03-29] I'm getting quite a few unknowns which should have been resolved by the MaGOG dump.

For example, "Castles" and "Castles 2" (shelf view) and "CASTLES" and "CASTLES 2" (list view) are listed as unknown but the MaGOG dump does have " CASTLES *** CASTLES 2 " as the value for the unbundles_str column on "Castles 1+2".

The shelf and list views dump the text of the unrecognized entries differently, since the shelf can't get GOG's title text without a lot of XMLHttpRequests (and currently generates a fallback title from game IDs so the entries won't be omitted entirely), so not all of them match MaGOG's dump, but the lists of unrecognized entries both contain things like the aforementioned Castles 1+2 issue.

I own somewhere between 2/3 and 3/4 of GOG's catalog and here are the unknowns I'm getting (a combined and deduped view of both the shelf-view and list-view dumps), with the shelf-view title's prettified capitalization used when both show up with the same spelling:

NOTE:

Also, while examples like Castles 1+2 aren't recent unbundles, keep in mind that the unbundling is still an ongoing process, so you'll want to retrieve the MaGOG dump periodically to update your mappings as more games get unbundled.

[Comment created by tfedor: 2015-03-29] It should work without unknown now. I've used MaGOGs data to add unrecognized titles and mark original games as packages. Please check if it works for you.

MarkitkaM commented 3 years ago

Solved within the conversation.