search

icue / SteamWishlistCalendar

Automatically tracks Steam wishlist release dates and publishes .ICS file that can be subscribed.
BSD 3-Clause "New" or "Revised" License
16 stars 2 forks source link

Random games getting dropped though they are NOT removed from the wishlist #11

Open icue opened 2 months ago

icue commented 2 months ago

For example,

They often make it back in the next run, but the drop is not expected in the first place.

icue commented 2 months ago

@Vinfall Curious if you have ever observed this issue. This seems pretty common on my end (almost happens every day).

Vinfall commented 2 months ago

Not exactly as I only have 20-ish games in wishlist (you have 180+). But sometimes the same game would be ordered slightly differently, which is probably related, e.g.

I suspect it's caused by one of these but have not tested yet:

icue commented 2 months ago

Thanks for the input!

I've seen order shuffles on my end as well. The game list returned from Steam should already be sorted by the app IDs of the games (at least that's what I observed). That said, I've added a manual sorting logic in 9e873719dacb7574082b831cbde525500921bf62, which hopefully will prevent the random shuffle.

Maybe it will (magically) solve this issue, too. If not, at least we ruled out a possibility. I'll keep monitoring the changes.

icue commented 2 months ago

Ok, I think I've found the reason. The URL https://store.steampowered.com/wishlist/profiles/{steamID}/wishlistdata/?p=0 is returning 99 games for me, instead of 100 - one game is missing. (On a side note, I've found out that using the https://store.steampowered.com/wishlist/id/{steamID}/wishlistdata/ URL for those who have a customized ID, the returned number of games per page would be far from 100 - I've seen numbers like 68, 37,18, etc. It seems so random.) Also, the returned list is stable in a short duration, but can change over time: Game A might be missing for this request and for an immediate request following it. However, after say 10 minutes, game A might come back and game B might be missing.

I think there's something funky going on with Steam's API. Perhaps the result returned depends on the caller/server's location, etc, but the games missing in my case are not even recently added to my wishlist, so cache is not accountable here.

This is bad, as it means there's no way to tell whether the removal of a game in the result is because the user has actually removed it, or beacuse the returned wishlist is not complete.

I think sending two duplicate requests one after another and combining the game list might at a very low possibility solve this issue, but it certainly is not an elegant solution.

Vinfall commented 2 months ago

It occurs to me that Steam API returns similar results (aka. inconsistent response) for user library regarding id & profiles. Sadly there is no such thing like https://steamcommunity.com/profiles/${STEAM_ID}/games/?tab=all&xml=1 to export everything at once for wishlist...

icue commented 2 months ago

Well, if the web API is just buggy like this, then I guess there's not much we can do.

I bumped into https://github.com/ValvePython/steam today, which might be able to get a user's wishlist, but it requires logging in / api key / stuff like that, which will make things way more complex. So I probably won't migrate to that just for 1 missing game.

Vinfall commented 2 months ago

Yeah, I discovered that a few days ago. Actually it cannot get wishlist data if I understand correctly. Searching wishlist in the docs only returned two functions that are removed.

There is also Steam Web API Documentation and Tester developed by xPaw (SteamDB guy), which requires developer key as well. The thing is, Steam Web API does NOT seem to have the required API at all... It has GetOwnedGames and GetWishlistItemCount (which returns only a number, not a list of wishlisted games) but not GetWishlistedGames or whatever.

icue commented 2 months ago

Ok. So the only way to get wishlist data is through the https://store.steampowered.com/wishlist... URL.

Actually, I found out that even visiting my wishlist from Steam webpage (either through Steam client or on my browser) suffers from the exact same issue - one random game is just missing. And that missing game is the same game missing from the JSON response. And this game changes over time.

I did another round of research and found that this bug has been reported by others as well:

I think the steps look like this:

  1. Scrape the wishlist webpage (https://store.steampowered.com/wishlist/profiles/XXX/), in addition to the JSON, to get g_rgWishlistData and g_Wishlist.rgVisibleApps. Comparing these two, I can get the misisng games.
  2. Scrape the store page of the missing games to get their release dates.

Maybe I'll work on it when I have the time, but defnitely not soon.

Vinfall commented 2 months ago

Or buy & play some games to keep wishlist below the 99 threshold XD, that's how I fix it.