Add Steam recent/all review count, positive/negative ratio and release date filters

[andrasfuchs]

This extension is still one of my favorites, and it's a lot of fun to use when I look for a game to play with my gamer friends.

If it was not too difficult add, it would be great to have a few more filters for reviews and game release date.

Ideally we would have a filter for minimum review count, minimum positive/all ratio, recent x-day or all-time review selection.

Another filter option would be the release date, ideally with a range slider stopping at not just the years, but quarters too, so that we could filter the games for example between 2021Q3 and 2020Q1.

Thank you for you efforts!

[Nebukam]

Date has been requested over at reddit too, so that's definitely something I'll add!
I'm currently busy with another project and the game finder needs some adjustments so I should get back on that around the end of April ^_^

Review filters on the other hand is tricky. Not only I failed to crawl the data, but it's also a very "living" stat, and the app is working with a (not updated often enough) cache. So while it's possible to add the feature, it'll be very misleading for recently released games. I'll look into it nonetheless, but don't get your hopes up for this one ;)

[andrasfuchs]

Great, thank you for the explanation!

Would it be possible to fetch review data just like you do with the price on the client side?

Would it be easier to get the data from steamdb.info?

My guess is that we couldn't make review filters for the games this way, but we could still see the up-to-date ratings if we opened the game details/price panel.

I don't know if it is possible, it's just a thought ;)

[Nebukam]

It would be possible to fetch the review data "on-demand" but won't be usable along with filters. Technically I could fetch all the data on the client side, but that's way too many requests to the Steam website and will end up with a 429 error super fast (that's why I ended up using a cache). It also makes everything very sluggish ^^

So yes, definitely a go for the on-demand.

Regarding streamdb.info, they don't allow anyone to fetch/crawl their data... Honestly I don't even know how they manage to aggregate it in the first place!

[andrasfuchs]

What if we (the users of your extension) sent the review data back to you after we fetched it on the client side? ;) In that way you could update your review cache over time on the games that are viewed within your extension.

[Nebukam]

That would work if I had a server to work with, I'm currently generating the cache on my local machine and then pushing https://github.com/Nebukam/steam-db from which I fetch data ^^'
This is much more barbaric & cheap than you think x)

[andrasfuchs]

What if you had a server with a web API? :) I could do you the server side programming and host it on a server.

If you think it would make sense, let me know and I put together something as a start to see if we could cooperate. ;)

[Nebukam]

Sure! That's be definitely interesting to do :)
I still have quite some work to go through before reaching that point, but I'll definitely poke you here when I get to that.

What do you have in mind as far the API goes?
Also looking at my answer above, I would still only be able to fetch data on-demand, per-game to circumvent the 429. The PWA version of the app is already using a free Glitch server in background to circumvent CORS but it's super slow to boot up, and it's not immune to 429 either.

There's also a bunch of things to account for when crawling the product page (which would be required unless we can partner up with steamdb.info, which sounds highly unlikely) :

• locale (you would need to make sure the server is fetching with a US origin as content is localized. Even if you can specify the locale in the url, it's overwritten server side based on request origin)
• age gating is a pain because you can't use stored cookies (this is an issue with the extension if you're not connected to steamcommunity.com, price check on age-gated product fails, same for the PWA)
• The only review data (and that could change) that is crawlable is this bit :
  
  as the more detailed customer review are loaded dynamically after initial page load if you scroll there. (so can't be crawled)
• Since this is all based on crawling, any update from Steam to the HTML code breaks everything (another reason I try to cache as much as I can) -- this happened recently when they revamped the friends web UI...
• Probably forgot about some other issues to keep in mind ^^

[andrasfuchs]

Obviously the idea of the API is very flexible in my head at the moment, but I had the following ideas in mind:

• Our API would be private, accessible from your extension only by its users
• The API would aim to get data from and update data in your steam-db
• Crawling would still be done in the extension, not in the backend
• The cache in the browser would be synced with our backend using the API
• The API would have methods like GetAppInfo(...), UpdateAppInfo(...), GetAppReviews(...), UpdateAppReviews(...) etc.
• The backend behind the API would keep the structure you have in steam-db with the JSON files
• Before your extension crawled a page related to a game, it could ask our API if we have a recently updated file about that game (probably with GetAppInfo(...))
• If the info is not available or if its outdated, the client would start the crawling as it does now for every game (I guess)
• After a successful crawling it would call the UpdateAppInfo(...) API method to indicate to our backend that there is new app info available, and we would update our version in steam-db
• We could do this for Steam app info, reviews, thumbnails, co-optimus data or we can even try streamdb.info if we are allowed
• Thanks to our users we would always have the most recent version of our database in our JSON files on our backend that we could publish in the steam-db repo every month or so

These things above would not solve the challenges you have with the crawling/changing HTML pages on the client, but it would allow us to think about more advanced features in the future.

I'm curious if you think it's exciting enough to put in the hours, and make this happen. I know that it would take a few weeks for me to do the API, but I think the end result would be awesome :)