search

lmmfranco / nintendo-switch-eshop

Crawler for Nintendo Switch eShop
Apache License 2.0
519 stars 82 forks source link

[QUESTION] Getting all games and a specific game #181

Closed HOllarves closed 4 years ago

HOllarves commented 4 years ago

Hey guys!

I've been reading the documentation and all the issue post in this repo.

First of all I think the maintainers of this repo are true rockstars as they've been very helpful and present in the maintenance of this lib.

Now to the actual question.

I'm getting a combine response of +5000 games when fetching all games using getGamesAmericaand getGamesEurope, but still I believe there are games missing... or perhaps not. The problem is that looking for a specific game seems quite hard (at least I've been bashing my head for a couple of hours now with no real results)

So to give a bit of context, I'm using RAWG video game API and exposing it through a GraphQL interface. I'm able to get Steam, Xbox and PSN IDs from the store urls given to me by RAWG. Unfortunately, Nintendo do not put their game IDS (NSUID) as URL params in their store.

So, for example when querying for Fire Emblem: Three Houses, I get a url like: https://www.nintendo.com/games/detail/fire-emblem-three-houses-switch

For PSN it would be: https://store.playstation.com/en-us/product/UP4497-CUSA00527_00-0000000000000002where UP4497-CUSA00527_00-0000000000000002is the game's unique ID. Basically the equivalent of 'NSUID` for Playstation Store.

Unless there's a way to query by slug or something I'm missing, the only thing I though that could be the same across all APIs is the game's name.

So, I ended up doing some blasphemous things like:

  const games = await Promise.all([eshop.getGamesAmerica(), eshop.getGamesEurope()])
  const rawData = games.flat().filter((g) => g.title === title)
  const { price_regular_f: usPrice, msrp: euPrice } = rawData.flat()
  return {
    // eslint-disable-next-line no-nested-ternary
    usPrice: usPrice ? usPrice * 100 : 0,
    euPrice: euPrice ? euPrice * 100 : 0,
    rawData,
  }

I'm trying to get prices for a game in all stores. Let's take as an example The Witcher 3, one of the few true Multiplatform titles out there. It's available in all modern platforms.

However, when title === 'The Witcher 3: Wild Hunt' I'm not getting a response. Still when title === 'Fire Emblem: Three Houses' it does work.

Now, I'm truly curious about the discord bot @Favna has, because for what I've seen he gives them a simple string like Pokemon sword and the bot is capable of matching the actual game and returning it's pricing data.

If I find a robust way of searching a specific game I'm planning to have the result of await Promise.all([eshop.getGamesAmerica(), eshop.getGamesEurope()]) to be stored in a DB, and refreshing it once every few hours, just to make querying a lot faster.

Any ideas?

Thanks!

favna commented 4 years ago

First of all, thank you for the awesome compliment and your interest in this project!

For Skyra (not sure if you were looking at Ribbon or Skyra, but Ribbon has been superseded by Skyra in all ways) we indeed allow users to search for specific games, the source code (which is Apache licensed) can be found here: https://github.com/skyra-project/skyra/blob/master/src/commands/Tools/Websearch/eshop.ts

You are free to use code from this file as long as you properly attribute it by having this in the file:

/**
 * Copyright 2019-2020 Antonio Román
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 *    http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

As for searching by NSUID, that is possible at least for Nintendo of America by using the above linked code, for example Witcher 3's NSUID gives us 1 result:

image

Nintendo EU (https://nintendo.eu) will be a whole different problem however as they do not use Algolia, at least publicly. For Nintendo of America (https://nintendo.com) you can go to the website, open your devtools to the network tab and you'll see the Algolia queries happen as you search. No such thing happens for Nintendo EU. That said, the query we do in this library can be pretty easily modified for single games, as you might have seen there is a q: '*' being passed to the query parameters, replacing this with a game title gives... somewhat limited results. Sadly they seem to do an "include" result so http://search.nintendo-europe.com/en/select?fq=type:GAME AND system_type:nintendoswitch* AND product_code_txt:*&q=The Witcher 3 Wild Hunt Complete Edition&sort=sorting_title asc&start=0 still returns 1200 results because of titles including The, Witcher, 3, ..... or Edition (you get the idea) and searching by NSUID seems to not be possible, or the query has to be modified in some weird way as just dumping the NSUID in the q queryparam doesn't work.

So I can make 2 suggestions:

  1. Use this library and store the data in a database capable of elasticsearch another great fuzzysearcher, for example your own implementation of Algolia or Sonic (it's in Rust, https://github.com/valeriansaliou/sonic) . Then when your users use your frontend it sends a query to a backend which uses the fuzzysearcher to your query your database. That returns some data that you pass back to the user on the frontend.
    • note: I strongly recommend a microservices architecture for this. I see at least 4. Frontend, gateway, fuzzysearcher and database.
  2. Only store game names from this library and use those to query for all their data using a single game search system as mentioned above.

Option 2 comes with the major caveat that Algolia is only applicable to Nintendo USA and not Nintendo EU and as mentioned EU doesn't seem to support lookup-by-NSUID. Therefore I think option 1 is the better one, however it will require more work on your end both in coding and setting up infrastructure.

Lastly specifically about this:

However, when title === 'The Witcher 3: Wild Hunt' I'm not getting a response. Still when title === 'Fire Emblem: Three Houses' it does work.

The aforementioned query for Skyra does work for getting that title: image

iRamos99 commented 4 years ago

@HOllarves In my project, I store all results from getGamesEurope() and custom getGamesAmerica(), that pulls all games, locally. The inconsistency with NSUIDs and game's title per eShop region is sad, so, I do something similar to what you're doing to dedupe results. I remove all special characters from the game titles and for the most part, I get a good de-duped set (Still have some dupes).

function stripString(s) {
    return (JSON.stringify(s).replace(/[^a-zA-Z0-9]/g, "").toLowerCase());
}

I update the local database nightly. For the price, I pull that in real-time using this lib. I have a simple API to get lists from my local db:

https://www.nsgreviews.com/list/query?search=The+Witcher&limit=1

I can provide the custom getGamesAmerica() if you'd like!

HOllarves commented 4 years ago

Hey guys! Thank you for your responses. It's pretty clear to me this is no easy feat, but thanks to your help I'm quite confident I can pull it off.

I'm going with option 1 in the long run, but to get started I'll focus on using the algolia url to fetch games from the US shop while I get going with the local DB infrastructure suggested by both of you.

I'm getting good results already.

Stay safe!

raekul commented 4 years ago

Hey all,

I've been tinkering and I don't know if this is of any help to anyone.

For europe at least, it seems like there is a way to get info about a particular game, providing you have the nsuid handy. 🙂 If you modify the fq query param (like below), you can get a collection of documents containing just the documents you are looking for.

?fq=nsuid_txt:70010000000023

You can also pass in a comma separated list of nsuid's and it'll return the documents back to you it seems.

I was trying to find a way to get info about a specific game, for europe at least and the only other api is some "contents" endpoint but it requires you to be logged in which is slightly annoying 😛