search

facundoolano / google-play-scraper

Node.js scraper to get data from Google Play
MIT License
2.28k stars 624 forks source link

Still facing crashing when getting app details for [kr, tw, hk] markets #567

Open Aliaksandr-Kasko-JazzTeam opened 2 years ago

Aliaksandr-Kasko-JazzTeam commented 2 years ago

Description:

Example code:

const app = await gplay.app({
  appId: "com.moo.ssulmo.ssulmo",
  country: "kr"
});

Error message:

First error on descriptionText

TypeError: Cannot read properties of undefined (reading 'replace')
    at Object.descriptionText [as fun] (/app/node_modules/google-play-scraper/lib/utils/mappingHelpers.js:6:51)
    at /app/node_modules/google-play-scraper/lib/utils/scriptData.js:52:19
    at XWrap.f (/app/node_modules/ramda/dist/ramda.js:5955:28)
    at XWrap.@@transducer/step (/app/node_modules/ramda/dist/ramda.js:392:25)
    at _arrayReduce (/app/node_modules/ramda/dist/ramda.js:5137:46)
    at _reduce (/app/node_modules/ramda/dist/ramda.js:5167:24)
    at map (/app/node_modules/ramda/dist/ramda.js:5954:20)
    at /app/node_modules/ramda/dist/ramda.js:586:23
    at Object.f2 [as map] (/app/node_modules/ramda/dist/ramda.js:473:22)
    at extractFields (/app/node_modules/google-play-scraper/lib/utils/scriptData.js:40:14)

If I add nullsafe to description like this function descriptionText (description = '') { then I got an error in another place:

TypeError: Cannot read properties of undefined (reading 'split')
    at Object.fun (app/node_modules/google-play-scraper/lib/app.js:97:29)
    at app/node_modules/google-play-scraper/lib/utils/scriptData.js:52:19
    at XWrap.f (app/node_modules/ramda/dist/ramda.js:5955:28)
    at XWrap.@@transducer/step (app/node_modules/ramda/dist/ramda.js:392:25)
    at _arrayReduce (app/node_modules/ramda/dist/ramda.js:5137:46)
    at _reduce (app/node_modules/ramda/dist/ramda.js:5167:24)
    at map (app/node_modules/ramda/dist/ramda.js:5954:20)
    at app/node_modules/ramda/dist/ramda.js:586:23
    at Object.f2 [as map] (app/node_modules/ramda/dist/ramda.js:473:22)
    at extractFields (app/node_modules/google-play-scraper/lib/utils/scriptData.js:40:14)

But here I can't find a quick fix anymore.

Maybe there's some mismatch with mappings for these markets

knirpsfass commented 2 years ago

seems like the data key for the whole app is ds:5 instead of ds:4. Maybe it can be done via the ServiceRequestId by searching for the "service reqeust id" Ws7gDc (not sure if it does change tho)
https://github.com/facundoolano/google-play-scraper/blob/91a945b4e60d8e051154b6a9cb8cdf82adea2253/lib/utils/scriptData.js#L13-L25 The above code snippet already searches the data key for the given id but you'd have to provide the useServiceRequestId to every individual mapping of the mapping const.

Example AF_dataServiceRequests for an app from the US store. The app from the KR, TW, HK store would return something similar but ds:4 -> ds:5 and ds:5 -> ds:6

{...
   "ds:4":{
      "id":"Ws7gDc",
      "request":[
         null,
         null,
         [
            [
               1,
               9,
               10,
               11
            ]
         ],
         null,
         [
            [
               "com.studiolb.tamingmaster",
               7
            ]
         ]
      ]
   },
   "ds:5":{
      "id":"Ws7gDc",
      "request":[
         null,
         null,
         [
            [
               137
            ]
         ],

         null,
         [
            [
               "com.studiolb.tamingmaster",
               7
            ]
         ]
      ]
   },...
}

Unfortunate i don't have any time to implement and test it

Aliaksandr-Kasko-JazzTeam commented 2 years ago

UPD. I also found that it affects on th and au markets.

knirpsfass commented 2 years ago

@Aliaksandr-Kasko-JazzTeam i've opened #572 but i still need to test it and also fix the family genre mapping

Aliaksandr-Kasko-JazzTeam commented 2 years ago

@knirpsfass for now I can confirm that for th market your PR works fine