n0madic / google-play-scraper

Golang scraper to get data from Google Play Store
GNU Lesser General Public License v3.0
78 stars 20 forks source link

Incomplete Permission Retrieval: Missing 'Other' Segment #23

Open jcastilloa opened 1 year ago

jcastilloa commented 1 year ago

First of all, I want to thank you for developing the 'google-play-scraper' library in Go. It has been extremely useful. However, I've noticed that the crawler is unable to retrieve all the permissions of an application from Google Play.

For example:

https://play.google.com/store/apps/details?id=com.einnovation.temu&hl=en

Selección_1472

jsonArray:

[[["Microphone",[null,2,null,[null,null,"https://play-lh.googleusercontent.com/daUjqbSOr2QpaqXS2HQbNzYzzqN2yWGzM_7AZxwFaWLT7_kIhX95HKi_HSpjeeQDOmFMENZxJqblbu_4qg"]],[[null,"record audio"]],["perm_camera_mic"]],["Photos/Media/Files",[null,2,null,[null,null,"https://play-lh.googleusercontent.com/pHtIujPWxciAZcfYSwlrGGq14Z984rKLMgcm9RPATLiOlbrWy-tVlelEWgED7gpktgcD1tZizVeHiO5fkw"]],[[null,"read the contents of your USB storage"],[null,"modify or delete the contents of your USB storage"]],["perm_media"]],["Wi-Fi connection information",[null,2,null,[null,null,"https://play-lh.googleusercontent.com/U-_SG8pHTsqU_IyZTGQRkVMdLaAUeq1OnKGrB06KHF1z7vkkIQK3iF0HcbfTe1RnGlh-ajnZkbphl2W3Gdk"]],[[null,"view Wi-Fi connections"]],["signal_wifi_4_bar"]],["Storage",[null,2,null,[null,null,"https://play-lh.googleusercontent.com/aWNKQedLTpw6u6yyMjQObmuoKu67A1czWnIcvID86oAmMT02r5mNdRn6l9ZN2t2MIyH6tNy-01v7ukeQ"]],[[null,"read the contents of your USB storage"],[null,"modify or delete the contents of your USB storage"]],["storage"]],["Camera",[null,2,null,[null,null,"https://play-lh.googleusercontent.com/xbP_oGuJ21iG29iVh0p-UIZPzi_fYj8PMYiqDd9-LvaZ_a1tRcwp0I2-arfXvgX9YtfZTTaqwcLRWPNQM_c"]],[[null,"take pictures and videos"]],["camera"]]],[["Other",[null,2,null,[null,null,"https://play-lh.googleusercontent.com/pkKXoPl5q7n8T0s7KREtdvUZn1PLRgx-Ox0t4tkO8af4JpgGbyAxLBTsvEKKBCjwBACQsZisSYNmHPGbBA"]],[[null,"run at startup"],[null,"full network access"],[null,"prevent device from sleeping"],[null,"view network connections"],[null,"control vibration"]],["quiz"]]],[[null,"receive data from Internet"]]]

But it cannot extract the "Others" segment:

Selección_1473

I've tried something like this:

func (app *App) LoadPermissions() error {
    payload := strings.Replace("f.req=%5B%5B%5B%22xdSrCf%22%2C%22%5B%5Bnull%2C%5B%5C%22{{appID}}%5C%22%2C7%5D%2C%5B%5D%5D%5D%22%2Cnull%2C%221%22%5D%5D%5D", "{{appID}}", app.ID, 1)

    js, err := util.BatchExecute(app.options.Country, app.options.Language, payload)
    if err != nil {
        return err
    }

    app.Permissions = make(map[string][]string)

    for _, path := range []string{"0", "1"} {

        for _, perm := range util.GetJSONArray(js, path) {
            key := util.GetJSONValue(perm.String(), "0")
            for _, permission := range util.GetJSONArray(perm.String(), "2") {
                app.Permissions[key] = append(app.Permissions[key], util.GetJSONValue(permission.String(), "1"))
            }
        }
    }

    return nil
}

And this code can extract all the permissions from "Others" except the first one, "receive data from Internet".

n0madic commented 1 year ago

nice, please make PR