search

BaseMax / GooglePlayWebServiceAPI

Tiny script to crawl information of a specific application in the Google play/store base on PHP.
MIT License
37 stars 9 forks source link

Unable to get latest version number/name #29

Open BaseMax opened 1 year ago

BaseMax commented 1 year ago

Hi There,

It seems this project is unable to get the version name/number of some applications. e.g: Instagram

The reason is these applications have different builds for special devices so they have a lot of builds for different devices and for this kind of application we are unable to get the latest version number I think it's really bad because we need to know what is the latest version number of each application since it's an important metrics to see applications updated or not.

Reported by @ATNoosh

Best, M.

IzzySoft commented 1 year ago

Can we get some applicationIds to test against? I'd then see if and what can be done about it. Don't want to rely on a single app with that :wink:

Further, as the version fields are literals (no arrays), if we can figure out we'd just return the highest one, regardless of the device, right?

BaseMax commented 1 year ago

He said me he is unable to get the Instagram version using our library. so here is the link: https://play.google.com/store/apps/details?id=com.instagram.android&hl=en&gl=US

Further, as the version fields are literals (no arrays), if we can figure out we'd just return the highest one, regardless of the device, right?

Yes, it will be great.

BaseMax commented 1 year ago

It seems some popular apps are the same and we are not able to get their version. But WhatsApp is okay and we are able to get the version.

The next examples are Skype, Netflix, Viber, TikTok which we are not able to get the version.

IzzySoft commented 1 year ago

I've added a dump method (#30) to make investigating sources easier in cases like this. Unfortunately, I could not find separate versions mentioned anywhere. I assume those are only reviled when device information is passed.

I've only checked with Insta, though, as that was the only one where the package name was mentioned here :wink:

BaseMax commented 1 year ago
IzzySoft commented 1 year ago

I could try those all as well, but as indicated above I'm afraid version info for them won't be available either unless logged-in and device details provided. I found no version details at all for Insta, and neither for Skype. More cross-checking would make sense to see if the structure is identical – but without any details at all, there's no structure to compare. So I fear there's not much we can do about this.

Be welcome to use the new method provided with the PR linked and take a look at the details available for yourself.

IzzySoft commented 1 year ago

PS: I think I've identified the corresponding section in ds:5. Quoting from Skype (com.skype.raider):

            [
                [
                    null,
                    []
                ],
                [
                    [
                        [
                            29,
                            "10"
                        ]
                    ],
                    [
                        null,
                        []
                    ]
                ]
            ],

Note the first null: this is the place usually holding the versionName. The next block is the targetSdk (29 aka Android 10), then comes the minSdk (again unspecified). For comparison, same section from another app:

            [
                [
                    [
                        "1.14.5"
                    ]
                ],
                [
                    [
                        [
                            30,
                            "11"
                        ]
                    ],
                    [
                        [
                            [
                                28,
                                "9"
                            ]
                        ]
                    ]
                ]
            ],

Here it's a "one-fits-all", so all values are filled.

TL;DR: Unless you want to find a way to fake device information and get "some" data (but then maybe "no data" on other apps not supported by the specified device), I'd say you can close this issue as wontfix – at least I see no way how we can fix it.

BaseMax commented 1 year ago

It's a great update.

IzzySoft commented 1 year ago

Maybe, but how does it help solving the issue? I wouldn't even know how to provide device details (especially without "logging in" to the lion's lair).

BaseMax commented 1 year ago

I think if it's a problem with the device or auths problem we need to send fake information (account needed?). Maybe send an agent of the latest model/brand phone to see what versions returned. Or maybe I am wrong?

IzzySoft commented 1 year ago

Most of the other crawlers require a Google account to start with. Guess this is one of the reasons. You cannot provide device details without specifying the account I'm afraid – and I wouldn't want to implement that (apart from the fact that I can't, as I don't even have an account to test with and neither want to create one).

I'd really love to know if there's a way to do that without account (ideally including fetching the APK). But I very much doubt it is. Been looking far too long for that already; if such a possibility would exist I should have encountered it. But I might be wrong, of course.

IzzySoft commented 1 year ago

Hm, I stand corrected on this part. Here's what I get for Skype when setting the UA to Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36:

            [
                [
                    [
                        "8.96.0.409"
                    ]
                ],
                [
                    [
                        [
                            31,
                            "12"
                        ]
                    ],
                    [
                        [
                            [
                                26,
                                "8.0"
                            ]
                        ]
                    ]
                ]
            ],

But I wouldn't want to have a UA hard-coded. So what to do? Suggestion:

Shall I go for that?

BaseMax commented 1 year ago

Nice.

I think we can set a default one, but it should be possible to change it as well.

IzzySoft commented 1 year ago

Well, false alert. I just played with it: Skype now returns a version even without the UA set. Let me try the other candidates from above:

So there doesn't seem to be a difference. UAs tried:

$ua = "Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36";
$ua = "Dalvik/2.1.0 (Linux; U; Android 10; HD1903 Build/QKQ1.190716.003)"; // OnePlus 7T
$ua = "Dalvik/2.1.0 (Linux; U; Android 7.1.1; SM-G5528 Build/NMF26X)"; // Samsung Galaxy On5
$ua = "Dalvik/2.1.0 (Linux; U; Android 13; Redmi Note 8 Build/TQ2A.230505.002.A1)";

I can still send a PR for the UA-handling, who knows where it might come in handy. Though it does not seem to have any effect now, it might have for other cases. Maybe leave it undocumented/debug-only.

And no, I'd rather refrain from setting a default one. Might falsify results or cause trouble in other places. Remember, some apps are only available for certain devices etc; if the device is not specified there's a chance you still get results – with the wrong one specified, you might not.

IzzySoft commented 1 year ago

PS: I'd wonder if the UA should change anything. It's not he UA Google looks for, it's an entire device profile. Take e.g. a look at Raccoon, which uses DummyDroid to create such profiles. And that needs – tada! – a Google account (see "System requirements"), as the profile is bound to a GSF ID.

So, do you still want me to make a PR for the UA spoofing, @BaseMax? As pointed out, I don't think it will be much help. But if you want it, I'd just have to commit and push; all set here in a separate branch.

BaseMax commented 1 year ago

Nice. If you did it, send it. Thanks.

IzzySoft commented 1 year ago

OK, done & merged. As there were other explicit stream_context_create() definitions, I've replaced them with calls to the new createStreamContext() method while on it, so at least we got some immediate use from it (and less redundant code).

Did not solve this issue here, though – and I don't think we can easily solve it anyway. But who knows what surprises might lurk out there…

BaseMax commented 1 year ago

Heya, Do you think this feature was completely added? We can close this issue or still there is something that need to think and work about.

IzzySoft commented 1 year ago

Did not solve this issue here, though – and I don't think we can easily solve it anyway.

So TL;DR: I did what I could, and I don't see what more could be done. So this issue could be closed – unless you want it to stay open until someone shows up with a "flash of inspiration".

PS: I don't think it would be a good idea to include "Google LogIn & device identification". I certainly will not add that (don't even have a Google account to test that with, and don't want to create one either).