search

ImranR98 / Obtainium

Get Android app updates straight from the source.
https://obtainium.imranr.dev
GNU General Public License v3.0
7.41k stars 164 forks source link

APKMirror: Set up custom user agent to bypass Cloudflare protection #1387

Open ryester19 opened 7 months ago

ryester19 commented 7 months ago

Prerequisites

I did see https://github.com/ImranR98/Obtainium/issues/957 and https://github.com/ImranR98/Obtainium/issues/1097, but those are closed

Describe the feature

The APKMirror source is pretty much useless to me in its current state. I always receive the forbidden error for APKMirror links, regardless of network or VPN I'm connected to. Using curl confirms that this is because of Cloudflare protection

I downloaded APKUpdater and saw that their implementation of APKMirror checking is working just fine on the same phone. Digging deeper into their issues section, I found rumboalla/apkupdater#16, where an APKMirror developer suggested setting a custom user agent as part of their agreement. Changing the user agent to "APKUpdater-v3.0.3" in curl now allows me to retrieve the RSS feed for any app on APKMirror

Seeing as though the APKMirror devs were willing to make an exception for APKUpdater, it shouldn't be too hard to request the same treatment for Obtainium

ImranR98 commented 7 months ago

Hey @archon810 would it be possible for Obtainium to use the user agent workaround? APKMirror is a track-only source so users do have to visit your page to download apps. Relevant: #44

DwainZwerg commented 7 months ago

Hello @ryester19 , @akramer-zibra is working on a PR to avoid few problems with websites that are protected by Cloudflare.

Grishnackh commented 6 months ago

The PR #1411 has been merged. It sets the user agent for GitLab only though, not for APKMirror or other sources.

DwainZwerg commented 6 months ago

The PR #1411 has been merged. It sets the user agent for GitLab only though, not for APKMirror or other sources.

@akramer-zibra I think it would definitely make sense to extend this change to all sources. I thought you had?

akramer-zibra commented 6 months ago

Hey @DwainZwerg , yes and no. #1411 added a referer header. This was enough to bypass the cloudflare protection in the case with Gitlab and the AuroraStore app. There was no need to send a User-Agent header aswell.

That is why this issue seems to be a bit different here. As @ryester19 described above cloudflare may require a User-Agent, but I can double check this on my local dev setup.

By the way: Obtainium has already a feature to configure specific HTTP request headers. But this feature is currently only implemented for HTML sources. This could be a solution for this issue here aswell.

UPDATE: I can't reproduce this issue, sorry. I think the point is as Imran said above: The APKMirror source version is "track-only" in the latest Obtainium version (1.0.5), which means that the user has to visit the download page manually and Obtainium does only track version updates, but does not retrieve any .apk files.