search

illogical-robot / apkmirror-public

APKMirror.com bugs
http://www.apkmirror.com
100 stars 32 forks source link

rss feeds return http 403, included in cloudflare ddos? #202

Closed phpmoli closed 2 years ago

phpmoli commented 2 years ago

Expected behavior

All rss feed urls should always return a valid xml, never-ever an html document, because that break rss feed reader clients, they cannot parse html, they expect a valid xml. Instead in case of http 500 or 429, return an empty body, if the rfc allows that.

Actual behavior

Feed urls return a html document and http 403 status code. Opening a feed url in a browser runs the javascript and redirects to the actual feed contents. It looks like Cloudflare DDoS protection? Maybe the feed urls should be exempted from the protection? When a client is malicious, return http 429.

Steps to reproduce the problem

In a command line prompt run the command: curl https://www.apkmirror.com/apk/google-inc/feed/ The result will be:

<html>
<head><title>403 Forbidden</title></head>
<body>
<center><h1>403 Forbidden</h1></center>
<hr><center>nginx/1.21.3</center>
<script defer src="https://static.cloudflareinsights.com/beacon.min.js/v652eace1692a40cfa3763df669d7439c1639079717194" integrity="sha512-Gi7xpJR8tSkrpF7aordPZQlW2DLtzUlZcumS8dMQjwDHEnw9I7ZLyiOj/6tZStRBGtGgN6ceN6cMH8z7etPGlw==" data-cf-beacon='{"rayId":"6d52cf01e8316936","token":"d9c7bb20e47748bba6112071835597f3","version":"2021.12.0","si":100}' crossorigin="anonymous"></script>
</body>
</html>
archon810 commented 2 years ago

We block some user agents, and curl is one of them. Please use a different user agent.

phpmoli commented 2 years ago

The problem above is not with the blocking, the problem is that the blocking http return does not give a valid .rss content back, so all the poorly written rss clients go dumbfounded when they get a .html or anything else than a valid .rss back in the http body. The solution would be to:

  1. fix all and every poorly written rss clients in the world
  2. or change apkmirror's blocking so it returns an .rss file containing 1 feed entry with an error message (might be annoying)
  3. or change apkmirror's blocking so it returns a 0 byte length empyt http body with a http 403 or 429 status error code

edit: curl was just used for the sake of the example, as it is the simplest way to make a http request in linux.