Cannot use some feed URLs

dogsleg commented 3 years ago

Adding some feeds results in HTTP 503. Say, adding ("https://dailynous.com/feed/" philosophy) gives [error]: https://dailynous.com/feed/: "HTTP 503" in elfeed-log. So, elfeed just cannot use such feeds. Probably because rss-xml file is generated on (I guess) each request with a new filename.

skeeto commented 3 years ago

That feed works fine for me just now. This looks like a temporary hosting issue at their end. These sorts of temporary errors are fairly common.

dogsleg commented 3 years ago

Hmmm... I still getting the same error message. In fact, downloading this feed via elfeed was never successful for me, and I tried it previously two or three times. May it be related to my platform? I use GNU Emacs 27.1 from the official Debian testing repository.

dogsleg commented 3 years ago

I found out that wget also gives HTTP 503 for me:

$ LANG=C.UTF-8 wget https://dailynous.com/feed/
--2021-04-17 09:49:05--  https://dailynous.com/feed/
Resolving dailynous.com (dailynous.com)... 104.21.64.52, 172.67.176.92
Connecting to dailynous.com (dailynous.com)|104.21.64.52|:443... connected.
HTTP request sent, awaiting response... 503 Service Temporarily Unavailable
2021-04-17 09:49:05 ERROR 503: Service Temporarily Unavailable.

But curl works properly:

$ LANG=C.UTF-8 curl https://dailynous.com/feed/
<!DOCTYPE HTML>
<html lang="en-US">
<head>
  <meta charset="UTF-8" />
  <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
  <meta http-equiv="X-UA-Compatible" content="IE=Edge,chrome=1" />
  <meta name="robots" content="noindex, nofollow" />
  <meta name="viewport" content="width=device-width,initial-scale=1" />
  <title>Just a moment...</title>
  <style type="text/css">
    html, body {width: 100%; height: 100%; margin: 0; padding: 0;}
    body {background-color: #ffffff; color: #000000; font-family:-apple-system, system-ui, BlinkMacSystemFont, "Segoe UI", Roboto, Oxygen, Ubuntu, "Helvetica Neue",Arial, sans-serif; font-size: 16px; line-height: 1.7em;-webkit-font-smoothing: antialiased;}
    h1 { text-align: center; font-weight:700; margin: 16px 0; font-size: 32px; color:#000000; line-height: 1.25;}
    p {font-size: 20px; font-weight: 400; margin: 8px 0;}
    p, .attribution, {text-align: center;}
    #spinner {margin: 0 auto 30px auto; display: block;}
    .attribution {margin-top: 32px;}
    @keyframes fader     { 0% {opacity: 0.2;} 50% {opacity: 1.0;} 100% {opacity: 0.2;} }
    @-webkit-keyframes fader { 0% {opacity: 0.2;} 50% {opacity: 1.0;} 100% {opacity: 0.2;} }
    #cf-bubbles > .bubbles { animation: fader 1.6s infinite;}
    #cf-bubbles > .bubbles:nth-child(2) { animation-delay: .2s;}
    #cf-bubbles > .bubbles:nth-child(3) { animation-delay: .4s;}
    .bubbles { background-color: #f58220; width:20px; height: 20px; margin:2px; border-radius:100%; display:inline-block; }
    a { color: #2c7cb0; text-decoration: none; -moz-transition: color 0.15s ease; -o-transition: color 0.15s ease; -webkit-transition: color 0.15s ease; transition: color 0.15s ease; }
    a:hover{color: #f4a15d}
    .attribution{font-size: 16px; line-height: 1.5;}
    .ray_id{display: block; margin-top: 8px;}
    #cf-wrapper #challenge-form { padding-top:25px; padding-bottom:25px; }
    #cf-hcaptcha-container { text-align:center;}
    #cf-hcaptcha-container iframe { display: inline-block;}
  </style>

      <meta http-equiv="refresh" content="12">
  <script type="text/javascript">
    //<![CDATA[
    (function(){

      window._cf_chl_opt={
        cvId: "2",
        cType: "non-interactive",
        cNounce: "13465",
        cRay: "641307e06c8d3380",
        cHash: "9eba98b9599c7f1",
        cFPWv: "b",
        cTTimeMs: "4000",
        cRq: {
          ru: "aHR0cHM6Ly9kYWlseW5vdXMuY29tL2ZlZWQv",
          ra: "Y3VybC83Ljc0LjA=",
          rm: "R0VU",
          d: "oDX3FfIfwDCYleOHZW73E0xscwHe2zkpChXyjeCw7neKiYvF/LR8apeNb7LH/Uv/3todMBuLHSSd8uoS6DOqmb5wo3Q6lJXU31J6q08YWZZ+z3/EvtEWkgZR/dNC0fOXqMOhfTaW+DxSD4RQ6l16C14zVcjaUSCL3SBYnue6T1y06djqny2LEvfx5uxxB1YSb7jzrggH+f0Z9HZL7Cdtwl6kN/FW9vlJyikmXhOtEYLZu0qDGK7bEjbsqbzv5st00j3Wd8916zx94zmm5gO0Tg75n/2bCnX88yei7MoN5H/GxymlVKkihvDYgMcwmFk0p78Y0d0hqpz8zAzo50OG1uhnnVCCI+aU/qQAAXFcPqWfLT4Sj0G+ErivEkbKC7ov7D270LUkx/gvnn1KbnC8OS7KxLGAH0x1ANj0ulnMS+5Dsc3oZRbSPOUswwj87ZFaokoToT5kymCwK+1NV+bkNRXu7wowQFKOeXEsf8FJ67swlheTqJpU5ynUSA3PnAe7Jq1m6RDdgCXU/kVXSe89XjkshhZbxz3FtpbNo4CHBKq8ChhU+eyPVPFTNy+3Z4DccRq1hzUNpAZRyUF+aCpkAE9rC9G87DfG49iSkD7hXyj6RR7m8nDTYFCNVR7BQ6pnUFghGpYa95Js2uMEnfYwrNnDJSD/fdVYY+1pLUzQYGZaagAkXpHf10m0Pb8qduMB",
          t: "MTYxODYzNTAwOC4wNjUwMDA=",
          m: "a0QxikllAfyKzcZz1zpoTRRN1ph+KHqM+opF1xN2Px4=",
          i1: "YJBnr1MN6bVf6GLN9HEOIA==",
          i2: "6Ms582Q03nZyB2AAnxnX7A==",
          zh: "tI5QNRtetDXO/gXPwJ5cG9H2O10ivevHzXYCTiWufTA=",
          uh: "zu+cQgicc5rF6Q8HHTJhdGCD3QVkjetpkn+QXGMpZBU=",
          hh: "Nw8exUUy7eZBU5qxdBMmMcPxkRdabVvI9RG6qLfEUGk=",
        }
      }
      window._cf_chl_enter = function(){window._cf_chl_opt.p=1};

    })();
    //]]>
  </script>

</head>
<body>
  <table width="100%" height="100%" cellpadding="20">
    <tr>
      <td align="center" valign="middle">
          <div class="cf-browser-verification cf-im-under-attack">
  <noscript>
    <h1 data-translate="turn_on_js" style="color:#bd2426;">Please turn JavaScript on and reload the page.</h1>
  </noscript>
  <div id="cf-content" style="display:none">

    <div id="cf-bubbles">
      <div class="bubbles"></div>
      <div class="bubbles"></div>
      <div class="bubbles"></div>
    </div>
    <h1><span data-translate="checking_browser">Checking your browser before accessing</span> dailynous.com.</h1>

    <div id="no-cookie-warning" class="cookie-warning" data-translate="turn_on_cookies" style="display:none">
      <p data-translate="turn_on_cookies" style="color:#bd2426;">Please enable Cookies and reload the page.</p>
    </div>
    <p data-translate="process_is_automatic">This process is automatic. Your browser will redirect to your requested content shortly.</p>
    <p data-translate="allow_5_secs" id="cf-spinner-allow-5-secs" >Please allow up to 5 seconds&hellip;</p>
    <p data-translate="redirecting" id="cf-spinner-redirecting" style="display:none">Redirecting&hellip;</p>
  </div>
   <a href="https://premedic.info/sixpenny.php?showtopic=669" style="display: none;">table</a>
  <form class="challenge-form" id="challenge-form" action="/feed/?__cf_chl_jschl_tk__=14cd68edc7d0285a7386e39f64b94e523b694ccc-1618635008-0-AQ4-r7G92yENXViXH2KI-o6pQ_IZhX99kM63PYdKyuPBoNFDyE_tlQsHgTw85IopP_ghtpjpyDBpcB-kn_2io4qIQLomDGDDX9NalYCPC_MA-VZK2EJMKINaqhFFXgIZ9DP98TN2Luk_7E1VUVaxWROYqnI5aPVIvynofKbxCe5yqdqj5mqMZ3TbfwaW1vHP65SmrrIV4fuYiwptwNI4fkQ90amAzhkgWz_FToJTuQkqBh16UzWp2wGxDHSCLjKZUufZ9gXi83XWtI0V1vJOeRRevD0DB3qdyaq_9qtFwSOk56d39VRc42WK0cPQIhnv-6QsoDpgfGhGsvDlDeO7R1BUbtD4VRmb4OA636E4noOIxzSskyixMwL_9YY3osQ5Tg" method="POST" enctype="application/x-www-form-urlencoded">
    <input type="hidden" name="r" value="1901cfd7c2132ae554481a15891368d9c03ab133-1618635008-0-AfshtmLswz3SbT/Zt3qEs2Q9q+O9O3m40dpZbmumwJtMbQgAUw2NdQlmhRE1DmdctqHt6MMjZg+dWDqNehWbBVxmXEImn/OnINt/TqnTlyXs3+6e3qbQbxPhBCGB0sQ+zPO+RMclkPX8K7QYox9Z/QEYbu3R46g2Hc7mTxvGvpuCofB7C5ki+gySsb59gR1e7n00W46JKiu9p+y+CGk2gA8MNk27imG2YopIw6CV00MU0LjdbpG9ospkdXtJtMV/mXgt2dsVQ8Nwh8Ap1TyC+6OWlo+5CTUrZs8pDkv8ZCYjPVByNuEHjn14sWzcXvf0zJywHandRLaWRXxroE2YWo0hkbejsDM7r2LI91Jx9tNwDn3BnZwwKkUJqaIjT7C09wb4JRxjkDR7LOW7nxhvehi62FcEQXqSFPPcsros4aISQXJwQf+EX+zUo8x5DezSrEZujPwMIYp6wyCZ3oirVmrDVaG4sQTB6zLNyt6fErWnWyLltibD94y2D8TiT+coAZgLWdCC9Z5jqNA17SBAvSfX6sdoF+AvAFFgcylbSQMIjoie/b+QMfPIcfR8hnW9/fLZmuSJQS2envLY8lZSYG7LyM46BXglTphIA7WonPIH01Sh3mHHsIiRxf1pxER9rlGCtbDKSXPl1ghPLu1nwyKwC7AFXrAE10qZtE5U0U6GGUsbblKVWBk2/4dm485T8hUdjgfKfL0tVlZJoP7vUy0QYQ5asRk0oR1zBf+yKTzHNkQr4C4+VmbZvAIL3+q1riUTHXLN7uehP/f1d5x5McP/7j7p+Rvj1k96O63IRSAFzZ4Nj/VqGOTD0gZVXeI5Sz97+h9Na/E0rf881vGLhXyn8bJlb8eRRGweR3LvQnAkBbWO0i8i4k/aaz6/nLb9pknD+VtgUvScrv6SVyqW3nyiKuQhIx6NGlUr0wEkC6Bget0xXbZymTPn4e6xtVaXH7+6CrBPHidII9DE769RMstxuD3beMBzibCbaKJcK8CuVnItntx59aiHMnmhqAjggIIynQOtCIAAdMf7gtzcDqnKRLjTiSxZnjWIk5iwXLcRwfNtdo2xJH1tJWdm488g/3OYbjeGOJITt7H/fIu1fVf9UJD0IQ/cKCeHWjsJ1qrtU2e58wxUJQgk85GRyToyg2LXo47R7Lormr4oRYnhlYjbqN/50DyeHmokC0HIvW6DzZCWXrVqgm/usxBwxqksAf3lbIKnMhFLEMAwHkwD1rxxa873q0Nhja+ZOuDG7imKf5PC+mUdSRo7u1JyILhqIhDrt807WAMNM+cARRDhi34tQdLPLPSjzoOAvlXmaAkhmnVLBmU7pQTmEDuBqbpIg4pJW2//dRXDYHE9DEoccvw52gaLeJzY0dUQIny22PIxrgq0/vmyv+nSLbfPczuD4TY6FgnpwaNrIh7DLIQK47clxZwU8/8Dev2/9gHCTI8B3BJMHBpkD88xDcQj15QUPGZ1b0zka24a1nsa24oo1rO7iO1bIMYathgMkAHe4bmMRm9HvLFicZ4cUgIUByl7GxER0gIKh7geZVImRTc+zndtslqrbB3TXq2SsYaRsfuqMG2t0gzoxqQ0bClFj3D7hAOfAYTFYsXJA1Pfn28Y5GrEoZn8zZAsOtAWqBJoRh+Lh4v3dMTn2+7jVbPWgIkq/gpkZ9a3a0gl7CmrfR4mLgu/b+iWo8E6/cYQiejzE83BsVTIcl0+ZmJujWTK6BNVZobHnSkLCTW4mnktegXQZOYaknMClhCF8UrqPLg2SjVB4k4M9KUiQ0/e8Uy4wQ9cBQ=="/>
    <input type="hidden" value="9d16deb2193f84ea35bb586948604b13" id="jschl-vc" name="jschl_vc"/>
    <!-- <input type="hidden" value="" id="jschl-vc" name="jschl_vc"/> -->
    <input type="hidden" name="pass" value="1618635012.065-xbJ3mOhs2G"/>
    <input type="hidden" id="jschl-answer" name="jschl_answer"/>
  </form>

    <script type="text/javascript">
      //<![CDATA[
      (function(){
          var a = document.getElementById('cf-content');
          a.style.display = 'block';
          var isIE = /(MSIE|Trident\/|Edge\/)/i.test(window.navigator.userAgent);
          var trkjs = isIE ? new Image() : document.createElement('img');
          trkjs.setAttribute("src", "/cdn-cgi/images/trace/jschal/js/transparent.gif?ray=641307e06c8d3380");
          trkjs.id = "trk_jschal_js";
          trkjs.setAttribute("alt", "");
          document.body.appendChild(trkjs);
          var cpo=document.createElement('script');
          cpo.type='text/javascript';
          cpo.src="/cdn-cgi/challenge-platform/h/b/orchestrate/jsch/v1?ray=641307e06c8d3380";
          document.getElementsByTagName('head')[0].appendChild(cpo);
        }());
      //]]>
    </script>

  <div id="trk_jschal_nojs" style="background-image:url('/cdn-cgi/images/trace/jschal/nojs/transparent.gif?ray=641307e06c8d3380')"> </div>
</div>

          <div class="attribution">
            DDoS protection by <a rel="noopener noreferrer" href="https://www.cloudflare.com/5xx-error-landing/" target="_blank">Cloudflare</a>
            <br />
            <span class="ray_id">Ray ID: <code>641307e06c8d3380</code></span>
          </div>
      </td>

    </tr>
  </table>
</body>
</html>

And it indicates that they use DDoS protection from Cloudflare, so curl simply cannot get through and download rss feed itself. Guess, the same happens to elfeed.

brittAnderson commented 2 years ago

Wondering if there are any new insights on this. The website wiley online most recent cognitive science works fine from firefox v 100. But fails in elfeed and running curl in the command line returns the same DDoS error as mentioned above.

DDoS protection by <a rel="noopener noreferrer" href="https://www.cloudflare.com/5xx-error-landing/" target="_blank">Cloudflare</a>

I tried copying the curl request from firefox web developer tools to see if I could use some option in curl to get it to work, but I couldn't and that isn't surprising since I don't really know what I am doing. Would be curious if anyone has suggestions what to try to figure out how to get things past this referrer problem. If so, we could add another elfeed variable to handle such issues like is used for elfeed-user-agents.

skeeto commented 2 years ago

There is elfeed-user-agent, which you could set to your web browser's user agent string if the feed works in your browser. That might help if you get a Cloudflare challenge (CAPTCHA, JavaScript), which you could answer in the browser and then hopefully continue on with Elfeed. Though that only works if it validates your particular IP address and doesn't depend a cookie. I don't know any of the details of Cloudflare's DDoS protection except that it tracks a "threat level" of an address and takes more aggressive measures for higher threats. Elfeed will never work from a high threat address on a feed with DDoS protection since only a full-fledged modern web browser can pass the challenge.

The real answer is that hosts must not put their feeds behind DDoS protection since that defeats the purpose of a feed. It's supposed to be requested by non-browsers and automated systems — essentially a misconfiguration of the host. If you can't work around it, the best you can do is ask the host to fix it.

skeeto / elfeed

Cannot use some feed URLs #419