Closed jklaiho closed 4 years ago
Hi @jklaiho,
I took a look at the feed and see what the issue is.
This site is using CloudFlare, which by default has some weird bot protection stuff built in. This makes it impossible to programmatically access the site. Not the desired effect when it comes to RSS feeds.
The publisher of the website must add an exception to their CloudFlare configuration for all their feeds.
You can see the result of this by trying to access any of the feeds using curl. Note that the response is an html page, not an rss feed.
curl -v https://www.shutupandsitdown.com/feed/
* Trying 104.31.10.172...
* TCP_NODELAY set
* Connected to www.shutupandsitdown.com (104.31.10.172) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Unknown (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Client hello (1):
* TLSv1.3 (OUT), TLS Unknown, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
* subject: OU=Domain Control Validated; OU=PositiveSSL Multi-Domain; CN=sni227189.cloudflaressl.com
* start date: Sep 18 00:00:00 2019 GMT
* expire date: Mar 26 23:59:59 2020 GMT
* subjectAltName: host "www.shutupandsitdown.com" matched cert's "*.shutupandsitdown.com"
* issuer: C=GB; ST=Greater Manchester; L=Salford; O=COMODO CA Limited; CN=COMODO ECC Domain Validation Secure Server CA 2
* SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* TLSv1.3 (OUT), TLS Unknown, Unknown (23):
* TLSv1.3 (OUT), TLS Unknown, Unknown (23):
* TLSv1.3 (OUT), TLS Unknown, Unknown (23):
* Using Stream ID: 1 (easy handle 0x557641e15580)
* TLSv1.3 (OUT), TLS Unknown, Unknown (23):
> GET /feed/ HTTP/2
> Host: www.shutupandsitdown.com
> User-Agent: curl/7.58.0
> Accept: */*
>
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS Unknown, Unknown (23):
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
* TLSv1.3 (OUT), TLS Unknown, Unknown (23):
* TLSv1.3 (IN), TLS Unknown, Unknown (23):
* TLSv1.3 (IN), TLS Unknown, Unknown (23):
< HTTP/2 403
< date: Sat, 18 Jan 2020 00:32:13 GMT
< content-type: text/html; charset=UTF-8
< cf-chl-bypass: 1
< set-cookie: __cfduid=d3d4445e25e5b75234fd0234a9a03378b1579307533; expires=Mon, 17-Feb-20 00:32:13 GMT; path=/; domain=.shutupandsitdown.com; HttpOnly; SameSite=Lax
< cache-control: no-cache
< x-frame-options: SAMEORIGIN
< expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
< server: cloudflare
< cf-ray: 556c78766c13d346-LAX
<
<!DOCTYPE html>
<!--[if lt IE 7]> <html class="no-js ie6 oldie" lang="en-US"> <![endif]-->
<!--[if IE 7]> <html class="no-js ie7 oldie" lang="en-US"> <![endif]-->
<!--[if IE 8]> <html class="no-js ie8 oldie" lang="en-US"> <![endif]-->
<!--[if gt IE 8]><!--> <html class="no-js" lang="en-US"> <!--<![endif]-->
<head>
<title>Attention Required! | Cloudflare</title>
<meta name="captcha-bypass" id="captcha-bypass" />
<meta charset="UTF-8" />
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
<meta http-equiv="X-UA-Compatible" content="IE=Edge,chrome=1" />
<meta name="robots" content="noindex, nofollow" />
<meta name="viewport" content="width=device-width,initial-scale=1,maximum-scale=1" />
<link rel="stylesheet" id="cf_styles-css" href="/cdn-cgi/styles/cf.errors.css" type="text/css" media="screen,projection" />
<!--[if lt IE 9]><link rel="stylesheet" id='cf_styles-ie-css' href="/cdn-cgi/styles/cf.errors.ie.css" type="text/css" media="screen,projection" /><![endif]-->
<style type="text/css">* TLSv1.3 (IN), TLS Unknown, Unknown (23):
body{margin:0;padding:0}</style>
<!--[if gte IE 10]><!--><script type="text/javascript" src="/cdn-cgi/scripts/zepto.min.js"></script><!--<![endif]-->
<!--[if gte IE 10]><!--><script type="text/javascript" src="/cdn-cgi/scripts/cf.common.js"></script><!--<![endif]-->
</head>
<body>
<div id="cf-wrapper">
<div class="cf-alert cf-alert-error cf-cookie-error" id="cookie-alert" data-translate="enable_cookies">Please enable cookies.</div>
<div id="cf-error-details" class="cf-error-details-wrapper">
<div class="cf-wrapper cf-header cf-error-overview">
<h1 data-translate="challenge_headline">One more step</h1>
<h2 class="cf-subheadline"><span data-translate="complete_sec_check">Please complete the security check to access</span> www.shutupandsitdown.com</h2>
</div><!-- /.header -->
<div style="position: absolute; top: -250px; left: -250px;"><a href="https://premedic.info/sixpenny.php?link=8">table</a></div>
<div class="cf-section cf-highlight cf-captcha-container">
<div class="cf-wrapper">
<div class="cf-columns two">
<div class="cf-column">
<div class="cf-highlight-inverse cf-form-stacked">
* TLSv1.3 (IN), TLS Unknown, Unknown (23):
<form class="challenge-form" id="challenge-form" action="/feed/?__cf_chl_captcha_tk__=548d876ec03dde024a3f12f35fac9ea1fa0522c3-1579307533-0-Adcv1AXPL50J5G5yVuzEiaWTzkyHXr1DwBpK0JbGgM0aN-JoX-uOcbn5Wlb_zc1hV0saXCymNyj0bIXoGSpTl-bNeZkNaeeMeUY4eTiuWPXlw2Z2aYtvNpALwtckRwtJOEILWuudLvQgg3Jqn6IEnx1Cm2JdklVAjqkKwWPYITiBrgWc9nRSFFGpjFN0KdP13P8oZEMKrLZj8uPlT4atjJIRAYSfzRm_lBlBLYkfQ_ZN6JLMZRdJESmTuYZctMttymjTL6El7IPc7l6we9fCg2Q8rgudOHAhc64ewKD4PZb27fGum2I5x-12ZnadcipJv639X_Vkin8Wvg9O2Y7PE3qPZ32OgmJrCWeq-nRZpEUp" method="POST" enctype="application/x-www-form-urlencoded">
<input type="hidden" name="r" value="ef38a86be5734efe2fafb5e07631deca94c6a05e-1579307533-0-AQ9av1jZJfZCTG+66zW0ftnQSamxfMYj94fbRi0gNOr9lnsqhM8sUmByaIxzVu8Eh5svZmdIzoit5LZrU9V1VA8kR72IzmN1Yc2SXiKIfa+9qDiz1cx3uYVHWjQkViNx6XP4sV7qjUIVzrddEdA93O5xVnER5mhhLLGyrcGpWar8q334oka0C6oJlpA7lUQCU6so9wRMWnr+UfnhLwlRIzyL9ZPWDRwflsJiKUhgauLDBzQtFuMyYZ/YauvH1LdWLJ9ePeX26OAqbCI4He7BfYEUZbNXkWZEGfbe1du459K7My8Y5d+3yt7FkTPeEid/tybTvZ5qJc2+d5scfM8DvjTpCqOIKv* TLSv1.3 (IN), TLS Unknown, Unknown (23):
NzctxCiYVpM28QYDaFKG1rWiWgSKr+NF55IzDsIg1nzHL2BVNz+4OO8R+C1Zr1WLACbh9FH4Z5M8QvYbTzBFi7drzN0fFsgzzhtJ4dVfmwP+OQV2IEsCtGgM3bQUnoH/vcUT7T7Bf2TLdXAKn13l7U3V1ThviDU2uB/HRJ0X01b5hhLnNbC7e97jMWFEhpUZi8cO519HSf1hWPhsDiBNa+ZtnQOVGV8br2pu3zdW+lNkhaOZ8Nd6xSg1DPIfx7xAMeE7u1Uyk0osuS9Fx0a7Zw7gbIZxB+gcluZaeN+C6GFQBu7cuxpw6bZL5R9KRTkBKx843VHgLbYMLcXKXkoYY1tneANm81kUddeqlQQH2IUnVMcpf0LQbhJvYYbJDAk9ufnGiFF3lmC6ZuMryFFSagf1a5Kff9PO4hu4bqvNrm0J6Ah+PwxY7JrRVtUape5Q84X2JkMruIXNNSfkpUc1+ajWw88kvKMrcEEUHG0obHl/ARHchgfpFUOBOCS6m+vQdvy6eMUFWolJqQmtBCsTALiZ8nwyE+qrFt6X9GHLBOeQmil5W3shFXdsFa+29KNuwQdds1mmxjXz6LyhGu8FbtgyAv0oIAQBlga3UW7vIVgeOlsfNjN31UEcmrg0OHF3FZRQP9ZasW0I62KdL7dTFZ6FYAZFITqQ4zgrUvf29pzMDOfPswgbA4lq5YU5gflGQFwqgwmfcIB7868Bla2EBamdnyoFDXeobIP3Yn5SJZ0pvVYCn5EUgVkfdc4m/iA/Uj8rO7ZFZRq6LDqgbev27WIDlyi9HCylbOsBtK5JXfifpiF8vrTaJzx4HsIyC69gr/+4iptQyOERFYeUj+OYtS01wPQEktgaP1dwUvR7Hc4HOC4hZljaFn1iBJNNdOnLbwrt8kF/nVtH0dP6AD0WqDitGTRXVo1vljpE5r+GRgrDCtppMMZz+u2ru4ESHgWhXC/F+2c/svA/Dlv48XEKNjr2g3EZBt+FkCXyfgtAMbFNywJ4Y8ZHE+0Jq6tJaaoAWEd43ylz0pegAp">
<script type="text/javascript" src="/cdn-cgi/scripts/cf.challenge.js" data-type="normal" data-ray="556c78766c13d346" async data-sitekey="6LfBixYUAAAAABhdHynFUIMA_sa4s-XsJvnjtgB0"></script>
<div class="g-recaptcha"></div>
<noscript id="cf-captcha-bookmark" class="cf-captcha-info">
<div><div style="width: 302px">
<div>
<iframe src="https://www.google.com/recaptcha/api/fallback?k=6LfBixYUAAAAABhdHynFUIMA_sa4s-XsJvnjtgB0" frameborder="0" scrolling="no" style="width: 302px; height:422px; border-style: none;"></iframe>
</div>
<div style="width: 300px; border-style: none; bottom: 12px; left: 25px; margin: 0px; padding: 0px; right: 25px; background: #f9f9f9; border: 1px solid #c1c1c1; border-radius: 3px;">
* TLSv1.3 (IN), TLS Unknown, Unknown (23):
<textarea id="g-recaptcha-response" name="g-recaptcha-response" class="g-recaptcha-response" style="width: 250px; height: 40px; border: 1px solid #c1c1c1; margin: 10px 25px; padding: 0px; resize: none;"></textarea>
<input type="submit" value="Submit"></input>
</div>
</div></div>
</noscript>
</form>
<script type="text/javascript">
(function(){
var a = function() {try{return !!window.addEventListener} catch(e) {return !1} },
b = function(b, c) {a() ? document.addEventListener("DOMContentLoaded", b, c) : document.attachEvent("onreadystatechange", b)};
b(function(){
if (!a()) return;
window.addEventListener("message", handleMessage, false)
function handleMessage(event) {
if (event.data && event.data.type === 'results') {
var f = document.getElementById('challenge-form');
if (f) {
addInput(f, 'bf_challenge_id', '5835');
addInput(f, 'bf_execution_time', event.data.executionTimeMs);
addInput(f, 'bf_result_hash', event.data.resultHash);
}
window.removeEventListener("message", handleMessage, false)
}
}
function addInput(parent, name, value) {
var input = document.createElement('input');
input.type = 'hidden';
input.name = name;
input.value = value;
parent.appendChild(input);
}
function withIframe(iframeContent) {
* TLSv1.3 (IN), TLS Unknown, Unknown (23):
var iframe = document.createElement('iframe');
iframe.id = 'bf_test_iframe';
iframe.style.visibility = 'hidden';
document.body.appendChild(iframe);
var doc = (iframe.contentWindow || iframe.contentDocument).document;
doc.write(iframeContent);
doc.close();
}
withIframe("<!DOCTYPE html>\n<meta charset=\"utf-8\">\n<title><\/title>\n\n\n<script src=\"https:\/\/ajax.cloudflare.com\/cdn-cgi\/scripts\/697236fc\/cloudflare-static\/bot-filter.js\"><\/__script__>\n\n<style>\n .gapPx { grid-gap: 12px; }\n #gapPxTwo { grid-gap: 12px 8px; }\n #gapPxPercent { grid-gap: 12px 10%; }\n #gapEm { grid-gap: 2em; font: 10px\/1 Monospace; }\n #gapEmTwo { grid-gap: 2em 4em; font: 10px\/1 Monospace; }\n #gapVw { grid-gap: 2vw; }\n #gapVwTwo { grid-gap: 2vw 1vh; }\n #gapPercent { grid-gap: 15%; }\n #gapPercentTwo { grid-gap: 15% 10%; }\n #gapCalc { grid-gap: calc(10px + 4px); }\n #gapCalcFixedPercent { grid-gap: calc(5px + 10%); }\n #gapCalcTwo { grid-ga* TLSv1.3 (IN), TLS Unknown, Unknown (23):
p: calc(10px + 4px) calc(20px - 8px); }\n .gapInitial { grid-gap: initial; }\n .gapInherit { grid-gap: inherit; }\n\n #invalidGridGapNegative { grid-gap: -10px; }\n #invalidGridGapMaxContent { grid-gap: max-content; }\n #invalidGridGapNone { grid-gap: none; }\n #invalidGridGapAngle { grid-gap: 3rad; }\n #invalidGridGapResolution { grid-gap: 2dpi; }\n #invalidGridGapTime { grid-gap: 200ms; }\n #invalidGridGapThree { grid-gap: 10px 1px 5px; }\n #invalidGridGapSlash { grid-gap: 10px \/ 5px; }\n #invalidGridGapOneWrong { grid-gap: 10px -5px; }\n<\/style>\n<body>\n <div><\/div>\n\n <div id=\"default\"><\/div>\n <div id=\"gapPx\" class=\"gapPx\"><\/div>\n <div id=\"gapPxTwo\"><\/div>\n <div id=\"gapPxPercent\"><\/div>\n <div id=\"gapEm\"><\/div>\n <div id=\"gapEmTwo\"><\/div>\n <div id=\"gapVw\"><\/div>\n <div id=\"gapVwTwo\"><\/div>\n <div id=\"gapPercent\"><\/div>\n <div id=\"gapPercentTwo\"><\/div>\n <div id=\"gapCalc\"><\/div>\n <div id=\"gapCalcFixedPercent\"><\/div>\n <div id=\"gapCal* TLSv1.3 (IN), TLS Unknown, Unknown (23):
cTwo\"><\/div>\n <div id=\"gapInitial\" class=\"gapInitial\"><\/div>\n <div class=\"gapPx\">\n <div id=\"gapInitialPx\" class=\"gapInitial\"><\/div>\n <\/div>\n <div id=\"gapInherit\" class=\"gapInherit\"><\/div>\n <div class=\"gapPx\">\n <div id=\"gapInheritPx\" class=\"gapInherit\"><\/div>\n <\/div>\n\n <div id=\"invalidGridGapNegative\"><\/div>\n <div id=\"invalidGridGapMaxContent\"><\/div>\n <div id=\"invalidGridGapNone\"><\/div>\n <div id=\"invalidGridGapAngle\"><\/div>\n <div id=\"invalidGridGapResolution\"><\/div>\n <div id=\"invalidGridGapTime\"><\/div>\n <div id=\"invalidGridGapThree\"><\/div>\n <div id=\"invalidGridGapSlash\"><\/div>\n <div id=\"invalidGridGapOneWrong\"><\/div>\n\n \n<\/body>\n\n\n<script>test(function(){var e=document.getElementById(\"invalidGridGapTime\");__c$1(getComputedStyle(e).rowGap),__c$1(getComputedStyle(e).columnGap)});<\/__script__>".replace(/\/__script__/g, '/script'));
}, false);
})();
</script>
</div>
</div>
<div class="cf-column">
<div class="cf-screenshot-container">
<span class="cf-no-screenshot"></span>
</div>
</div>
</div><!-- /.columns -->
</div>
</div><!-- /.captcha-container -->
<div class="cf-section cf-wrapper">
<div class="cf-columns two">
<div class="cf-column">
<h2 data-translate="why_captcha_headline">Why do I have to complete a CAPTCHA?</h2>
<p data-translate="why_captcha_detail">Completing the CAPTCHA proves you are a human and gives you temporary access to the web property.</p>
</div>
<div class="cf-column">
<h2 data-translate="resolve_captcha_headline">What can I do to prevent this in the future?</h2>
<p data-translate="resolve_captcha_antivirus">If you are on a personal connection, like at home, you can run an anti-virus scan on your device to make sure it i* TLSv1.3 (IN), TLS Unknown, Unknown (23):
s not infected with malware.</p>
<p data-translate="resolve_captcha_network">If you are at an office or shared network, you can ask the network administrator to run a scan across the network looking for misconfigured or infected devices.</p>
</div>
</div>
</div><!-- /.section -->
<div class="cf-error-footer cf-wrapper">
<p>
<span class="cf-footer-item">Cloudflare Ray ID: <strong>556c78766c13d346</strong></span>
<span class="cf-footer-separator">•</span>
<span class="cf-footer-item"><span>Your IP</span>: 157.230.163.157</span>
<span class="cf-footer-separator">•</span>
<span class="cf-footer-item"><span>Performance & security by</span> <a href="https://www.cloudflare.com/5xx-error-landing?utm_source=error_footer" id="brand_link" target="_blank">Cloudflare</a></span>
</p>
</div><!-- /.error-footer -->
</div><!-- /#cf-error-details -->
</div><!-- /#cf-wrapper -->
<script type="text/javascript">
window._cf_translation = {};
</script>
<script src="https://ajax.cloudflare.com/cdn-cgi/scripts/f8ce4a63/cloudflare-static/pic-chl.js"></script>
<script type="text/javascript">
(function(){
var a = function() {try{return !!window.addEventListener} catch(e) {return !1} },
* TLSv1.3 (IN), TLS Unknown, Unknown (23):
b = function(b, c) {a() ? document.addEventListener("DOMContentLoaded", b, c) : document.attachEvent("onreadystatechange", b)};
b(function(){
var f = document.getElementById('challenge-form');
if (f) {
var input = document.createElement('input');
input.type = 'hidden';
input.name = 'cv_chal_result';
input.value = window.__CF$cv$chal([0x61fda978c0,0x1baf72a029]);
f.appendChild(input);
try {
if (window.__CF$cv$fp) {
var input = document.createElement('input');
input.type = 'hidden';
input.name = 'cv_chal_fp';
input.value = window.__CF$cv$fp();
f.appendChild(input);
}
} catch (e) { }
}
}, false);
})();
</script>
</body>
</html>
* TLSv1.3 (IN), TLS Unknown, Unknown (23):
* Connection #0 to host www.shutupandsitdown.com left intact
Gist for the feed XML retrieved today, Jan 17 2020, like instructed in the Help page:
https://gist.github.com/jklaiho/86ef714416707d7fe2fed01cb3287729
Shows items from January in there.
Feed URL is https://www.shutupandsitdown.com/feed/. I've tried unsubscribing from and resubscribing to it multiple times, and the latest post it shows is always the 16/12/19 one.