[FR]: overcome cloudflare js handshake protection

chpasha commented 3 weeks ago

Brief description of the feature request

some feeds, guarded by cloudflare, return a 403 response AND html page with javascript and then after some check the rss. Maybe there is a way to overcome that - liferea somehow does that, but quiterss and rssguard fail to fetch such feeds

curl -v [mydealz.de/rss…hot](https://www.mydealz.de/rss/hot) [8:28:49]
* Trying 104.18.201.116:443...
* TCP_NODELAY set
* Connected to [mydealz.de](http://www.mydealz.de/) (104.18.201.116) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
* subject: CN=[mydealz.de](http://mydealz.de/)
* start date: Sep 25 06:12:53 2024 GMT
* expire date: Dec 24 06:12:52 2024 GMT
* subjectAltName: host "[mydealz.de](http://www.mydealz.de/)" matched cert's "*.[mydealz.de](http://mydealz.de/)"
* issuer: C=US; O=Google Trust Services; CN=WE1
* SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0xaaab048e1e90)
> GET /rss/hot HTTP/2
> Host: [mydealz.de](http://www.mydealz.de/)
> user-agent: curl/7.68.0
> accept: */*
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
< HTTP/2 403

< server: cloudflare

this is the html returned by cloudflare

<!DOCTYPE html><html lang="en-US"><head><title>Just a moment...</title><meta http-equiv="Content-Type" content="text/html; charset=UTF-8"><meta http-equiv="X-UA-Compatible" content="IE=Edge"><meta name="robots" content="noindex,nofollow"><meta name="viewport" content="width=device-width,initial-scale=1"><style>*{box-sizing:border-box;margin:0;padding:0}html{line-height:1.15;-webkit-text-size-adjust:100%;color:#313131;font-family:system-ui,-apple-system,BlinkMacSystemFont,Segoe UI,Roboto,Helvetica Neue,Arial,Noto Sans,sans-serif,Apple Color Emoji,Segoe UI Emoji,Segoe UI Symbol,Noto Color Emoji}body{display:flex;flex-direction:column;height:100vh;min-height:100vh}.main-content{margin:8rem auto;max-width:60rem;padding-left:1.5rem}@media (width <= 720px){.main-content{margin-top:4rem}}.h2{font-size:1.5rem;font-weight:500;line-height:2.25rem}@media (width <= 720px){.h2{font-size:1.25rem;line-height:1.5rem}}#challenge-error-text{background-image:url(data:image/svg+xml;base64,PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHdpZHRoPSIzMiIgaGVpZ2h0PSIzMiIgZmlsbD0ibm9uZSI+PHBhdGggZmlsbD0iI0IyMEYwMyIgZD0iTTE2IDNhMTMgMTMgMCAxIDAgMTMgMTNBMTMuMDE1IDEzLjAxNSAwIDAgMCAxNiAzbTAgMjRhMTEgMTEgMCAxIDEgMTEtMTEgMTEuMDEgMTEuMDEgMCAwIDEtMTEgMTEiLz48cGF0aCBmaWxsPSIjQjIwRjAzIiBkPSJNMTcuMDM4IDE4LjYxNUgxNC44N0wxNC41NjMgOS41aDIuNzgzem0tMS4wODQgMS40MjdxLjY2IDAgMS4wNTcuMzg4LjQwNy4zODkuNDA3Ljk5NCAwIC41OTYtLjQwNy45ODQtLjM5Ny4zOS0xLjA1Ny4zODktLjY1IDAtMS4wNTYtLjM4OS0uMzk4LS4zODktLjM5OC0uOTg0IDAtLjU5Ny4zOTgtLjk4NS40MDYtLjM5NyAxLjA1Ni0uMzk3Ii8+PC9zdmc+);background-repeat:no-repeat;background-size:contain;padding-left:34px}@media (prefers-color-scheme:dark){body{background-color:#222;color:#d9d9d9}}</style><meta http-equiv="refresh" content="390"></head><body class="no-js"><div class="main-wrapper" role="main"><div class="main-content"><noscript><div class="h2"><span id="challenge-error-text">Enable JavaScript and cookies to continue</span></div></noscript></div></div><script>(function(){window._cf_chl_opt={cvId: '3',cZone: "www.mydealz.de",cType: 'managed',cRay: '8da16a55fc50dbe0',cH: 'oZw5rz_QV__1wMQ7Nt8oAvuq9cOfFPAk406m_eCcnVc-1730186555-1.2.1.1-5rj5di_K5TzPwA2VxNsyiounpkz1kfa88JzJ2B_GzMsJCVqSoNOR1zk6tNqWgToc',cUPMDTk: "\/rss\/hot?__cf_chl_tk=ChQueCchzyXcc9xld3cBm8Xu6H6vx.nAUFx8Mj9hBd0-1730186555-1.0.1.1-D2LNW1W63BFkNA2gPpnY8_puVZ3F0.n5g.jkWz4NPjE",cFPWv: 'g',cITimeS: '1730186555',cTTimeMs: '1000',cMTimeMs: '390000',cTplC: 0,cTplV: 5,cTplB: 'cf',cK: "",fa: "\/rss\/hot?__cf_chl_f_tk=ChQueCchzyXcc9xld3cBm8Xu6H6vx.nAUFx8Mj9hBd0-1730186555-1.0.1.1-D2LNW1W63BFkNA2gPpnY8_puVZ3F0.n5g.jkWz4NPjE",md: "lT9240DY9AjqWA4OVhaUEQXIMoOi1_VXWqnGHcpu.xc-1730186555-1.2.1.1-yutBacz0HMf9sn2S8l3ykhzYqoGmrSpNNC8sVvBluHZpmh.XB3fQfeAsXUQrtIUar5X3Sgddh1oysvdubYJOJL.nNk8okoRKXyvZpaqquMdxlz2pREg6ANzWTTH01xXhxVH0LI1NsWSbKxjkVVTxVikO8h2zugJCwP0EWNd9OBpcb9oyItFGDlkxjomac94z8hK_6_p9v6O1Eu3vIOm5YeRol9PpgTzPqAMwMRu5mar35P6.r7CovSlYE49wifpITKVTo.E0wKU4WHgCVqCoPg__RuC0UMiuTbmnLZD174eJr9NTWOzOjQqWh.n.zypk_cbYpaq1GwGpI1qCdVgPa.aJiuiYJkfF1sJ5SvBjP5EFvqxNv7b2Um0Xla5ps9JcgiKaqdyKawoTxjlfKfbwb1JqrkmRb5XQZC3Ln7nqCK_y9nTjpwIp_a1j3prA3WqOZCf8bctCKMC01F6A6OVn7kP03yIVog2BrXRrUmXVoR_G_Ydh6sdt93lLPlifjwK1Cn.tykzPeAUxEw2ip8g7AdCfxIWzJ0vHP77Zkh8mHVwVqXNoQ5C1LFHTQuYtxpZeKZOJGGAtyuTpqyGeSsoMKoyzQKRVmto.KVZqEI2TusRtZKoOOuuqPjzhs.5i.CpvNazzNVJ5mURelj5mLt6WZhkMNqNUN3_QpuP.RKRfNw7AScQF3mBpRi9DvMgA9I7Hnl5whvrIhBJTYnaUBnjlzeW_cu9qN3YF5k4eBAHJguENtPJmMOh67nFx0BLMTQ46twhQ9S8m7_Ae1id5Ucl3AIyyv9Y1L9.hCdoEv._vqytM9FCbdyVtdDqDE9ya0pyz2tchP66okV5HUObaTNNvggsVfcPTY4ocZ4zlRtTdt8jhiIU3YLtr.FakuUFg3Vl5um.IBmzUHxEdY_JD5dsMwwM7p7K.pvTKqCGcN0EhEKoyy7m906UxMGg33Vld5d__MH5NWLyzRQrvslm4MATk8dg.ggjjJiyPVbXyUdzXZ0w6KZjXDYMdJl_PFZD.7W.RmsJkXV0l7A5op9KRTdR5Ru2gCn9ViM5vk_X47tTJ7_A.hzXOhhSfzJHB8oFAsHns3rxkJChU3u0r7p2suZ.Gqo93EJYkM9.sIlIQEKY0VyZdIaqbNF_8W3Ln2YJE0TteSVSY0xWqfA7FD9J5AWkDvzHl6bjvc.VE1DTgEQwtrbTP1LcIwep3SpFZFLe525eMtRAT8vPbt_NwQAzih840Xd.K.W7fUzk8L.54rhem2v1BnpDL27e_y5Pg4lBsCLOkQk.7Fsa3j9Ahz3S.zpjROdRqK9k9FtayMGgboJfC_9AbVa3mua5Xip2ULjoJMfAid0fLAajjwgNXQhXageheAe8GsW7yzo48iGft2JWGB_X1UMpOaamJxbxRtQkmdryFs3GtZffZkG4.bbCzRiBjB73IqRfC8YGELiEIeT9JlCbe00PNUwrQJojcwC4dZ70NdmOcBw8DXqCb6AYh.tTbqKKIrgsLGQLOSKLsUkBHnUYaMw6cxyRz.sA5kFN8or1AQNZaFLFNZAmGhy8f.eV7YIK8JMW_q3mINJpyGacvnb4oUfo6Vt.kvdxATTAwXEHF428VfBHrvmbEE5AVRYFNspia0TNLM_vu9kWJWlVS67VRl8AIzaqjTb0xaIXcsurD7_NlDkVoYjr8K3M9ajEkYfennqv5ks8ZQi7jnoovWZUo9PpAfXw9qtadZIUmWdscBAdLT92EISzYjfkasRsXep3oC8vbT5vPjleQpUyUDnepjjXczBJ75OS1xemKN.A4zSiKbZMWbBfQpm7NPE2XmxM5p000vyr2lpwYI3Waqc2.YHt1aNCXpoufSWNwFTkEhb_RrXTeD3DlefNhrPupBGtq3SPp_rHAKRWhCt23J2q_xIe_06UIuR7o_6GjL73d_97JszhNjGqAiGxhDimDR9EH6fLYAUu1DvIfY3H3WTLAY6W2eDWNVs5JQ1Jql9SjLTiTJjtdxCQiyMMzFUMwctdXig2OCk0kRzNX3xrvyP4wiycPvoHMNgfg5jKY2cs26nW6PwBRgs27gr4zr7fNtVJrElGctOTdYzUNx55gWYE5wl0DuPjBZz.johZ91G6WLvfXkL0zsW5DtptnIVXBuQQzH0WNgHrpT2plzcwsddsgSzXnDN_B39.mluCiA0EXArvUBjJv1Kr2OSvYoo5mjg",mdrd: "BpV8SxFpqmWzX4tvh9QlYAmT2PAsHvNtD3SZ9W0Yw9c-1730186555-1.2.1.1-zWm3RvyTIUNRa7mmTFPKY_NybztNgaDbRERlWPsZ7B5Eri8yNYE7IBorwiBfd_whxGpP4r7l84JtVFSgwGZgM4lizVQ1K.Ch9ZGh8T4az5Q6c_rBYuHlbxO0qoZzROcQE3Vot6pzpXhG2EKbFcSH4SwYcWwuYJaWvIyWQPLy_1rZEoXcV4TTsyTydxMiTjKENdz23pP.HMCBqC7SF8p0E5prpQ7n8qXCAApE1HALm_sB621yhh7uz8MjLSARiVDfB_wsOdWPM3HrYmTQAFEckr.rbt2IFCfFQGbAK5VePkFCoGN37qMft53rjIwWs9Dg._lT0Xrbzwc9mYzp5McOGoxmsIgnFCocssXPhvWiJu957fdYyVRccxd5mwa0bANQSQm2V3F0M8oBttBAGgX7_S_BkjwP4MoJqCrXIMHwAELYEoUH2mhAxLlS55steGRIfB9BCqUYis9lCD3U73UnLoOMVI8ZNVTnRX940Ov21rrKiKMvqObmyMIIuWeaPy.0dL3AES9rBnk5IjQ_P8DbtpCCpY0y73ky40qYPcsTdo2JWx6X9P96hPw0r4HVI0O_vquzj_fIXhr206TAK47KTSrbyaVcABPOuZ4NcUyY1oTj5qAmBCHc6NZwdmr.fAwfujLIiLEjpKU5.bcUmm.Kf2JvhEr_9b4ipZHpQfTKnyqwuUIRdhDXwf0liTSTkMeUqp_olZ_7CF6.ITpY1EAY_Z6iIZX6hHDgzf_WfLZXock4QRAHV0zeAc0PfNizLAfUocnnElb0ZgxUmfUz9kDtN4spT9wlUMcqnH1O_Jruss3Wl0bE7KiEYMQimqnfTWrg4bwt06zsJCa0xhemBDjxuvdj4U2_fo6S6RxVPPIxNOpuSXRgHV3KYKDgloGgyiicH4Bj_FFOZCkaReSjf0aPi1dyv3RU8kxrSDCvWq2UF0Hdele0H.3XTb0cMvRfuiZ43vIZ8BxCwzFN4wk1UlxMgjxscdr2IOEwUYuCxuRvgDk8T5H94QxuALvw9ufZIJsnupf8zI.DZXwwV6gGSzsb468.XmcMDwqFnQFIbHlQxA2tiDDN8n86IOc7yAt1ruo5TF_tQ8NCaFeucQ4l6ycIGNYNZmjN7BXn1D8TBKmFE_mYDvuxB9gYmRvbobnuhUChBAAPaobKQDRxtdz3Uj46Fk_y_mjJe0H9s1QpQ_93fH1pgz3vd5B4tst76wQtedhrDG0pEDm8kyXGyggDHM4VRXgZ9zdHjqIJQvvsLdCmhGkUXIzyf9S.mJ5LH5KrTLqjqtvGXwiN00Z1Qw1j4fwfAVWolAYPDFL9wCOagjGZibw75Q1t5jD9Ox05G_t4w4jdSjqBqvnKth8albcuAyjTtkzZC4kU4dQ9W8xT2E68gHiskfyaSjvH8rLKQgzK3uuD3yOdJ4KjZVQ4et0Sqb.BNDmh5cQfCXFVJzJRFdpg60RtGaZi0a6QF9bnrgN5QNmPXIZQrBXIRuBH3JOM7baQKQiN0FUXRUSMLAOWthdfY7zN46fpLef73CtRZqU4DWvbz3Nh0a1q155WS_YhwuJ_W8iYKIlzYlUkZgkvj1SzDLb.44YLpy811EPYxnkC8zCivBEkSBDxEio9kYFOCuX4V8FhJZDSmICLfqSGzkby6YnPnm3OOWxNidaj9YvdTrwwTRNqsX3T02K216YW_FBb9rn6fnQdYWpy0jzazLZUX9mRvGlCzd3ycfGRbXe2vP6N8xIwLQmXqE682RQkQoVvRJHLiqLTDYWJ7sSpY.pFDnoFVkOF_Zec_teKwM8y_yov_Yxx_7DhZNcDM3N_Qu5c8sZjxWyLX1tjGyrSOSrC_OJyUtP6.UC5DA2EEDETl_y2DGmn3rrGfIJwfGiKUTU74dhUxjMh7V32_FkkBNDJLcvVxE_QzqkZdeK4bWDNyA0iQQwyXRAfZbqBUIMgQVkNlhY5UnMs6mvKGEN3rUHM85h46.UsEEVx.K8_v._QWv0JD8ClW_TIHIyqh0AJ4Zn_3lL_aXGeajg5R80aJsAuS05p2WYNZa9ASZBL9gzbTRsLYUKZahvBDbHcq4WqVnSOJ.4ee2xeLs5z7d6AHSxpkjvBKFRD8AYlorxTE0PJI_P.NbmPcEs.brfjMa8bEJU117D5F.yY.aWIhrmJxXmIamxQ7oMMm26_oUg61TEynX44TCO7E97Of3EkFaBDFp26h.IlrCXsMMg0PEHgaFGRpL6Bji4e8axsKWzh99Wn94i7cxys.q9RHpVOCsx4y8KTsglYcOtpfN0CxvknX4jYues"};var cpo = document.createElement('script');cpo.src = '/cdn-cgi/challenge-platform/h/g/orchestrate/chl_page/v1?ray=8da16a55fc50dbe0';window._cf_chl_opt.cOgUHash = location.hash === '' && location.href.indexOf('#') !== -1 ? '#' : location.hash;window._cf_chl_opt.cOgUQuery = location.search === '' && location.href.slice(0, location.href.length - window._cf_chl_opt.cOgUHash.length).indexOf('?') !== -1 ? '?' : location.search;if (window.history && window.history.replaceState) {var ogU = location.pathname + window._cf_chl_opt.cOgUQuery + window._cf_chl_opt.cOgUHash;history.replaceState(null, null, "\/rss\/hot?__cf_chl_rt_tk=ChQueCchzyXcc9xld3cBm8Xu6H6vx.nAUFx8Mj9hBd0-1730186555-1.0.1.1-D2LNW1W63BFkNA2gPpnY8_puVZ3F0.n5g.jkWz4NPjE" + window._cf_chl_opt.cOgUHash);cpo.onload = function() {history.replaceState(null, null, ogU);}}document.getElementsByTagName('head')[0].appendChild(cpo);}());</script></body></html>

martinrotter commented 3 weeks ago

Tested and for me the feed works without any problems.

martinrotter commented 3 weeks ago

Try to enable HTTP/2 support in RSS Guard and try again, the website seems to have cloudflare censorshipt "disabled" if HTTP/2 is used.

andrewpros commented 3 weeks ago

if changing settings wont work its not really the app issue this is a public rss for anyone to use either u are blocked for some reason or they have some wrong config and u should write to the service owners

chpasha commented 3 weeks ago

Try to enable HTTP/2 support in RSS Guard and try again, the website seems to have cloudflare censorshipt "disabled" if HTTP/2 is used.

OK, I see, perhaps it should be it. Unfortunately, I have rssguard 4.0.4 - it is the latest available in debian repo and it doesn't have this setting yet. I've found alternative rss url for the same feed, which doesn't have that protection (yet), will stick to it for the time being.

either u are blocked for some reason or they have some wrong config

The latter, see discussion here https://github.com/martinrotter/rssguard/issues/1490 Ac314 was able to reproduce it. Unfortunately people don't care for rss anymore so it is almost impossible to get attraction from the site owners and make them fix that, they are not interested in people not visiting their page :-(

martinrotter commented 3 weeks ago

You have prehistoric RSS Guard version, you really really really really should upgrade to latest.

Perhaps use flatpak or appimage versions of RSS Guard which are known to work well.

chpasha commented 3 weeks ago

yes, I've tried the latest from AppImage, it does work with the mentioned feed, thanks

martinrotter / rssguard

[FR]: overcome cloudflare js handshake protection #1527

Brief description of the feature request