fossar / selfoss

multipurpose rss reader, live stream, mashup, aggregation web application
https://selfoss.aditu.de
GNU General Public License v3.0
2.35k stars 343 forks source link

YouTube feed discovery broken #1468

Open jtojnar opened 7 months ago

jtojnar commented 7 months ago

Using YouTube spout with https://www.youtube.com/c/BreakingTaps URL now fails with the following:

2023-11-12 11:02:24error loading feed content: https://www.youtube.com/c/BreakingTaps is invalid XML, likely due to invalid characters. XML error: > required at line 1, column 117

It is caused by YouTube deciding to start redirecting to a consent page:

Logs ``` Nov 12 01:25:38 azazel php[1147758]: [2023-11-12T00:25:38.948869+00:00] selfoss.DEBUG: start fetching source "Breaking Taps (id: 463) Nov 12 01:25:38 azazel php[1147758]: [2023-11-12T00:25:38.948883+00:00] selfoss.DEBUG: spout successfully loaded: spouts\youtube\youtube Nov 12 01:25:38 azazel php[1147758]: [2023-11-12T00:25:38.948892+00:00] selfoss.DEBUG: fetch content Nov 12 01:25:39 azazel php[1147758]: [2023-11-12T00:25:39.007424+00:00] selfoss.DEBUG: >>>>>>>> Nov 12 01:25:39 azazel php[1147758]: GET /c/BreakingTaps HTTP/1.1 Nov 12 01:25:39 azazel php[1147758]: Host: www.youtube.com Nov 12 01:25:39 azazel php[1147758]: User-Agent: Selfoss/2.20-cdc7f3e (+https://selfoss.aditu.de) Nov 12 01:25:39 azazel php[1147758]: Referer: https://www.youtube.com/c/BreakingTaps Nov 12 01:25:39 azazel php[1147758]: Accept: application/atom+xml, application/rss+xml, application/rdf+xml;q=0.9, application/xml;q=0.8, text/xml;q=0.8, text/html;q=0.7, unknown/unknown;q=0.1, application/unknown;q=0.1, */*;q=0.1 Nov 12 01:25:39 azazel php[1147758]: <<<<<<<< Nov 12 01:25:39 azazel php[1147758]: HTTP/1.1 302 Found Nov 12 01:25:39 azazel php[1147758]: Content-Type: application/binary Nov 12 01:25:39 azazel php[1147758]: X-Content-Type-Options: nosniff Nov 12 01:25:39 azazel php[1147758]: Cache-Control: no-cache, no-store, max-age=0, must-revalidate Nov 12 01:25:39 azazel php[1147758]: Pragma: no-cache Nov 12 01:25:39 azazel php[1147758]: Expires: Mon, 01 Jan 1990 00:00:00 GMT Nov 12 01:25:39 azazel php[1147758]: Date: Sun, 12 Nov 2023 00:25:38 GMT Nov 12 01:25:39 azazel php[1147758]: Location: https://consent.youtube.com/m?continue=https%3A%2F%2Fwww.youtube.com%2Fc%2FBreakingTaps%3Fcbrd%3D1&gl=CZ&m=0&pc=yt&cm=2&hl=cs&src=1 Nov 12 01:25:39 azazel php[1147758]: X-Frame-Options: SAMEORIGIN Nov 12 01:25:39 azazel php[1147758]: Strict-Transport-Security: max-age=31536000 Nov 12 01:25:39 azazel php[1147758]: Report-To: {"group":"youtube_main","max_age":2592000,"endpoints":[{"url":"https://csp.withgoogle.com/csp/report-to/youtube_main"}]} Nov 12 01:25:39 azazel php[1147758]: Permissions-Policy: ch-ua-arch=*, ch-ua-bitness=*, ch-ua-full-version=*, ch-ua-full-version-list=*, ch-ua-model=*, ch-ua-wow64=*, ch-ua-form-factor=*, ch-ua-platform=*, ch-ua-platform-version=* Nov 12 01:25:39 azazel php[1147758]: Cross-Origin-Opener-Policy: same-origin-allow-popups; report-to="youtube_main" Nov 12 01:25:39 azazel php[1147758]: Origin-Trial: AvC9UlR6RDk2crliDsFl66RWLnTbHrDbp+DiY6AYz/PNQ4G4tdUTjrHYr2sghbkhGQAVxb7jaPTHpEVBz0uzQwkAAAB4eyJvcmlnaW4iOiJodHRwczovL3lvdXR1YmUuY29tOjQ0MyIsImZlYXR1cmUiOiJXZWJWaWV3WFJlcXVlc3RlZFdpdGhEZXByZWNhdGlvbiIsImV4cGlyeSI6MTcxOTUzMjc5OSwiaXNTdWJkb21haW4iOnRydWV9 Nov 12 01:25:39 azazel php[1147758]: P3P: CP="This is not a P3P policy! See http://support.google.com/accounts/answer/151657?hl=cs for more info." Nov 12 01:25:39 azazel php[1147758]: Server: ESF Nov 12 01:25:39 azazel php[1147758]: Content-Length: 0 Nov 12 01:25:39 azazel php[1147758]: X-XSS-Protection: 0 Nov 12 01:25:39 azazel php[1147758]: Set-Cookie: SOCS=CAAaBgiAq8CqBg; Domain=.youtube.com; Expires=Wed, 11-Dec-2024 00:25:38 GMT; Path=/; Secure; SameSite=lax, YSC=k-IrTaIrtsc; Domain=.youtube.com; Path=/; Secure; HttpOnly; SameSite=none, __Secure-YEC=CgtuTjV2S2diREVOVSiCt8CqBjIICgJDWhICEgA%3D; Domain=.youtube.com; Expires=Wed, 11-Dec-2024 00:25:37 GMT; Path=/; Secure; HttpOnly; SameSite=lax, VISITOR_PRIVACY_METADATA=CgJDWhICEgA%3D; Domain=.youtube.com; Expires=Wed, 11-Dec-2024 00:25:38 GMT; Path=/; Secure; HttpOnly; SameSite=lax, VISITOR_INFO1_LIVE=; Domain=.youtube.com; Expires=Mon, 15-Feb-2021 00:25:38 GMT; Path=/; Secure; HttpOnly; SameSite=none, CONSENT=PENDING+472; expires=Tue, 11-Nov-2025 00:25:38 GMT; path=/; domain=.youtube.com; Secure Nov 12 01:25:39 azazel php[1147758]: Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000 Nov 12 01:25:39 azazel php[1147758]: -------- Nov 12 01:25:39 azazel php[1147758]: NULL Nov 12 01:25:39 azazel php[1147758]: [2023-11-12T00:25:39.058897+00:00] selfoss.DEBUG: >>>>>>>> Nov 12 01:25:39 azazel php[1147758]: GET /m?continue=https%3A%2F%2Fwww.youtube.com%2Fc%2FBreakingTaps%3Fcbrd%3D1&gl=CZ&m=0&pc=yt&cm=2&hl=cs&src=1 HTTP/1.1 Nov 12 01:25:39 azazel php[1147758]: Host: consent.youtube.com Nov 12 01:25:39 azazel php[1147758]: User-Agent: Selfoss/2.20-cdc7f3e (+https://selfoss.aditu.de) Nov 12 01:25:39 azazel php[1147758]: Accept: application/atom+xml, application/rss+xml, application/rdf+xml;q=0.9, application/xml;q=0.8, text/xml;q=0.8, text/html;q=0.7, unknown/unknown;q=0.1, application/unknown;q=0.1, */*;q=0.1 Nov 12 01:25:39 azazel php[1147758]: <<<<<<<< Nov 12 01:25:39 azazel php[1147758]: HTTP/1.1 303 See Other Nov 12 01:25:39 azazel php[1147758]: Content-Type: application/binary Nov 12 01:25:39 azazel php[1147758]: Vary: Sec-Fetch-Dest, Sec-Fetch-Mode, Sec-Fetch-Site Nov 12 01:25:39 azazel php[1147758]: Cache-Control: no-cache, no-store, max-age=0, must-revalidate Nov 12 01:25:39 azazel php[1147758]: Pragma: no-cache Nov 12 01:25:39 azazel php[1147758]: Expires: Mon, 01 Jan 1990 00:00:00 GMT Nov 12 01:25:39 azazel php[1147758]: Date: Sun, 12 Nov 2023 00:25:39 GMT Nov 12 01:25:39 azazel php[1147758]: Location: https://www.youtube.com/c/BreakingTaps?cbrd=1&ucbcb=1 Nov 12 01:25:39 azazel php[1147758]: Content-Security-Policy: require-trusted-types-for 'script';report-uri /_/ConsentUi/cspreport, script-src 'report-sample' 'nonce-HhJ_V8ae9aH6QoLE0b614Q' 'unsafe-inline';object-src 'none';base-uri 'self';report-uri /_/ConsentUi/cspreport;worker-src 'self' Nov 12 01:25:39 azazel php[1147758]: Cross-Origin-Opener-Policy: unsafe-none Nov 12 01:25:39 azazel php[1147758]: Accept-CH: Sec-CH-UA-Arch, Sec-CH-UA-Bitness, Sec-CH-UA-Full-Version, Sec-CH-UA-Full-Version-List, Sec-CH-UA-Model, Sec-CH-UA-WoW64, Sec-CH-UA-Form-Factor, Sec-CH-UA-Platform, Sec-CH-UA-Platform-Version Nov 12 01:25:39 azazel php[1147758]: Permissions-Policy: ch-ua-arch=*, ch-ua-bitness=*, ch-ua-full-version=*, ch-ua-full-version-list=*, ch-ua-model=*, ch-ua-wow64=*, ch-ua-form-factor=*, ch-ua-platform=*, ch-ua-platform-version=* Nov 12 01:25:39 azazel php[1147758]: Cross-Origin-Resource-Policy: same-site Nov 12 01:25:39 azazel php[1147758]: Server: ESF Nov 12 01:25:39 azazel php[1147758]: Content-Length: 0 Nov 12 01:25:39 azazel php[1147758]: X-XSS-Protection: 0 Nov 12 01:25:39 azazel php[1147758]: X-Frame-Options: SAMEORIGIN Nov 12 01:25:39 azazel php[1147758]: X-Content-Type-Options: nosniff Nov 12 01:25:39 azazel php[1147758]: Set-Cookie: CONSENT=PENDING+507; expires=Tue, 11-Nov-2025 00:25:39 GMT; path=/; domain=.youtube.com; Secure Nov 12 01:25:39 azazel php[1147758]: P3P: CP="This is not a P3P policy! See g.co/p3phelp for more info." Nov 12 01:25:39 azazel php[1147758]: Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000 Nov 12 01:25:39 azazel php[1147758]: -------- Nov 12 01:25:39 azazel php[1147758]: NULL Nov 12 01:25:39 azazel php[1147758]: [2023-11-12T00:25:39.244316+00:00] selfoss.DEBUG: >>>>>>>> Nov 12 01:25:39 azazel php[1147758]: GET /c/BreakingTaps?cbrd=1&ucbcb=1 HTTP/1.1 Nov 12 01:25:39 azazel php[1147758]: Host: www.youtube.com Nov 12 01:25:39 azazel php[1147758]: User-Agent: Selfoss/2.20-cdc7f3e (+https://selfoss.aditu.de) Nov 12 01:25:39 azazel php[1147758]: Accept: application/atom+xml, application/rss+xml, application/rdf+xml;q=0.9, application/xml;q=0.8, text/xml;q=0.8, text/html;q=0.7, unknown/unknown;q=0.1, application/unknown;q=0.1, */*;q=0.1 Nov 12 01:25:39 azazel php[1147758]: <<<<<<<< Nov 12 01:25:39 azazel php[1147758]: HTTP/1.1 200 OK Nov 12 01:25:39 azazel php[1147758]: Content-Type: text/html; charset=utf-8 Nov 12 01:25:39 azazel php[1147758]: X-Content-Type-Options: nosniff Nov 12 01:25:39 azazel php[1147758]: Cache-Control: no-cache, no-store, max-age=0, must-revalidate Nov 12 01:25:39 azazel php[1147758]: Pragma: no-cache Nov 12 01:25:39 azazel php[1147758]: Expires: Mon, 01 Jan 1990 00:00:00 GMT Nov 12 01:25:39 azazel php[1147758]: Date: Sun, 12 Nov 2023 00:25:39 GMT Nov 12 01:25:39 azazel php[1147758]: X-Frame-Options: SAMEORIGIN Nov 12 01:25:39 azazel php[1147758]: Strict-Transport-Security: max-age=31536000 Nov 12 01:25:39 azazel php[1147758]: Report-To: {"group":"youtube_main","max_age":2592000,"endpoints":[{"url":"https://csp.withgoogle.com/csp/report-to/youtube_main"}]} Nov 12 01:25:39 azazel php[1147758]: Permissions-Policy: ch-ua-arch=*, ch-ua-bitness=*, ch-ua-full-version=*, ch-ua-full-version-list=*, ch-ua-model=*, ch-ua-wow64=*, ch-ua-form-factor=*, ch-ua-platform=*, ch-ua-platform-version=* Nov 12 01:25:39 azazel php[1147758]: Origin-Trial: AvC9UlR6RDk2crliDsFl66RWLnTbHrDbp+DiY6AYz/PNQ4G4tdUTjrHYr2sghbkhGQAVxb7jaPTHpEVBz0uzQwkAAAB4eyJvcmlnaW4iOiJodHRwczovL3lvdXR1YmUuY29tOjQ0MyIsImZlYXR1cmUiOiJXZWJWaWV3WFJlcXVlc3RlZFdpdGhEZXByZWNhdGlvbiIsImV4cGlyeSI6MTcxOTUzMjc5OSwiaXNTdWJkb21haW4iOnRydWV9 Nov 12 01:25:39 azazel php[1147758]: Cross-Origin-Opener-Policy: same-origin-allow-popups; report-to="youtube_main" Nov 12 01:25:39 azazel php[1147758]: P3P: CP="This is not a P3P policy! See http://support.google.com/accounts/answer/151657?hl=cs for more info." Nov 12 01:25:39 azazel php[1147758]: Server: ESF Nov 12 01:25:39 azazel php[1147758]: X-XSS-Protection: 0 Nov 12 01:25:39 azazel php[1147758]: Set-Cookie: YSC=f2hckoP0eKc; Domain=.youtube.com; Path=/; Secure; HttpOnly; SameSite=none, __Secure-YEC=CgsyNHAtTTJIUUQ0USiDt8CqBjIICgJDWhICEgA%3D; Domain=.youtube.com; Expires=Wed, 11-Dec-2024 00:25:38 GMT; Path=/; Secure; HttpOnly; SameSite=lax, VISITOR_PRIVACY_METADATA=CgJDWhICEgA%3D; Domain=.youtube.com; Expires=Wed, 11-Dec-2024 00:25:39 GMT; Path=/; Secure; HttpOnly; SameSite=lax, VISITOR_INFO1_LIVE=; Domain=.youtube.com; Expires=Mon, 15-Feb-2021 00:25:39 GMT; Path=/; Secure; HttpOnly; SameSite=none, CONSENT=PENDING+796; expires=Tue, 11-Nov-2025 00:25:39 GMT; path=/; domain=.youtube.com; Secure Nov 12 01:25:39 azazel php[1147758]: Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000 Nov 12 01:25:39 azazel php[1147758]: Accept-Ranges: none Nov 12 01:25:39 azazel php[1147758]: Vary: Accept-Encoding Nov 12 01:25:39 azazel php[1147758]: Transfer-Encoding: chunked Nov 12 01:25:39 azazel php[1147758]: -------- Nov 12 01:25:39 azazel php[1147758]: NULL Nov 12 01:25:39 azazel php[1147758]: [2023-11-12T00:25:39.304459+00:00] selfoss.DEBUG: >>>>>>>> Nov 12 01:25:39 azazel php[1147758]: GET /c/BreakingTaps HTTP/1.1 Nov 12 01:25:39 azazel php[1147758]: Host: www.youtube.com Nov 12 01:25:39 azazel php[1147758]: User-Agent: Selfoss/2.20-cdc7f3e (+https://selfoss.aditu.de) Nov 12 01:25:39 azazel php[1147758]: Referer: https://www.youtube.com/c/BreakingTaps Nov 12 01:25:39 azazel php[1147758]: Accept: application/atom+xml, application/rss+xml, application/rdf+xml;q=0.9, application/xml;q=0.8, text/xml;q=0.8, text/html;q=0.7, unknown/unknown;q=0.1, application/unknown;q=0.1, */*;q=0.1 Nov 12 01:25:39 azazel php[1147758]: <<<<<<<< Nov 12 01:25:39 azazel php[1147758]: HTTP/1.1 302 Found Nov 12 01:25:39 azazel php[1147758]: Content-Type: application/binary Nov 12 01:25:39 azazel php[1147758]: X-Content-Type-Options: nosniff Nov 12 01:25:39 azazel php[1147758]: Cache-Control: no-cache, no-store, max-age=0, must-revalidate Nov 12 01:25:39 azazel php[1147758]: Pragma: no-cache Nov 12 01:25:39 azazel php[1147758]: Expires: Mon, 01 Jan 1990 00:00:00 GMT Nov 12 01:25:39 azazel php[1147758]: Date: Sun, 12 Nov 2023 00:25:39 GMT Nov 12 01:25:39 azazel php[1147758]: Location: https://consent.youtube.com/m?continue=https%3A%2F%2Fwww.youtube.com%2Fc%2FBreakingTaps%3Fcbrd%3D1&gl=CZ&m=0&pc=yt&cm=2&hl=cs&src=1 Nov 12 01:25:39 azazel php[1147758]: Strict-Transport-Security: max-age=31536000 Nov 12 01:25:39 azazel php[1147758]: X-Frame-Options: SAMEORIGIN Nov 12 01:25:39 azazel php[1147758]: Permissions-Policy: ch-ua-arch=*, ch-ua-bitness=*, ch-ua-full-version=*, ch-ua-full-version-list=*, ch-ua-model=*, ch-ua-wow64=*, ch-ua-form-factor=*, ch-ua-platform=*, ch-ua-platform-version=* Nov 12 01:25:39 azazel php[1147758]: Cross-Origin-Opener-Policy: same-origin-allow-popups; report-to="youtube_main" Nov 12 01:25:39 azazel php[1147758]: Origin-Trial: AvC9UlR6RDk2crliDsFl66RWLnTbHrDbp+DiY6AYz/PNQ4G4tdUTjrHYr2sghbkhGQAVxb7jaPTHpEVBz0uzQwkAAAB4eyJvcmlnaW4iOiJodHRwczovL3lvdXR1YmUuY29tOjQ0MyIsImZlYXR1cmUiOiJXZWJWaWV3WFJlcXVlc3RlZFdpdGhEZXByZWNhdGlvbiIsImV4cGlyeSI6MTcxOTUzMjc5OSwiaXNTdWJkb21haW4iOnRydWV9 Nov 12 01:25:39 azazel php[1147758]: Report-To: {"group":"youtube_main","max_age":2592000,"endpoints":[{"url":"https://csp.withgoogle.com/csp/report-to/youtube_main"}]} Nov 12 01:25:39 azazel php[1147758]: P3P: CP="This is not a P3P policy! See http://support.google.com/accounts/answer/151657?hl=cs for more info." Nov 12 01:25:39 azazel php[1147758]: Server: ESF Nov 12 01:25:39 azazel php[1147758]: Content-Length: 0 Nov 12 01:25:39 azazel php[1147758]: X-XSS-Protection: 0 Nov 12 01:25:39 azazel php[1147758]: Set-Cookie: SOCS=CAAaBgiAq8CqBg; Domain=.youtube.com; Expires=Wed, 11-Dec-2024 00:25:39 GMT; Path=/; Secure; SameSite=lax, YSC=DkWZnYEFvcs; Domain=.youtube.com; Path=/; Secure; HttpOnly; SameSite=none, __Secure-YEC=CgtOVnZKTEgtZHFRUSiDt8CqBjIICgJDWhICEgA%3D; Domain=.youtube.com; Expires=Wed, 11-Dec-2024 00:25:38 GMT; Path=/; Secure; HttpOnly; SameSite=lax, VISITOR_PRIVACY_METADATA=CgJDWhICEgA%3D; Domain=.youtube.com; Expires=Wed, 11-Dec-2024 00:25:39 GMT; Path=/; Secure; HttpOnly; SameSite=lax, VISITOR_INFO1_LIVE=; Domain=.youtube.com; Expires=Mon, 15-Feb-2021 00:25:39 GMT; Path=/; Secure; HttpOnly; SameSite=none, CONSENT=PENDING+335; expires=Tue, 11-Nov-2025 00:25:39 GMT; path=/; domain=.youtube.com; Secure Nov 12 01:25:39 azazel php[1147758]: Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000 Nov 12 01:25:39 azazel php[1147758]: -------- Nov 12 01:25:39 azazel php[1147758]: NULL Nov 12 01:25:39 azazel php[1147758]: [2023-11-12T00:25:39.331475+00:00] selfoss.DEBUG: >>>>>>>> Nov 12 01:25:39 azazel php[1147758]: GET /m?continue=https%3A%2F%2Fwww.youtube.com%2Fc%2FBreakingTaps%3Fcbrd%3D1&gl=CZ&m=0&pc=yt&cm=2&hl=cs&src=1 HTTP/1.1 Nov 12 01:25:39 azazel php[1147758]: Host: consent.youtube.com Nov 12 01:25:39 azazel php[1147758]: User-Agent: Selfoss/2.20-cdc7f3e (+https://selfoss.aditu.de) Nov 12 01:25:39 azazel php[1147758]: Accept: application/atom+xml, application/rss+xml, application/rdf+xml;q=0.9, application/xml;q=0.8, text/xml;q=0.8, text/html;q=0.7, unknown/unknown;q=0.1, application/unknown;q=0.1, */*;q=0.1 Nov 12 01:25:39 azazel php[1147758]: <<<<<<<< Nov 12 01:25:39 azazel php[1147758]: HTTP/1.1 303 See Other Nov 12 01:25:39 azazel php[1147758]: Content-Type: application/binary Nov 12 01:25:39 azazel php[1147758]: Vary: Sec-Fetch-Dest, Sec-Fetch-Mode, Sec-Fetch-Site Nov 12 01:25:39 azazel php[1147758]: Cache-Control: no-cache, no-store, max-age=0, must-revalidate Nov 12 01:25:39 azazel php[1147758]: Pragma: no-cache Nov 12 01:25:39 azazel php[1147758]: Expires: Mon, 01 Jan 1990 00:00:00 GMT Nov 12 01:25:39 azazel php[1147758]: Date: Sun, 12 Nov 2023 00:25:39 GMT Nov 12 01:25:39 azazel php[1147758]: Location: https://www.youtube.com/c/BreakingTaps?cbrd=1&ucbcb=1 Nov 12 01:25:39 azazel php[1147758]: Content-Security-Policy: script-src 'report-sample' 'nonce-Pn2ZNms9ulOluE0dBy6lcQ' 'unsafe-inline';object-src 'none';base-uri 'self';report-uri /_/ConsentUi/cspreport;worker-src 'self', require-trusted-types-for 'script';report-uri /_/ConsentUi/cspreport Nov 12 01:25:39 azazel php[1147758]: Cross-Origin-Resource-Policy: same-site Nov 12 01:25:39 azazel php[1147758]: Cross-Origin-Opener-Policy: unsafe-none Nov 12 01:25:39 azazel php[1147758]: Permissions-Policy: ch-ua-arch=*, ch-ua-bitness=*, ch-ua-full-version=*, ch-ua-full-version-list=*, ch-ua-model=*, ch-ua-wow64=*, ch-ua-form-factor=*, ch-ua-platform=*, ch-ua-platform-version=* Nov 12 01:25:39 azazel php[1147758]: Accept-CH: Sec-CH-UA-Arch, Sec-CH-UA-Bitness, Sec-CH-UA-Full-Version, Sec-CH-UA-Full-Version-List, Sec-CH-UA-Model, Sec-CH-UA-WoW64, Sec-CH-UA-Form-Factor, Sec-CH-UA-Platform, Sec-CH-UA-Platform-Version Nov 12 01:25:39 azazel php[1147758]: Server: ESF Nov 12 01:25:39 azazel php[1147758]: Content-Length: 0 Nov 12 01:25:39 azazel php[1147758]: X-XSS-Protection: 0 Nov 12 01:25:39 azazel php[1147758]: X-Frame-Options: SAMEORIGIN Nov 12 01:25:39 azazel php[1147758]: X-Content-Type-Options: nosniff Nov 12 01:25:39 azazel php[1147758]: Set-Cookie: CONSENT=PENDING+657; expires=Tue, 11-Nov-2025 00:25:39 GMT; path=/; domain=.youtube.com; Secure Nov 12 01:25:39 azazel php[1147758]: P3P: CP="This is not a P3P policy! See g.co/p3phelp for more info." Nov 12 01:25:39 azazel php[1147758]: Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000 Nov 12 01:25:39 azazel php[1147758]: -------- Nov 12 01:25:39 azazel php[1147758]: NULL Nov 12 01:25:39 azazel php[1147758]: [2023-11-12T00:25:39.479455+00:00] selfoss.DEBUG: >>>>>>>> Nov 12 01:25:39 azazel php[1147758]: GET /c/BreakingTaps?cbrd=1&ucbcb=1 HTTP/1.1 Nov 12 01:25:39 azazel php[1147758]: Host: www.youtube.com Nov 12 01:25:39 azazel php[1147758]: User-Agent: Selfoss/2.20-cdc7f3e (+https://selfoss.aditu.de) Nov 12 01:25:39 azazel php[1147758]: Accept: application/atom+xml, application/rss+xml, application/rdf+xml;q=0.9, application/xml;q=0.8, text/xml;q=0.8, text/html;q=0.7, unknown/unknown;q=0.1, application/unknown;q=0.1, */*;q=0.1 Nov 12 01:25:39 azazel php[1147758]: <<<<<<<< Nov 12 01:25:39 azazel php[1147758]: HTTP/1.1 200 OK Nov 12 01:25:39 azazel php[1147758]: Content-Type: text/html; charset=utf-8 Nov 12 01:25:39 azazel php[1147758]: X-Content-Type-Options: nosniff Nov 12 01:25:39 azazel php[1147758]: Cache-Control: no-cache, no-store, max-age=0, must-revalidate Nov 12 01:25:39 azazel php[1147758]: Pragma: no-cache Nov 12 01:25:39 azazel php[1147758]: Expires: Mon, 01 Jan 1990 00:00:00 GMT Nov 12 01:25:39 azazel php[1147758]: Date: Sun, 12 Nov 2023 00:25:39 GMT Nov 12 01:25:39 azazel php[1147758]: X-Frame-Options: SAMEORIGIN Nov 12 01:25:39 azazel php[1147758]: Strict-Transport-Security: max-age=31536000 Nov 12 01:25:39 azazel php[1147758]: Permissions-Policy: ch-ua-arch=*, ch-ua-bitness=*, ch-ua-full-version=*, ch-ua-full-version-list=*, ch-ua-model=*, ch-ua-wow64=*, ch-ua-form-factor=*, ch-ua-platform=*, ch-ua-platform-version=* Nov 12 01:25:39 azazel php[1147758]: Cross-Origin-Opener-Policy: same-origin-allow-popups; report-to="youtube_main" Nov 12 01:25:39 azazel php[1147758]: Origin-Trial: AvC9UlR6RDk2crliDsFl66RWLnTbHrDbp+DiY6AYz/PNQ4G4tdUTjrHYr2sghbkhGQAVxb7jaPTHpEVBz0uzQwkAAAB4eyJvcmlnaW4iOiJodHRwczovL3lvdXR1YmUuY29tOjQ0MyIsImZlYXR1cmUiOiJXZWJWaWV3WFJlcXVlc3RlZFdpdGhEZXByZWNhdGlvbiIsImV4cGlyeSI6MTcxOTUzMjc5OSwiaXNTdWJkb21haW4iOnRydWV9 Nov 12 01:25:39 azazel php[1147758]: Report-To: {"group":"youtube_main","max_age":2592000,"endpoints":[{"url":"https://csp.withgoogle.com/csp/report-to/youtube_main"}]} Nov 12 01:25:39 azazel php[1147758]: P3P: CP="This is not a P3P policy! See http://support.google.com/accounts/answer/151657?hl=cs for more info." Nov 12 01:25:39 azazel php[1147758]: Server: ESF Nov 12 01:25:39 azazel php[1147758]: X-XSS-Protection: 0 Nov 12 01:25:39 azazel php[1147758]: Set-Cookie: YSC=B9f4hlGdRP0; Domain=.youtube.com; Path=/; Secure; HttpOnly; SameSite=none, __Secure-YEC=CgtKcUdscEtJc2pkdyiDt8CqBjIICgJDWhICEgA%3D; Domain=.youtube.com; Expires=Wed, 11-Dec-2024 00:25:38 GMT; Path=/; Secure; HttpOnly; SameSite=lax, VISITOR_PRIVACY_METADATA=CgJDWhICEgA%3D; Domain=.youtube.com; Expires=Wed, 11-Dec-2024 00:25:39 GMT; Path=/; Secure; HttpOnly; SameSite=lax, VISITOR_INFO1_LIVE=; Domain=.youtube.com; Expires=Mon, 15-Feb-2021 00:25:39 GMT; Path=/; Secure; HttpOnly; SameSite=none, CONSENT=PENDING+051; expires=Tue, 11-Nov-2025 00:25:39 GMT; path=/; domain=.youtube.com; Secure Nov 12 01:25:39 azazel php[1147758]: Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000 Nov 12 01:25:39 azazel php[1147758]: Accept-Ranges: none Nov 12 01:25:39 azazel php[1147758]: Vary: Accept-Encoding Nov 12 01:25:39 azazel php[1147758]: Transfer-Encoding: chunked Nov 12 01:25:39 azazel php[1147758]: -------- Nov 12 01:25:39 azazel php[1147758]: NULL Nov 12 01:25:39 azazel php[1147758]: [2023-11-12T00:25:39.484557+00:00] selfoss.ERROR: error loading feed content for Breaking Taps {"exception":"[object] (Exception(code: 0): https://www.youtube.com/c/BreakingTaps is invalid XML, likely due to invalid characters. XML error: > required at line 1, column 117 at /nix/store/zwvd6rdhahqkxgn8nqd9d36zyr79c6pg-selfoss-2.20-cdc7f3e/src/helpers/FeedReader.php:61) Nov 12 01:25:39 azazel php[1147758]: [stacktrace] Nov 12 01:25:39 azazel php[1147758]: #0 /nix/store/zwvd6rdhahqkxgn8nqd9d36zyr79c6pg-selfoss-2.20-cdc7f3e/src/spouts/rss/feed.php(63): helpers\\FeedReader->load('https://www.you...') Nov 12 01:25:39 azazel php[1147758]: #1 /nix/store/zwvd6rdhahqkxgn8nqd9d36zyr79c6pg-selfoss-2.20-cdc7f3e/src/spouts/youtube/youtube.php(50): spouts\\rss\\feed->load(Array) Nov 12 01:25:39 azazel php[1147758]: #2 /nix/store/zwvd6rdhahqkxgn8nqd9d36zyr79c6pg-selfoss-2.20-cdc7f3e/src/helpers/ContentLoader.php(117): spouts\\youtube\\youtube->load(Array) Nov 12 01:25:39 azazel php[1147758]: #3 /nix/store/zwvd6rdhahqkxgn8nqd9d36zyr79c6pg-selfoss-2.20-cdc7f3e/src/helpers/ContentLoader.php(56): helpers\\ContentLoader->fetch(Array) Nov 12 01:25:39 azazel php[1147758]: #4 /nix/store/zwvd6rdhahqkxgn8nqd9d36zyr79c6pg-selfoss-2.20-cdc7f3e/cliupdate.php(23): helpers\\ContentLoader->update(Object(helpers\\UpdateVisitor@anonymous)) Nov 12 01:25:39 azazel php[1147758]: #5 {main} Nov 12 01:25:39 azazel php[1147758]: "} ```
jtojnar commented 7 months ago

Looks like it is user-agent dependent

$ curl https://www.youtube.com/c/BreakingTaps -H 'User-Agent: Selfoss/2.20-cdc7f3e (+https://selfoss.aditu.de)' 
$ curl --silent https://www.youtube.com/c/BreakingTaps -H 'User-Agent: Selfoss/2.20-cdc7f3e (Googlebot; +https://selfoss.aditu.de)' | rg --only-matching 'rel="alternate".+?href=".+?"'
rel="alternate" media="handheld" href="https://m.youtube.com/c/BreakingTaps"
rel="alternate" media="only screen and (max-width: 640px)" href="https://m.youtube.com/c/BreakingTaps"
rel="alternate" type="application/rss+xml" title="RSS" href="https://www.youtube.com/feeds/videos.xml?channel_id=UC06HVrkOL33D5lLnCPjr6NQ"
rel="alternate" href="android-app://com.google.android.youtube/http/www.youtube.com/channel/UC06HVrkOL33D5lLnCPjr6NQ"
rel="alternate" href="ios-app://544007664/vnd.youtube/www.youtube.com/channel/UC06HVrkOL33D5lLnCPjr6NQ"

Do-Not-Track and Global Privacy Control headers do not appear to help:

$ curl https://www.youtube.com/c/BreakingTaps -H 'User-Agent: Selfoss/2.20-cdc7f3e (+https://selfoss.aditu.de)' -H 'Sec-GPC: 1'
$ curl https://www.youtube.com/c/BreakingTaps -H 'User-Agent: Selfoss/2.20-cdc7f3e (+https://selfoss.aditu.de)' -H 'Do-Not-Track: 1'

But it is possible to set a SOCS=CAI cookie to prevent the redirect:

$ curl --silent https://www.youtube.com/c/BreakingTaps -H 'User-Agent: Selfoss/2.20-cdc7f3e (+https://selfoss.aditu.de)' --cookie 'SOCS=CAI' | rg --only-matching 'rel="alternate".+?href=".+?"'
rel="alternate" media="handheld" href="https://m.youtube.com/c/BreakingTaps"
rel="alternate" media="only screen and (max-width: 640px)" href="https://m.youtube.com/c/BreakingTaps"
rel="alternate" type="application/rss+xml" title="RSS" href="https://www.youtube.com/feeds/videos.xml?channel_id=UC06HVrkOL33D5lLnCPjr6NQ"
rel="alternate" href="android-app://com.google.android.youtube/http/www.youtube.com/channel/UC06HVrkOL33D5lLnCPjr6NQ"
rel="alternate" href="ios-app://544007664/vnd.youtube/www.youtube.com/channel/UC06HVrkOL33D5lLnCPjr6NQ"

Thanks to https://github.com/gpodder/gpodder/pull/1549 for the info.