search

ybd-project-ver1 / ytdl-core

Temporary transfer repository for ybd-project/ytdl-core
MIT License
11 stars 3 forks source link

Proxy error when running on Hetzner server: No such format found #25

Open ruidpm opened 3 days ago

ruidpm commented 3 days ago

Describe the bug

Tried OAuth, does not work on Hetzner servers (probably IP blocked). Tried to implement a proxy on CloudFlare worker, works fine locally but fails again on Hetzner server.

CloudFlare worker code: 

const BASE_HEADERS = {
  'Access-Control-Allow-Origin': '*',
  'Access-Control-Allow-Methods': '*',
  'Access-Control-Allow-Headers': '*',
  'Cache-Control': 'public, max-age=2592000',
};

const USER_AGENTS = {
  IOS: 'com.google.ios.youtube/19.29.1 (iPhone16,2; U; CPU iOS 17_5_1 like Mac OS X;)',
  ANDROID: 'com.google.android.youtube/19.29.37 (Linux; U; Android 11) gzip',
  TV: 'Mozilla/5.0 (ChromiumStylePlatform) Cobalt/Version',
  DEFAULT: 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36 Edg/124.0.0.0',
};

addEventListener('fetch', event => {
  event.respondWith(handleRequest(event.request));
});

async function handleRequest(req) {
  if (req.method === 'OPTIONS') {
      return new Response(null, { status: 200, headers: BASE_HEADERS });
  }
  const url = new URL(req.url);
  const REQUEST_URL = decodeURIComponent(url.searchParams.get('url') || '');
  if (!REQUEST_URL || !/(youtube\.com|googlevideo\.com)/.test(REQUEST_URL)) {
      return new Response(null, { status: 400, headers: BASE_HEADERS });
  }
  if (url.pathname.includes('/download')) {
      const C = new URL(REQUEST_URL).searchParams.get('c') || 'WEB';
      const USER_AGENT = USER_AGENTS[C] || USER_AGENTS.DEFAULT;
      console.log('[DEBUG]: Selected user agent:', USER_AGENT);
      const RESPONSE = await fetch(REQUEST_URL, {
          headers: {
              Range: req.headers.get('range') || 'bytes=0-',
              'cache-control': 'no-cache',
              'Accept-Encoding': 'identity;q=1, *;q=0',
              'User-Agent': USER_AGENT,
          },
      });
      if (!RESPONSE.body) {
          return new Response(null, { status: 500, headers: BASE_HEADERS });
      }
      return new Response(RESPONSE.body, { status: 200, headers: { Connection: 'keep-alive', ...BASE_HEADERS } });
  }
  try {
      let contentType = 'text/plain';
      const HEADERS = new Headers(req.headers);
      const METHOD = req.method;
      const BODY = await req.text();
      const RESPONSE_DATA = await fetch(REQUEST_URL, {
          method: METHOD,
          headers: {
              ...Object.fromEntries(HEADERS),
              'User-Agent': USER_AGENTS.DEFAULT,
          },
          body: BODY ? BODY : undefined,
      }).then((response) => {
          const CONTENT_TYPE = response.headers.get('content-type') || '';
          if (CONTENT_TYPE) {
              contentType = CONTENT_TYPE;
          }
          return response.text();
      });
      return new Response(RESPONSE_DATA, { status: 200, headers: { 'Content-Type': contentType, ...BASE_HEADERS } });
  } catch (err) {
      return new Response(JSON.stringify({ error: err.message }), { status: 500, headers: { 'Content-Type': 'application/json', ...BASE_HEADERS } });
  }
}

Server code:

 const ytdl = new YtdlCore({
      streamType: "nodejs",
      logDisplay: ["debug", "info", "success", "warning", "error"],
      originalProxy: {
        base: PROXY_URL,
        download: `${PROXY_URL}/download`,
        urlQueryName: "url",
      },
    });

 const videoInfo = await ytdl.getFullInfo(youtubeUrl);
 const audioStream = await ytdl.downloadFromInfo(videoInfo, {
      filter: "audioonly",
    });

Tried with several different options, with and without filters, with and without PO token, always get the same error

Error Details (Log)

[  DEBUG  ]: [ web ]: Success
[  DEBUG  ]: [ webCreator ]: Error
Reason: Please sign in
[  DEBUG  ]: [ tvEmbedded ]: Error
Reason: Please sign in
[  DEBUG  ]: [ ios ]: Success
[  DEBUG  ]: [ android ]: Success
[  DEBUG  ]: [ web ]: Success
[  DEBUG  ]: [ FileCache ]: "decipherFunction" is cached.
[  DEBUG  ]: [ FileCache ]: "nTransformFunction" is cached.
[  DEBUG  ]: [ web ]: The URL for the video did not return a successful response. Got another format.
Reason: Status code: 403
[  DEBUG  ]: [ ios ]: The URL for the video did not return a successful response. Got another format.
Reason: Status code: 403
[  DEBUG  ]: [ android ]: The URL for the video did not return a successful response. Got another format.
Reason: Status code: 403
Error: No such format found: highest
    at Function.chooseFormat (/app/node_modules/@ybd-project/ytdl-core/src/utils/Format.ts:220:19)
    at /app/node_modules/@ybd-project/ytdl-core/src/core/Download/Download.ts:87:42
    at processTicksAndRejections (node:internal/process/task_queues:95:5)

Environment

ybd-project-ver1 commented 3 days ago

Thank you for your report, please try to find the URL of the video file (xxx.googlevideo.com) from the data retrieved from the getFullInfo function and open it locally. Note that we have not seen a 403 error occur with clients such as the IOS client, so we will need to investigate this in detail.

ruidpm commented 3 days ago

Some urls open fine when pasted locally while some error with 403. I'll try to find if there's any correlation.

ybd-project-ver1 commented 3 days ago

Note that you need to check which client the format is being obtained from. Web clients return 403 for most formats; IOS clients, Tv clients, etc. are stable and we have not seen many 403s.

ruidpm commented 3 days ago

Did a few more tests:

  1. Extracted the non web player formats from the full info
  2. Select one and do a range request through the proxy
  3. If response status is 200 pass that format to the download options, otherwise try the next one

Locally if works fine once again, in the server even though the request returns a 200 the same error still occurs.

Logs:

Checking format: 251 audio/webm; codecs="opus"
Response status: 200
Format is accessible: 251
Selected audio format: {
  itag: 251,
  url: 'https://rr5---sn-o097znsk.googlevideo.com/videoplayback?expire=1729195339&ei=6xgRZ5bPGteYsfIPwZ6agQI&ip=172.71.158.221&id=o-ANo86ymSYi2w-aCglxzZo1Wcw_OzjIw3u5EDyadPUJZ3&itag=251&source=youtube&requiressl=yes&xpc=EgVo2aDSNQ%3D%3D&met=1729173739%2C&mh=sm&mm=31%2C26&mn=sn-o097znsk%2Csn-bg0eznzr&ms=au%2Conr&mv=m&mvi=5&pl=24&rms=au%2Cau&initcwndbps=1997500&bui=AXLXGFT_S9lz7tfafFRTzOANj2wKix7x6dFMKcVLIcENmc_g6drdaRVbTwDBTaJoW-3h5N2dA4hD3Cma&vprv=1&svpuc=1&mime=audio%2Fwebm&ns=TRv4T2ttVy3vshqHOe86bnEQ&rqh=1&gir=yes&clen=4299193&dur=271.781&lmt=1701386145493458&mt=1729173238&fvip=3&keepalive=yes&fexp=51300761%2C51312688&c=TVHTML5_SIMPLY_EMBEDDED_PLAYER&sefc=1&txp=6308224&n=XOV0LXy6BSuGbA&sparams=expire%2Cei%2Cip%2Cid%2Citag%2Csource%2Crequiressl%2Cxpc%2Cbui%2Cvprv%2Csvpuc%2Cmime%2Cns%2Crqh%2Cgir%2Cclen%2Cdur%2Clmt&sig=AJfQdSswRQIgDse-EtJ5lebKbaYZXyp0fArANvG425uPZ60W3C7Z0fUCIQDf15gUOcviRl6hubc73PwT1ofdUM5J3jv0dk7El7F82A%3D%3D&lsparams=met%2Cmh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Crms%2Cinitcwndbps&lsig=ACJ0pHgwRQIhAMe0xhJGV4LCNW7c54_2NNrmJ1dFaNZbEQCgFL_8MJbNAiBiXTeEPc0QIoPY8rEh2fW73Ep1tw36gXu2kONKBh7iNQ%3D%3D',
  mimeType: 'audio/webm; codecs="opus"',
  codec: { text: 'opus', video: null, audio: 'opus' },
  quality: { text: 'tiny', label: 'audio' },
  bitrate: 150391,
  audioBitrate: 160,
  contentLength: '4299193',
  container: 'webm',
  hasVideo: false,
  hasAudio: true,
  isLive: false,
  isHLS: false,
  isDashMPD: false,
  sourceClientName: 'tvEmbedded'
}
[  DEBUG  ]: [ tvEmbedded ]: The URL for the video did not return a successful response. Got another format.
Reason: Status code: 403
[  DEBUG  ]: [ android ]: The URL for the video did not return a successful response. Got another format.
Reason: Status code: 403
[  DEBUG  ]: [ ios ]: The URL for the video did not return a successful response. Got another format.
Reason: Status code: 403
Error: No such format found: highest
    at Function.chooseFormat (/app/node_modules/@ybd-project/ytdl-core/src/utils/Format.ts:220:19)
    at /app/node_modules/@ybd-project/ytdl-core/src/core/Download/Download.ts:87:42
    at processTicksAndRejections (node:internal/process/task_queues:95:5)

The url opens fine locally (which I found is not always the case even though the range request gets a 200) but it errors all the same.

For a bit more context Im running this behind Apache.

ybd-project-ver1 commented 3 days ago

Are you saying that a request to googlelevideo.com using your own proxy will return 200? Then it is possible that the proxy is not correctly specified by ytdl-core when requesting the download.

ruidpm commented 2 days ago

Maybe, the thing that stumps me is that it all works when I run it locally. Thanks for looking into it

ybd-project-ver1 commented 2 days ago

The next version, v6.0.5, will display the URL used for the download in the debug log. Anything further on this issue?

ruidpm commented 2 days ago

I still don't understand why it's not working as the whole point of the proxy is to bypass these restrictions, but I'll try with different packages/setups. Thanks

s123121 commented 1 day ago

running into this issue on gcp as well, local run fine. NOTE: I do not use proxy

[  DEBUG  ]: [ ios ]: The URL for the video did not return a successful response. Got another format.                                                                   
Reason: Status code: 403
[  DEBUG  ]: [ android ]: The URL for the video did not return a successful response. Got another format.                                                               
 Reason: Status code: 403
 /usr/src/app/backend/node_modules/.pnpm/@ybd-project+ytdl-core@6.0.4/node_modules/@ybd-project/ytdl-core/package/utils/Format.js:180                            │
             throw new Error(`No such format found: ${QUALITY}`);

Edit: This bug is beyond weird, cause I have the same setup for production and staging. Both stay under same IP but production work while staging does not. (Same IP , same poToken, same visitorData)

s123121 commented 1 day ago

Upon further investigation, I do not know if this is the same issue. Some links work fine, some links do not work at all no matter what I try. For example, this will always fail for me: https://www.youtube.com/watch?v=CkK3W0lOKcc (getStream audioonly)

ybd-project-ver1 commented 1 day ago

I understand. The cause remains unknown at this time, but we have determined that the IP is not banned, so we will rule this out as the cause.

ybd-project-ver1 commented 1 day ago

Can you view the following URL? Note: After a few hours (6 hours) from obtaining the download URL, it will no longer be viewable.

https://billowing-night-77e7.ybd-project.workers.dev/download/?url=https%3A%2F%2Frr3---sn-oguelnzz.googlevideo.com%2Fvideoplayback%3Fexpire%3D1729358835%26ei%3Dk5cTZ_j1IrKD7OsPocWF8Qc%26ip%3D162.158.119.204%26id%3Do-AOtLpu2zfvS9g2KdGZVR3IPpVRp5-vhRRnkBvXZFb5il%26itag%3D18%26source%3Dyoutube%26requiressl%3Dyes%26xpc%3DEgVo2aDSNQ%253D%253D%26met%3D1729337235%252C%26mh%3DOG%26mm%3D31%252C29%26mn%3Dsn-oguelnzz%252Csn-oguesndr%26ms%3Dau%252Crdu%26mv%3Dm%26mvi%3D3%26pl%3D24%26rms%3Dau%252Cau%26gcr%3Djp%26initcwndbps%3D1421250%26bui%3DAXLXGFTnX-wGO5R2vO7wQMvsUsjII0DR8AqIW-W92KS4RZf3b-Dkqr42D6yhpx08PmINrTkilw0V97Ei%26spc%3D54MbxdjarFMJJR4MeHNR74ZVHe6Pj5A3_TLlE46I4S4oQiKhGlCfnS6fH8OXS3M%26vprv%3D1%26svpuc%3D1%26mime%3Dvideo%252Fmp4%26ns%3DiIS1KFjOdiWfzDkyGu8vFc4Q%26rqh%3D1%26gir%3Dyes%26clen%3D83999008%26ratebypass%3Dyes%26dur%3D1723.222%26lmt%3D1729144493371578%26mt%3D1729336895%26fvip%3D1%26fexp%3D51312688%252C51326336%26c%3DWEB%26sefc%3D1%26txp%3D4438434%26n%3D4WJOMzqPr88YtQ%26sparams%3Dexpire%252Cei%252Cip%252Cid%252Citag%252Csource%252Crequiressl%252Cxpc%252Cgcr%252Cbui%252Cspc%252Cvprv%252Csvpuc%252Cmime%252Cns%252Crqh%252Cgir%252Cclen%252Cratebypass%252Cdur%252Clmt%26sig%3DAJfQdSswRgIhAIvtv3hTjTngxkOX_Q6bI5VesuxA7xeMFvXQ3YITKaalAiEAoetVPTdIcn2mZVFcx4TOx2bTaBeFF-wJiDu9Xx2bYLM%253D%26lsparams%3Dmet%252Cmh%252Cmm%252Cmn%252Cms%252Cmv%252Cmvi%252Cpl%252Crms%252Cinitcwndbps%26lsig%3DACJ0pHgwRQIgf7kGkdPIeF50AwqPqFTl5nxU6WmTSJl2Etw6aGRPkfECIQCGyc1tLgAtnwG10UBGFZOks1cZPpwTT8r3mtgiWEdIIg%253D%253D

s123121 commented 1 day ago

Nope, blank page. From what I remember, the IP you get the URL have to be the same with the IP download the video

ybd-project-ver1 commented 1 day ago

Nope, blank page. From what I remember, the IP you get the URL have to be the same with the IP download the video

I was able to play it in my environment, so the URL may have expired. (Mine is a blank page now too.)

ybd-project-ver1 commented 1 day ago

running into this issue on gcp as well, local run fine. NOTE: I do not use proxy

[  DEBUG  ]: [ ios ]: The URL for the video did not return a successful response. Got another format.                                                                   
Reason: Status code: 403
[  DEBUG  ]: [ android ]: The URL for the video did not return a successful response. Got another format.                                                               
 Reason: Status code: 403
 /usr/src/app/backend/node_modules/.pnpm/@ybd-project+ytdl-core@6.0.4/node_modules/@ybd-project/ytdl-core/package/utils/Format.js:180                            │
             throw new Error(`No such format found: ${QUALITY}`);

Edit: This bug is beyond weird, cause I have the same setup for production and staging. Both stay under same IP but production work while staging does not. (Same IP , same poToken, same visitorData)

Just a thought, what client are you using? Once you try to specify all clients as options. clients: ['web', 'webCreator', 'webEmbedded', 'android', 'ios', 'tv', 'tvEmbedded', 'mweb']

s123121 commented 1 day ago

Just a thought, what client are you using? Once you try to specify all clients as options. clients: ['web', 'webCreator', 'webEmbedded', 'android', 'ios', 'tv', 'tvEmbedded', 'mweb']

Test with the above clients:

[  DEBUG  ]: The specified OAuth2 token is valid.
[  DEBUG  ]: [ FileCache ]: "html5Player" is cached.
[  DEBUG  ]: [ web ]: Success
[  DEBUG  ]: [ webCreator ]: Error
Reason: Please sign in
[  DEBUG  ]: [ tvEmbedded ]: Success
[  DEBUG  ]: [ ios ]: Success
[  DEBUG  ]: [ android ]: Success
[  DEBUG  ]: [ webEmbedded ]: Success
[  DEBUG  ]: [ tv ]: Success
[  DEBUG  ]: [ mweb ]: Success
[  DEBUG  ]: [ web ]: Success
[  DEBUG  ]: [ FileCache ]: "decipherFunction" is cached.
[  DEBUG  ]: [ FileCache ]: "nTransformFunction" is cached.
[  DEBUG  ]: [ web ]: The URL for the video did not return a successful response. Got another format.
Reason: Status code: 403
[  DEBUG  ]: [ ios ]: The URL for the video did not return a successful response. Got another format.
Reason: Status code: 403
[  DEBUG  ]: [ android ]: The URL for the video did not return a successful response. Got another format.
Reason: Status code: 403
/usr/src/app/backend/node_modules/.pnpm/@ybd-project+ytdl-core@6.0.4/node_modules/@ybd-project/ytdl-core/package/utils/Format.js:180
            throw new Error(`No such format found: ${QUALITY}`);
                  ^

Error: No such format found: highest
    at FormatUtils.chooseFormat (/usr/src/app/backend/node_modules/.pnpm/@ybd-project+ytdl-core@6.0.4/node_modules/@ybd-project/ytdl-core/package/utils/Format.js:180:19)
    at /usr/src/app/backend/node_modules/.pnpm/@ybd-project+ytdl-core@6.0.4/node_modules/@ybd-project/ytdl-core/package/core/Download/Download.js:73:51
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)

Test I did:

Same thing, some links worked, some links don't. Local run fine, only gcp servers do not. One more thing to note is running generate poToken from youtube-po-token-generator hang forever on gcp server

ybd-project-ver1 commented 23 hours ago

Can you watch this video? https://billowing-night-77e7.ybd-project.workers.dev/download/?url=https%3A%2F%2Frr3---sn-oguelnzz.googlevideo.com%2Fvideoplayback%3Fexpire%3D1729424180%26ei%3D1JYUZ6aSK7Ha7OsP3pqlwQE%26ip%3D172.71.24.20%26id%3Do-APcr26akFCiKEnCW0FpqVXmIHh71Yqu2J2uvh7f7YeER%26itag%3D18%26source%3Dyoutube%26requiressl%3Dyes%26xpc%3DEgVo2aDSNQ%253D%253D%26met%3D1729402580%252C%26mh%3DOG%26mm%3D31%252C29%26mn%3Dsn-oguelnzz%252Csn-oguesndr%26ms%3Dau%252Crdu%26mv%3Dm%26mvi%3D3%26pl%3D24%26rms%3Dau%252Cau%26gcr%3Djp%26initcwndbps%3D1841250%26bui%3DAXLXGFT6n_k9Y7-mVey0xZhSbbDSify8VNSxcvzmdNSz7JQ7cq-euP1jt-0LfXtLrzWFBxlOzFp1jJTt%26spc%3D54MbxZF5byi-QorwNV9mWC6D-veldtjP4GqxErvpS-FqNLc-2F12VIFzEUNUTME%26vprv%3D1%26svpuc%3D1%26mime%3Dvideo%252Fmp4%26ns%3DKGPjxgFcEkAXVF1iiI89nEkQ%26rqh%3D1%26gir%3Dyes%26clen%3D83999008%26ratebypass%3Dyes%26dur%3D1723.222%26lmt%3D1729144493371578%26mt%3D1729402154%26fvip%3D1%26fexp%3D51312688%26c%3DWEB%26sefc%3D1%26txp%3D4438434%26n%3DqpR1uVX_r6rIGg%26sparams%3Dexpire%252Cei%252Cip%252Cid%252Citag%252Csource%252Crequiressl%252Cxpc%252Cgcr%252Cbui%252Cspc%252Cvprv%252Csvpuc%252Cmime%252Cns%252Crqh%252Cgir%252Cclen%252Cratebypass%252Cdur%252Clmt%26sig%3DAJfQdSswRQIgPKwV2gYHbNng98M5-Gd3RJ9vmbbx9j80josFgxrsaU0CIQDOLjniidG5idZjWPJkJo5dGBFeS70_c6pb4XhKmh2wxg%253D%253D%26lsparams%3Dmet%252Cmh%252Cmm%252Cmn%252Cms%252Cmv%252Cmvi%252Cpl%252Crms%252Cinitcwndbps%26lsig%3DACJ0pHgwRQIgG80wQdybKvmBUvKUP-mIHv85sIei_5U9NX-62LeZ-LUCIQCA-OmixeADCnimQ9cG7bCWERbFnKVSydKrlLYE4p1PBw%253D%253D

s123121 commented 22 hours ago

nope, blank page (403)

ybd-project-ver1 commented 21 hours ago

nope, blank page (403)

The URL I am submitting uses Cloudflare Workers, so it should be viewable by any IP address that accesses it... This issue may require a very detailed investigation.

ybd-project-ver1 commented 20 hours ago

Update: This problem seems to have been a bug in ytdl-core and has been fixed. (Will be released in v6.0.5)

image

ybd-project-ver1 commented 16 hours ago

We release the beta version (v6.0.5-beta.1), please try using it. We still have no assurance that this problem has been completely cured...

npm i @ybd-project/ytdl-core@6.0.5-beta.1
ruidpm commented 16 hours ago

Tried it with a couple of videos and it worked as expected, nice one

ybd-project-ver1 commented 16 hours ago

Tried it with a couple of videos and it worked as expected, nice one

Really!? I'm very happy!