search

LuanRT / BgUtils

https://www.npmjs.com/package/bgutils-js
MIT License
33 stars 0 forks source link

Is it possible to run this module in Cloudflare? #3

Closed rogerpadilla closed 3 weeks ago

rogerpadilla commented 1 month ago

Question

the idea is to use bgutils to get tokens for the deciphered URL with youtubei…

when trying to combine bgutils with youtubei in cloudflare i get a lot of errors about unsupported modules (seems to be more because jsdom relies on node js apis).

running youtubei alone in cloudflare as in the examples works fine, but i guess something as bgutils is necessary to avoid get blocked.

i also tried happy-dom but also fails there.

any ideas please?

Other details

No response

Checklist

LuanRT commented 1 month ago

Is it actually failing when you try to use it in Cloudflare? The warnings don't matter much unless they're causing issues. The only thing needed from jsdom is the document & window objects, which should work anywhere from what I can tell.

rogerpadilla commented 1 month ago

Yep, the error prevents me from deploying the code, let me attach my current code to explain my current case better.

Basically, I am following this quick tutorial https://developers.cloudflare.com/cloudflare-for-platforms/workers-for-platforms/get-started/configuration/

When I run npx wrangler deploy --dispatch-namespace staging without jsdom nor bgutils it does work, the thing is when I use these dependencies in the code I get the following error:

With node_compat = true

Uncaught TypeError: globalThis.XMLHttpRequest is not a constructor
    at null.<anonymous>
  (file:///Users/rpc/projects/variability/cf/ytdl-1/node_modules/rollup-plugin-node-polyfills/polyfills/http-lib/capability.js:20:11)
  in checkTypeSupport
    at null.<anonymous>
  (file:///Users/rpc/projects/variability/cf/ytdl-1/node_modules/rollup-plugin-node-polyfills/polyfills/http-lib/capability.js:39:45)
  in
  node_modules/rollup-plugin-node-polyfills/polyfills/http-lib/capability.js
    at null.<anonymous> (index.js:15:56) in __init
    at null.<anonymous>
  (file:///Users/rpc/projects/variability/cf/ytdl-1/node_modules/rollup-plugin-node-polyfills/polyfills/http-lib/request.js:1:1)
  in
  node_modules/rollup-plugin-node-polyfills/polyfills/http-lib/request.js
    at null.<anonymous> (index.js:15:56) in __init
    at null.<anonymous> (node-modules-polyfills:http:30:1) in
  node-modules-polyfills:http
    at null.<anonymous> (index.js:15:56) in __init
    at null.<anonymous> (node-modules-polyfills-commonjs:http:2:18) in
  node-modules-polyfills-commonjs:http
    at null.<anonymous> (index.js:18:50) in __require2
    at null.<anonymous>
  (file:///Users/rpc/projects/variability/cf/ytdl-1/node_modules/jsdom/lib/jsdom/living/xhr/XMLHttpRequest-impl.js:3:27)
  in node_modules/jsdom/lib/jsdom/living/xhr/XMLHttpRequest-impl.js
   [code: 10021]

And if comment node_compat = true out and add compatibility_flags = ["nodejs_compat"] the error is different:

⛅️ wrangler 3.77.0
-------------------

✘ [ERROR] Could not resolve "http"

    node_modules/agent-base/dist/helpers.js:27:34:
      27 │ const http = __importStar(require("http"));
         ╵                                   ~~~~~~

  The package "http" wasn't found on the file system but is built into node.
  Add "node_compat = true" to your wrangler.toml file and make sure to prefix the module name with "node:" to enable Node.js compatibility.

✘ [ERROR] Could not resolve "https"

    node_modules/agent-base/dist/helpers.js:28:35:
      28 │ const https = __importStar(require("https"));
         ╵                                    ~~~~~~~

  The package "https" wasn't found on the file system but is built into node.
  Add "node_compat = true" to your wrangler.toml file and make sure to prefix the module name with "node:" to enable Node.js compatibility.

✘ [ERROR] Could not resolve "net"

    node_modules/agent-base/dist/index.js:30:33:
      30 │ const net = __importStar(require("net"));

wrangler.toml

#:schema node_modules/wrangler/config-schema.json
name = "ytdl-1"
main = "src/index.ts"
compatibility_date = "2024-09-09"
# compatibility_flags = ["nodejs_compat"]
node_compat = true

Worker code index.ts:

import { UniversalCache, Innertube } from 'youtubei.js';
import { JSDOM } from 'jsdom';
import { BG, BgConfig } from 'bgutils-js';

export default {
  async fetch(request, env, ctx): Promise<Response> {
    const url = await getDownloadUrl('https://www.youtube.com/watch?v=39rBzRd4M0k');
    return new Response('Hello World!, url' + url);
  },
} satisfies ExportedHandler<Env>;

const id = '39rBzRd4M0k';

async function getDownloadUrl(url: string) {
  const yt = await getYt();
  const info = await yt.getInfo(id);
  // console.error('** yti getDownloadUrl info', JSON.stringify(info, null, 2));
  // console.info('** yti getDownloadUrl info.playability_status', info.playability_status);
  const format = chooseFormat(info);
  const decipheredUrl = format.decipher(yt.session.player);
  // console.info('** yti getDownloadUrl decipheredUrl', decipheredUrl);
  // console.log('** yti getDownloadUrl', format);
  return decipheredUrl;
}

function chooseFormat(info: VideoInfo) {
  let format: ReturnType<typeof info.chooseFormat>;
  try {
    format = info.chooseFormat?.({ type: 'video+audio', quality: 'bestefficiency' });
  } catch (err) {
    console.error(err);
  }
  return format ?? info.streaming_data?.formats?.[0];
}

async function scrapeYtSession() {
  const yt = await Innertube.create({ retrieve_player: false });

  const requestKey = 'O43z0dpjhgX20SCx4KAo';
  const visitorData = yt.session.context.client.visitorData;

  const dom = new JSDOM();

  Object.assign(globalThis, {
    window: dom.window,
    document: dom.window.document,
  });

  const bgConfig: BgConfig = {
    fetch: (url, init) => fetch(url, init),
    globalObj: globalThis,
    identifier: visitorData,
    requestKey,
  };

  const challenge = await BG.Challenge.create(bgConfig);

  if (!challenge) throw new TypeError('Could not get challenge');

  if (challenge.script) {
    const script = challenge.script.find((sc) => sc !== null);
    if (script) new Function(script)();
  } else {
    console.warn('Unable to load Botguard.');
  }

  const poToken = await BG.PoToken.generate({
    program: challenge.challenge,
    globalName: challenge.globalName,
    bgConfig,
  });

  const data = { poToken, visitorData };

  return data;
}

let _yt: Innertube & { myCreatedAt?: Date };

async function getYt() {
  const randomMilliseconds = getRandomMilliseconds();
  const thresholdDate = new Date(Date.now() - randomMilliseconds);

  if (thresholdDate <= _yt?.myCreatedAt) {
    return _yt;
  }

  const { poToken, visitorData } = await scrapeYtSession();

  _yt = await Innertube.create({
    po_token: poToken,
    visitor_data: visitorData,
    generate_session_locally: true,
    cache: new UniversalCache(true),
  });

  _yt.myCreatedAt = new Date();

  return _yt;
}

function getRandomMilliseconds() {
  const minutes = Math.floor(Math.random() * 30) + 30;
  return minutes * 60 * 1000;
}

package.json

{
  "name": "ytdl-1",
  "version": "0.0.0",
  "private": true,
  "scripts": {
    "deploy": "wrangler deploy",
    "dev": "wrangler dev",
    "start": "wrangler dev",
    "test": "vitest",
    "cf-typegen": "wrangler types"
  },
  "devDependencies": {
    "@cloudflare/vitest-pool-workers": "^0.4.5",
    "@cloudflare/workers-types": "^4.20240909.0",
    "typescript": "^5.5.2",
    "vitest": "1.5.0",
    "wrangler": "^3.60.3"
  },
  "dependencies": {
    "bgutils-js": "^2.0.1",
    "jsdom": "^25.0.0",
    "youtubei.js": "^10.4.0"
  }
}

I've tried a lot of things, this is getting me crazy lol