[ENHANCEMENT] Performance impact when getting playlist with high videos count

[SteeledSlagle13]

Describe the bug Hey so I'm seeing when I call

scrapeYt.getPlaylist(playlistMatch[1]).then(playlist => {
    const playlistVideos: Video[] = playlist['videos'];

   console.log(playlistVideos);
});

Where playlistMatch[1] = PLqFSI8ggBE67YFjGyLykEqeh0c37uC8fi

My audio playing on discord has a gap when this is called

To Reproduce

• should be able to reproduce on any playlist over 30+ if not 30+ then try 100 or more
• lower numbered playlist dont have this issue as prevalent as when it is a bigger playlist

Expected behavior No gap in audio when calling this method

Screenshots N/A

Additional context

• this is with a discord bot
• seeing in all 3 of my environments (mac - local, widows - local, ubuntu - aws)
• im using typescript

sorry for opening so many issue man :/

i tried separating the call into a diff method that isnt connected to the player and calling the getPlaylist function while the something is playing and got the same issue.... I dont see the issue with getVideo() and im not seeing it with the search as well

[SteeledSlagle13]

if u have discord i can invite u to my test server to show u

[SuspiciousLookingOwl]

Can you share your minimum code for discord bot with the command to play song and get playlist? (Don't forget to remove any sensitive data like discord token)

[SteeledSlagle13]

ye :D ill get that up here in a bit

[SteeledSlagle13]

are u on windows?

[SuspiciousLookingOwl]

Yes, but i also have VPS running on Ubuntu, if i can't reproduce it in my system, i will try to host it on my VPS.

[SteeledSlagle13]

okie dokie ill remove the part with ffmpeg location instead of worrying about that

[SteeledSlagle13]

hey u cool if i email it to u?

[SuspiciousLookingOwl]

Sure, me@vincentjonathan.com

[SteeledSlagle13]

just sent email with code and steps lmk if you have questions!!!

[SuspiciousLookingOwl]

Try update to 1.2.5 (alpha), npm i scrape-yt@1.2.5 and try again.

[SteeledSlagle13]

okie dokie give me like two minutes XD

[SuspiciousLookingOwl]

Oh nvm it didn't fixed it lol, i thought it did! I will take a look again on what causes it.

[SteeledSlagle13]

XD hahahah sounds good im on 1.2.5 now and no dice I'll keep checking back here i appreciate you!

[SuspiciousLookingOwl]

This seems like a performance issue from the scraper (cheerio). I'm still looking for a lighter alternative to try.

[SteeledSlagle13]

oof sounds good im not in a hurry for the fix i just wanted to bring it up thank you so much for looking into it i really appreciate it

[SuspiciousLookingOwl]

Np! One easy temporary solution that i can think of is just make a local HTTP server as an API to get youtube data using my library, so instead of using scrape-yt directly on the discord code, you just send a HTTP request to that local server so the scraper isn't hogging the discord node js thread.

[SteeledSlagle13]

oh thats an idea i can look into that :D

[SteeledSlagle13]

hey just to let you know.... i tried the node server... that works very nicely!!!! thank you very much

[SuspiciousLookingOwl]

I've added worker thread support which allows the lib to run the scraper in parallel. Tried it on my VPS and seems like it fixed the issue. It's not really a performance improvement, but still 😬. Update not yet published on npm.

[SuspiciousLookingOwl]

Published on npm (@1.3.2). You just need to add options on the second parameter and add useWorkerThread: true

import scrapeYt from "scrape-yt";
// Changed from -> import { scrapeYt } from "scrape-yt";

scrapeYt.getPlaylist("PLx65qkgCWNJIgVrndMrhsedBz1VDp0kfm", { useWorkerThread: true }).then(playlist => {
  console.log(playlist);
});

[SteeledSlagle13]

dang thats awesome!!!! thank you so much ill get this implemented this week 😀