SuspiciousLookingOwl / scrape-yt

Simple lib to scrape information from youtube such as search results, video information, related videos, playlist information and up next video
https://www.npmjs.com/package/scrape-yt
MIT License
9 stars 3 forks source link

.search() sometimes return empty array without error #2

Closed KareemH closed 4 years ago

KareemH commented 4 years ago

Hello again, just want to let you know that after a few test runs, it seems as if the ads are filtered out now. Thank you so much for this fix!

I mentioned earlier that I get an "UnhandledPromiseRejectionWarning: TypeError: Cannot read property 'id' of undefined at addSong" image

addSong is basically my function when I take the video id and store it onto my MongoDB database. Like I mentioned before, I loop through another npm module about billboard music chart data, and then call addSong

image

addSong does something like this: image

This is really a simple call to your module using the name of the artist and title of the songs as the keyword search query. However, the await makes the code behave synchronously, waiting for the videoId to come back as a resolved Promise. However, as shown previously, I get this: image So, I am able to populate my database with 216 video ids when I made 220 calls in reality. On average, 10 videos ids may be missing so that means 10 Promises weren't resolved and I'm 10 songs short at times. Is this an issue involving the way you may have structured your Promise code? Or are my function calls to excessive and can't handle the asynchronous behavior? (I plan on making these function calls every Sunday, not every day, so max 220 calls a week).

Thanks for the swift modifications and cooperation. I hope you can figure something out :)

SuspiciousLookingOwl commented 4 years ago

Cannot read property 'id' of undefined means results[0] is undefined, which most likely means results is empty.

This is actually a known issue where sometimes it successfully retrieve DOM from youtube search, but doesn't have any search result (even though it supposed to have the search result) and will return empty array. Still trying to figure out how to make .search() always have the search result.

Best thing you can do right now is to check whether the result is empty or not, if it's empty, try to .search() again, something like this will do

const maxRetryCount = 3;

var retryCount = 1;
var results = [];

while(results.length == 0 && retryCount < maxRetryCount){
    results = await scrapeYoutube.search(artist + title);
    retryCount++;
}

if (results.length == 0) return false //No search result found

var videoId = results[0].id;
KareemH commented 4 years ago

Oh okay I see, thank you for the quick fix. Your web scraper is still very functional, I highly appreciate it :) A few songs missing is not a problem, I'm grateful that I'm getting nearly the video ids for all songs

Hope this issue can be resolved

Thanks and stay safe!

SuspiciousLookingOwl commented 4 years ago

24670a423e843aa9fb5ae626de6ff8f5c7aff76d should fixed this. Tested with more than 1000 request without any empty result. Published to npm (v1.0.8)

Feel free to open this issue if the issue still persists.

SuspiciousLookingOwl commented 4 years ago

After more tests with different keyword, turns out this problem still persists, however 31d8e800a1591c8588fd559474d13cdc6029ae43 should fixed this, or reduce the error probability at least. But definitely way better than previous version.

Update published to NPM (v1.0.11).