twlite / youtube-sr

A dead-simple youtube metadata scraper
https://npmjs.com/package/youtube-sr
MIT License
116 stars 15 forks source link

Error: Could not parse video metadata! #63

Closed lihuelworks closed 5 months ago

lihuelworks commented 6 months ago

Been having this error with long playlists (+1k).

Error: Could not parse video metadata! at Util.getVideo (C:\Users\yerba\Documents\code\youtube_sr_test\node_modules.pnpm\youtube-sr@4.3.11\node_modules\youtube-sr\dist\mod.js:891:13) at YouTube. (C:\Users\yerba\Documents\code\youtube_sr_test\node_modules.pnpm\youtube-sr@4.3.11\node_modules\youtube-sr\dist\mod.js:1281:27) at Generator.next () at fulfilled (C:\Users\yerba\Documents\code\youtube_sr_test\node_modules.pnpm\youtube-sr@4.3.11\node_modules\youtube-sr\dist\mod.js:44:24) at process.processTicksAndRejections (node:internal/process/task_queues:95:5)

I tried "retrying" the call if it fails, with 3 seconds of sleep between the calls, but I'm still getting errors. What should I do? Does it have something to do with YT API rate-limiting calls?

twlite commented 6 months ago

what's the url? also it seems to be calling .getVideo, but you said playlist 🤔

lihuelworks commented 6 months ago

Sorry, here is the expanded info: I'm making a .csv of all info on the videos of my Favorites playlist. For that, I first take all the id's from the videos using .getPlaylist then loop ever the array of ids using .getVideo. I don't know if it's the speed or the ammount of calls that I make (it's a +1k playlist) but I randomly get the error that I mentioned.

Here is a gist with the code I'm currently using: https://gist.github.com/lihuelworks/05f891acf6b5355824b80144b75f90b0

Edit: Forgot to say: the IDs that give me error are ALL available, and I can click them when I paste them next to the usual https://www.youtube.com/watch?v=-o url. Some examples:

twlite commented 6 months ago

It could be YouTube rate limiting, can you try the lazy method

YouTube.getPlaylist("some_youtube_playlist")
    .then((playlist) => playlist.next()) // fetches next 100 items
    .then(console.log) // all parsable videos
    .catch(console.error);
twlite commented 6 months ago

With this you can control when to fet h the remaining videos

lihuelworks commented 6 months ago

Sorry for the late reply. I've worked a tiny script to get all the IDs of my videos, but I didn't got the real total ammount, it was off by about 150 videos:

image

image

Here is the code I've used: https://gist.github.com/lihuelworks/04a7d8503b89e51dc3efd771148a07eb

lihuelworks commented 6 months ago

Ok, I managed to get the total amount of videos, but I'm still getting random metadata errors.

Edit: I´ve ended up solving it with a combination of the lazy method, and falling back to yt-dlp for any metadata not found with youtube-sr (Called the yt-dlp command directly). Hope this helps anyone else in this situation.