search

banhao / scrape-youtube-channel-videos-url

This Python script is used to scrape all the video links from a youtube channel.
MIT License
52 stars 19 forks source link

threaded script #5

Closed sleepingcat4 closed 5 months ago

sleepingcat4 commented 5 months ago

I had enocuntered some limitations and issues with this repo. These are

  1. lower time-out (very inefficient to load the YouTube page while scrolling and handling potential network errors)
  2. single threaded nature (limits and puts load onto one thread and makes the process slower and in an event of error, it losts all the work it did in the past)
  3. non-headless setup (it crashes in some instances if the links are too many for a particular channel)

I fixed all these aforementioned limitations. Like increased the timeout to better error and exception handling. made it to use double thread so one can scroll down while the other thread can extract and write the links. Made the setup headless by default so potential UI issues can be avoided and puts less contraints on the system. Overall much cleaner.

I added a completely new script so the user can have choice between my script and the orignial script and use that's most convinent for them.