search

alexmercerind / youtube-search-python

🔎 Search for YouTube videos, channels & playlists. Get 🎞 video & 📑 playlist info using link. Get search suggestions. WITHOUT YouTube Data API v3.
MIT License
741 stars 164 forks source link

Direct stream URLs not working #134

Closed dscrofts closed 2 years ago

dscrofts commented 3 years ago

Hi,

In the async documentation there is an example for getting the direct stream URL of a video.

This works fine for the example and can be verified by pasting the generated URL into VLC which plays the audio. However, trying with other YouTube videos does not seem to produce a working URL (VLC throws an error). Here is a minimal example:

import asyncio
from youtubesearchpython.__future__ import *
fetcher = StreamURLFetcher()

asyncio.run(fetcher.getJavaScript())
video = asyncio.run(Video.get("https://www.youtube.com/watch?v=kXYiU_JCYtU"))
url = asyncio.run(fetcher.get(video, 251))
print(url)

I've tried testing with several other videos and their respective direct URLs but no luck with them either.

mytja commented 3 years ago

Direct stream URL is working for me. Did you install latest release from git?

pip install git+https://github.com/alexmercerind/youtube-search-python

EDIT: Applying signature is failing on GitHub Actions servers, but not on my computer. It is strange, and could be error while retrieving JavaScript on our side, or error when deciphering and applying signature on PyTube's side

dscrofts commented 3 years ago

@mytja Yes I have the latest release (1.5.0).

I think it may be an issue with the signature although I'm not 100% sure. For example, the following produces a working direct URL:

youtube-dl --get-url -f 251 https://www.youtube.com/watch?v=kXYiU_JCYtU

What's strange is I've tested with many different videos and some work while others don't.

Edit: I'm also using the latest PyTube version 11.0.1

mytja commented 3 years ago

@mytja Yes I have the latest release (1.5.0).

I think it may be an issue with the signature although I'm not 100% sure. For example, the following produces a working direct URL:

youtube-dl --get-url -f 251 https://www.youtube.com/watch?v=kXYiU_JCYtU

What's strange is I've tested with many different videos and some work while others don't.

Edit: I'm also using the latest PyTube version 11.0.1

This is strange. No matter what I can't reproduce it. It is indeed signature applying that is failing. I will try to fix it, but I can't promise anything.

mytja commented 2 years ago

I installed Linux on my machine and now I can for some reason reproduce it. I already pushed some things, but those aren't optimized. I will optimize everything very soon.

dscrofts commented 2 years ago

Sadly it seems like something has changed again (just today!)

Running the example above yields:

$ python3 test.py
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/youtubesearchpython/core/streamurlfetcher.py", line 86, in _decipher
    apply_signature(
  File "/usr/local/lib/python3.10/site-packages/pytube/extract.py", line 409, in apply_signature
    cipher = Cipher(js=js)
  File "/usr/local/lib/python3.10/site-packages/pytube/cipher.py", line 44, in __init__
    self.throttling_array = get_throttling_function_array(js)
  File "/usr/local/lib/python3.10/site-packages/pytube/cipher.py", line 323, in get_throttling_function_array
    str_array = throttling_array_split(array_raw)
  File "/usr/local/lib/python3.10/site-packages/pytube/parser.py", line 158, in throttling_array_split
    match_start, match_end = match.span()
AttributeError: 'NoneType' object has no attribute 'span'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/dan/Downloads/test.py", line 9, in <module>
    url = asyncio.run(fetcher.get(video, 251))
  File "/usr/local/Cellar/python@3.10/3.10.0_2/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/Cellar/python@3.10/3.10.0_2/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/base_events.py", line 641, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.10/site-packages/youtubesearchpython/__future__/streamurlfetcher.py", line 55, in get
    self._getDecipheredURLs(videoFormats)
  File "/usr/local/lib/python3.10/site-packages/youtubesearchpython/core/streamurlfetcher.py", line 51, in _getDecipheredURLs
    self._decipher()
  File "/usr/local/lib/python3.10/site-packages/youtubesearchpython/core/streamurlfetcher.py", line 96, in _decipher
    self._decipher(retry = True)
  File "/usr/local/lib/python3.10/site-packages/youtubesearchpython/core/streamurlfetcher.py", line 92, in _decipher
    raise e
  File "/usr/local/lib/python3.10/site-packages/youtubesearchpython/core/streamurlfetcher.py", line 86, in _decipher
    apply_signature(
  File "/usr/local/lib/python3.10/site-packages/pytube/extract.py", line 409, in apply_signature
    cipher = Cipher(js=js)
  File "/usr/local/lib/python3.10/site-packages/pytube/cipher.py", line 44, in __init__
    self.throttling_array = get_throttling_function_array(js)
  File "/usr/local/lib/python3.10/site-packages/pytube/cipher.py", line 323, in get_throttling_function_array
    str_array = throttling_array_split(array_raw)
  File "/usr/local/lib/python3.10/site-packages/pytube/parser.py", line 158, in throttling_array_split
    match_start, match_end = match.span()
AttributeError: 'NoneType' object has no attribute 'span'
mytja commented 2 years ago

Yeah, seen this. It's an issue with PyTube itself (https://github.com/pytube/pytube/issues/1163). I deployed a patch onto my fork, and you can use it meanwhile

pip install git+https://github.com/mytja/pytube
dscrofts commented 2 years ago

Looks like this issue is cropping up again :(

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/youtubesearchpython/core/streamurlfetcher.py", line 86, in _decipher
    apply_signature(
  File "/usr/local/lib/python3.10/site-packages/pytube/extract.py", line 409, in apply_signature
    cipher = Cipher(js=js)
  File "/usr/local/lib/python3.10/site-packages/pytube/cipher.py", line 44, in __init__
    self.throttling_array = get_throttling_function_array(js)
  File "/usr/local/lib/python3.10/site-packages/pytube/cipher.py", line 323, in get_throttling_function_array
    str_array = throttling_array_split(array_raw)
  File "/usr/local/lib/python3.10/site-packages/pytube/parser.py", line 158, in throttling_array_split
    match_start, match_end = match.span()
AttributeError: 'NoneType' object has no attribute 'span'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/dan/Downloads/test.py", line 7, in <module>
    url = asyncio.run(fetcher.get(video, 251))
  File "/usr/local/Cellar/python@3.10/3.10.0_2/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/Cellar/python@3.10/3.10.0_2/Frameworks/Python.framework/Versions/3.10/lib/python3.10/asyncio/base_events.py", line 641, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.10/site-packages/youtubesearchpython/__future__/streamurlfetcher.py", line 55, in get
    self._getDecipheredURLs(videoFormats)
  File "/usr/local/lib/python3.10/site-packages/youtubesearchpython/core/streamurlfetcher.py", line 51, in _getDecipheredURLs
    self._decipher()
  File "/usr/local/lib/python3.10/site-packages/youtubesearchpython/core/streamurlfetcher.py", line 96, in _decipher
    self._decipher(retry = True)
  File "/usr/local/lib/python3.10/site-packages/youtubesearchpython/core/streamurlfetcher.py", line 92, in _decipher
    raise e
  File "/usr/local/lib/python3.10/site-packages/youtubesearchpython/core/streamurlfetcher.py", line 86, in _decipher
    apply_signature(
  File "/usr/local/lib/python3.10/site-packages/pytube/extract.py", line 409, in apply_signature
    cipher = Cipher(js=js)
  File "/usr/local/lib/python3.10/site-packages/pytube/cipher.py", line 44, in __init__
    self.throttling_array = get_throttling_function_array(js)
  File "/usr/local/lib/python3.10/site-packages/pytube/cipher.py", line 323, in get_throttling_function_array
    str_array = throttling_array_split(array_raw)
  File "/usr/local/lib/python3.10/site-packages/pytube/parser.py", line 158, in throttling_array_split
    match_start, match_end = match.span()
AttributeError: 'NoneType' object has no attribute 'span'

Another PyTube issue perhaps?

mytja commented 2 years ago

PyTube just released a patch. Try to update PyTube:

pip install --upgrade pytube
dscrofts commented 2 years ago

Sadly issues with PyTube cropping up again as per https://github.com/pytube/pytube/issues/1199

Edit: also see https://github.com/alexmercerind/youtube-search-python/pull/154#issuecomment-1019023819

mytja commented 2 years ago

@dscrofts #155 will end this finally (I hope). I used ytdlp, as it is much much faster and also really stable and very well maintained (last commit was like 15 hours ago). This will require installation of ytdlp: pip install yt-dlp

Also from my tests, it decrypted the URLs up to 5x faster (most likely also due to my optimizations of the code). The StreamURLFetcher also automatically re-fetches the video info if it is age-restricted.

dscrofts commented 2 years ago

@mytja really nice! Testing it out now and it seems fast and stable. Great work!

dscrofts commented 2 years ago

@mytja actually after further testing it doesn't seem to be working as intended! It's fast and generates the URL but the URL isn't valid/won't play in VLC. Using youtube-dl as per this comment https://github.com/alexmercerind/youtube-search-python/issues/134#issuecomment-962479903 does produce a working URL. I'm thinking this is a similar issue to before with the signature.

Edit: and occasionally even youtube-dl direct links seem to be failing! It seems a bit random. I'll keep testing and see what I can find.

mytja commented 2 years ago

Hmmm. This is interesting. For me it works as intended, and all links are valid. I will do further testing and will see what I can find.

dscrofts commented 2 years ago

Hmmm. This is interesting. For me it works as intended, and all links are valid. I will do further testing and will see what I can find.

I'm testing with: pip install git+https://github.com/alexmercerind/youtube-search-python (youtube-search-python 1.6.0) pip install yt-dlp (yt-dlp 2022.1.21)

mytja commented 2 years ago

Hmmm. This is interesting. For me it works as intended, and all links are valid. I will do further testing and will see what I can find.

I'm testing with: pip install git+https://github.com/alexmercerind/youtube-search-python (youtube-search-python 1.6.0) pip install yt-dlp (yt-dlp 2022.1.21)

Ohhhh. You are not on right branch. It seems like you are still using PyTube version as it hasn't been merged into main branch.

Try with pip install git+https://github.com/alexmercerind/youtube-search-python@ytdlp

You said you installed forks of PyTube that should resolve this issue, but you also said none of them worked. Well, maybe PyTube deciphering didn't work correctly, but it still outputed a URL.

dscrofts commented 2 years ago

@mytja Whoops you are right! I think something messed up on my end with python/pip versions (using homebrew). I just reinstalled the pip packages and it looks like the direct URLs are working again in VLC. And wow it's fast now! At least twice as fast to retrieve the direct URLs from my testing. :D

mytja commented 2 years ago

I've published everything to PyPi. Now you are able to install it using pip install youtube-search-python --upgrade

I hope this error doesn't occour again.