search

joetats / youtube_search

Tool for searching for youtube videos to avoid using their heavily rate-limited API
MIT License
220 stars 64 forks source link

KeyError: 'descriptionSnippet' - Search is inconsistent #12

Closed shneeba closed 4 years ago

shneeba commented 4 years ago

Hi guys,

Really hate to be posting here as it's probably a config I've got somewhere, but I can't for the life of me get this working to return any data on a search term properly, it seems almost random and I can't think of why it's doing this.

>>> YoutubeSearch('search terms', max_results=1).to_dict()
[{'id': '6ihNMNTBdjk', 'thumbnails': ['https://i.ytimg.com/vi/6ihNMNTBdjk/hqdefault.jpg?sqp=-oaymwEiCKgBEF5IWvKriqkDFQgBFQAAAAAYASUAAMhCPQCAokN4AQ==&rs=AOn4CLDohvKhRqlBMKaSWiA0_jIMQXVbqA', 'https://i.ytimg.com/vi/6ihNMNTBdjk/hqdefault.jpg?sqp=-oaymwEiCMQBEG5IWvKriqkDFQgBFQAAAAAYASUAAMhCPQCAokN4AQ==&rs=AOn4CLCD3ayrPiREK6X76-vSClblWE4QIw', 'https://i.ytimg.com/vi/6ihNMNTBdjk/hqdefault.jpg?sqp=-oaymwEjCPYBEIoBSFryq4qpAxUIARUAAAAAGAElAADIQj0AgKJDeAE=&rs=AOn4CLBydsvefOrgm0l0RjEXwBEfe57s-w', 'https://i.ytimg.com/vi/6ihNMNTBdjk/hqdefault.jpg?sqp=-oaymwEjCNACELwBSFryq4qpAxUIARUAAAAAGAElAADIQj0AgKJDeAE=&rs=AOn4CLDKfWtAZYgHu1TO3oXRxWtzP0CjPA'], 'title': 'How to improve your search terms', 'long_desc': 'How to improve your ', 'channel': 'Brock University Library', 'duration': '3:05', 'views': '10,766 views', 'url_suffix': '/watch?v=6ihNMNTBdjk'}]
>>>
>>> YoutubeSearch('prizm', max_results=1).to_dict()
[{'id': 'Z3BWqUZ3Ka0', 'thumbnails': ['https://i.ytimg.com/vi/Z3BWqUZ3Ka0/hqdefault.jpg?sqp=-oaymwEiCKgBEF5IWvKriqkDFQgBFQAAAAAYASUAAMhCPQCAokN4AQ==&rs=AOn4CLCn1Iqp-WmH2AO5CeV6K_MdCCY6EA', 'https://i.ytimg.com/vi/Z3BWqUZ3Ka0/hqdefault.jpg?sqp=-oaymwEiCMQBEG5IWvKriqkDFQgBFQAAAAAYASUAAMhCPQCAokN4AQ==&rs=AOn4CLDmD-SwIRNtAXwphN2KaaVA1_gdzQ', 'https://i.ytimg.com/vi/Z3BWqUZ3Ka0/hqdefault.jpg?sqp=-oaymwEjCPYBEIoBSFryq4qpAxUIARUAAAAAGAElAADIQj0AgKJDeAE=&rs=AOn4CLBS2JJqU2bWn6EvRhCLraqR3EUnZg', 'https://i.ytimg.com/vi/Z3BWqUZ3Ka0/hqdefault.jpg?sqp=-oaymwEjCNACELwBSFryq4qpAxUIARUAAAAAGAElAADIQj0AgKJDeAE=&rs=AOn4CLBa1DLaU3WLiVJf4VqBPo1_ppR45g'], 'title': 'PRIZM - All Night (Official Music Video)', 'long_desc': 'PRIZM', 'channel': 'FiXT Neon', 'duration': '3:53', 'views': '19,005 views', 'url_suffix': '/watch?v=Z3BWqUZ3Ka0'}]
>>>
>>> YoutubeSearch('prizm all night', max_results=1).to_dict()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.7/dist-packages/youtube_search/__init__.py", line 10, in __init__
    self.videos = self.search()
  File "/usr/local/lib/python3.7/dist-packages/youtube_search/__init__.py", line 19, in search
    results = self.parse_html(response)
  File "/usr/local/lib/python3.7/dist-packages/youtube_search/__init__.py", line 48, in parse_html
    res["long_desc"] = video_data["descriptionSnippet"]["runs"][0]["text"]
KeyError: 'descriptionSnippet'
>>> YoutubeSearch('prizm all', max_results=1).to_dict()
[{'id': 'Z3BWqUZ3Ka0', 'thumbnails': ['https://i.ytimg.com/vi/Z3BWqUZ3Ka0/hqdefault.jpg?sqp=-oaymwEiCKgBEF5IWvKriqkDFQgBFQAAAAAYASUAAMhCPQCAokN4AQ==&rs=AOn4CLCn1Iqp-WmH2AO5CeV6K_MdCCY6EA', 'https://i.ytimg.com/vi/Z3BWqUZ3Ka0/hqdefault.jpg?sqp=-oaymwEiCMQBEG5IWvKriqkDFQgBFQAAAAAYASUAAMhCPQCAokN4AQ==&rs=AOn4CLDmD-SwIRNtAXwphN2KaaVA1_gdzQ', 'https://i.ytimg.com/vi/Z3BWqUZ3Ka0/hqdefault.jpg?sqp=-oaymwEjCPYBEIoBSFryq4qpAxUIARUAAAAAGAElAADIQj0AgKJDeAE=&rs=AOn4CLBS2JJqU2bWn6EvRhCLraqR3EUnZg', 'https://i.ytimg.com/vi/Z3BWqUZ3Ka0/hqdefault.jpg?sqp=-oaymwEjCNACELwBSFryq4qpAxUIARUAAAAAGAElAADIQj0AgKJDeAE=&rs=AOn4CLBa1DLaU3WLiVJf4VqBPo1_ppR45g'], 'title': 'PRIZM - All Night (Official Music Video)', 'long_desc': 'PRIZM', 'channel': 'FiXT Neon', 'duration': '3:53', 'views': '19,005 views', 'url_suffix': '/watch?v=Z3BWqUZ3Ka0'}]
>>>
>>> YoutubeSearch('prizm', max_results=1).to_dict()
[]
>>> YoutubeSearch('prizm want', max_results=1).to_dict()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.7/dist-packages/youtube_search/__init__.py", line 10, in __init__
    self.videos = self.search()
  File "/usr/local/lib/python3.7/dist-packages/youtube_search/__init__.py", line 19, in search
    results = self.parse_html(response)
  File "/usr/local/lib/python3.7/dist-packages/youtube_search/__init__.py", line 48, in parse_html
    res["long_desc"] = video_data["descriptionSnippet"]["runs"][0]["text"]
KeyError: 'descriptionSnippet'
>>>

Again, sorry to post here but I'm properly stumped.

Cheers

joetats commented 4 years ago

Interesting, yeah the latest change was just to get it working again quickly. Might need to use some more .get() calls to address the missing data fields, but that will be frustrating to use if the data is only there some of the time.

We're at the mercy of what youtube sends to the browser, so inconsistency is the price we pay for not using the API haha. I might try and run some tests to see if it's worth sending multiple requests off until all the data fields are filled - it currently does that to make sure the javascript object is present but that can be extended if there isn't a huge performance hit.

shneeba commented 4 years ago

Thanks so much for the quick response, was starting to go a little crazy trying to get this to work and not done much with python before.

Yeah I completely understand we're at the mercy of what's sent to the browser and thanks again for this little tool. I'll try and work my way around it for now.

AsjadSiddiqui commented 4 years ago

I have the same issue whenever I search for, e.g, Eastside, or Kinda Crazy.

araa47 commented 4 years ago

I have the same issue here, would recommend using dict.get("key", fallback-value), to avoid these errors.

joetats commented 4 years ago

Thank you @araa47 ! Closing this issue