search

ytdl-org / youtube-dl

Command-line program to download videos from YouTube.com and other video sites
http://ytdl-org.github.io/youtube-dl/
The Unlicense
131.69k stars 9.97k forks source link

[ciscolive] KeyError due to outdated rainfocus api keys and failing tests #30864

Open L0wbyte opened 2 years ago

L0wbyte commented 2 years ago

Checklist

Verbose log

$ youtube-dl -v https://www.ciscolive.com/on-demand/on-demand-library.html?search=#/session/16360600004400017rMx
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-v', 'https://www.ciscolive.com/on-demand/on-demand-library.html?search=#/session/16360600004400017rMx']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2021.12.17
[debug] Python version 3.6.9 (CPython) - Linux-4.15.0-163-generic-x86_64-with-Ubuntu-18.04-bionic
[debug] exe versions: ffmpeg 3.4.8, ffprobe 3.4.8
[debug] Proxy map: {}
[CiscoLiveSession] 16360600004400017rMx: Downloading webpage
ERROR: An extractor error has occurred. (caused by KeyError('items',)); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
Traceback (most recent call last):
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 534, in extract
    ie_result = self._real_extract(url)
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/ciscolive.py", line 93, in _real_extract
    return self._parse_rf_item(rf_result['items'][0])
KeyError: 'items'
Traceback (most recent call last):
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 534, in extract
    ie_result = self._real_extract(url)
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/ciscolive.py", line 93, in _real_extract
    return self._parse_rf_item(rf_result['items'][0])
KeyError: 'items'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 815, in wrapper
    return func(self, *args, **kwargs)
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 836, in __extract_info
    ie_result = ie.extract(url)
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 547, in extract
    raise ExtractorError('An extractor error has occurred.', cause=e)
youtube_dl.utils.ExtractorError: An extractor error has occurred. (caused by KeyError('items',)); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

Description

Raising this bug report to submit (and link) a pull request fixing the issues with the ciscolive extractor outlined below:

youtube-dl fails to find valid video items for download when calling rainfocus api's (eg https://events.rainfocus.com/api/session).

# api response as captured with --dump-pages
{"responseCode":"104","responseMessage":"No session to display"}

after testing with various inputs this appears to be caused by invalid api tokens (these are currently static in the extractor, but have changed). https://github.com/ytdl-org/youtube-dl/blob/master/youtube_dl/extractor/ciscolive.py#L24-L25 these tokens have been present since the initial commit in Oct 2018 https://github.com/ytdl-org/youtube-dl/blob/05bd5e9c77e0e8acb95f47396be4c970fc9f39c4/youtube_dl/extractor/ciscolive.py
however, it may be more ideal to fetch these tokens 'live' just as a browser would.




additionally tests executed by

python test/test_download.py TestDownload.test_CiscoLiveSession
python test/test_download.py TestDownload.test_CiscoLiveSearch

both fail due to a mixture of the above problem, and the sessions/video's & search parameters referenced are no longer valid on the ciscolive site.

CiscoLiveSession video's/sessions appear to eventually be removed/become inaccessible on the ciscolive site (likely after the technical material becomes less relevant after 3-4 years). The oldest searchable event appears to now be from 2018. meaning session id's such as 1423353499155001FoSs (the test url for this class) results in a 104 - No session to display even if substituted into the new url format. https://ciscolive.cisco.com/on-demand-library/?#/session/1423353499155001FoSs -> https://www.ciscolive.com/on-demand/on-demand-library.html?search=#/session/1423353499155001FoSs tests should be updated to pass and repeatable within the browser (the previous tests url's are no longer valid)

CiscoLiveSearch old: https://ciscolive.cisco.com/on-demand-library/?search.event=ciscoliveus2018&search.technicallevel=scpsSkillLevel_aintroductory&search.focus=scpsSessionFocus_designAndDeployment#/ new: https://www.ciscolive.com/on-demand/on-demand-library.html?search.event=1636046385176005FbBU&search.technology=scpsTechnology_dataCenter&search.technicallevel=scpsSkillLevel_aintroductory#/ note search.event now contains id's 1636046385176005FbBU vs ciscoliveus2018 tests should be updated to pass and repeatable within the browser (the previous tests url's are no longer valid)




lastly the domain ciscolive.cisco.com is no longer valid for the site (dns returns an NXDOMAIN status) the correct domain is https://www.ciscolive.com/ and the url path has changed.

old: https://ciscolive.cisco.com/on-demand-library/?#/session/1423353499155001FoSs new: https://www.ciscolive.com/on-demand/on-demand-library.html?search=#/session/16360600004400017rMx


i will submit a pull request to address these shortly.

dirkf commented 2 years ago

Thanks!

L0wbyte commented 4 months ago

reviving this fix after 2 years ;) the cisco search api changed format for search results, so latest changes fix this.

additionally synced from master.

@dirkf mind taking a look and perhaps run the CI again?