search

ytdl-org / youtube-dl

Command-line program to download videos from YouTube.com and other video sites
http://ytdl-org.github.io/youtube-dl/
The Unlicense
132.32k stars 10.03k forks source link

Unable to download Youtube stories anymore #30725

Closed gbr000 closed 2 years ago

gbr000 commented 2 years ago

Verbose log

~$ youtube-dl -v https://youtube.com/shorts/9Vsdft81Q6w?feature=share                                       [debug] System config: []                             [debug] User config: []
[debug] Custom config: []                             [debug] Command-line args: ['-v', 'https://youtube.com/shorts/9Vsdft81Q6w?feature=share']                   [debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[debug] youtube-dl version 2021.12.17
[debug] Python version 3.10.2 (CPython) - Linux-4.4.177-22825772-aarch64-with-libc
[debug] exe versions: ffmpeg 5.0, ffprobe 5.0
[debug] Proxy map: {}
[youtube:tab] shorts: Downloading webpage
ERROR: Unable to recognize tab page; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
Traceback (most recent call last):
  File "/data/data/com.termux/files/usr/lib/python3.10/site-packages/youtube_dl/YoutubeDL.py", line 815, in wrapper
    return func(self, *args, **kwargs)                  File "/data/data/com.termux/files/usr/lib/python3.10/site-packages/youtube_dl/YoutubeDL.py", line 836, in __extract_info
    ie_result = ie.extract(url)
  File "/data/data/com.termux/files/usr/lib/python3.10/site-packages/youtube_dl/extractor/common.py", line 534, in extract
    ie_result = self._real_extract(url)
  File "/data/data/com.termux/files/usr/lib/python3.10/site-packages/youtube_dl/extractor/youtube.py", line 2862, in _real_extract
    raise ExtractorError('Unable to recognize tab page')
youtube_dl.utils.ExtractorError: Unable to recognize tab page; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

Description

This issue has persisted for about 6 months around. I wasn't able to fix it. So my turnaround was to use third-party websites where I could do download the story videos. Normal videos just work as expected.

I tried both manually and scripting to do the same and both failed.

my ffmpeg is 5.0

I don't have any more clues

gbr000 commented 2 years ago

It works!

I'll need to edit my bash script to delete part of the URL. Any possibility to implement that in the future youtube-dl releases?

dirkf commented 2 years ago

The next release will handle YT 'short' format URLs, already fixed in git master (PR #30577).

If that's what it takes to access YT 'stories' then that's the answer. 'anymore' in the issue title is confusing, because I don't think that format was ever supported before.

gbr000 commented 2 years ago

I was able to download Youtube stories for months before, inclusive I have a folder in which I put all my video stories there.

A similar issue happened in a Reddit post where a user said somewhere in the lines "youtube-dl before handled European cookies pop-up then suddenly stop supporting it"

I know it always worked unless we constantly live in a Mandela effect, which that will be awkward.

dirkf commented 2 years ago

What distinguishes a 'story' for yt-dl? Example URLs, please, since the YT site is silent about it.

If stories are universally shared as /shorts/..., then the merged PR which implements this

Simple extension of YoutubeIE._VALID_URL to cover new Shorts format https://youtube.com/shorts/{id}.

already resolves the issue.

pukkandan commented 2 years ago

YouTube stories are not the same as shorts. Stories have the same URL as normal videos, but expire after a week or so, and normally opens only in the app. We can fore it to open in browser (and make it work in ytdl) by adding a params=8AEB query to the URL.

But the URL given by OP is a short and not a story. So I assume that is what they meant to ask about

gbr000 commented 2 years ago

youtube-dl still giving errors with /shorts/ID URLs. --> correction I'm talking about shorts but some user said that stories and shorts are not the same, if they have the same URL [Why aren't they the same?] <--

youtube-dl -f 247+251 https://www.youtube.com/shorts/UEpd3l0sxPc --> doesn't work! <--

youtube-dl -f 247+251 UEpd3l0sxPc - - > works! < - -

But your reply was: "It will be implemented" and "already resolves the issue." and that's not true!!

Nothing is implemented I still need to fix every single URL manually every time.

When you will decided to FIX that!

pukkandan commented 2 years ago

https://github.com/ytdl-org/youtube-dl/pull/30577#issuecomment-1184701287

dirkf commented 2 years ago

YouTube stories are not the same as shorts. Stories have the same URL as normal videos, ...

So do we have to check for some error return and retry with params=8AEB if it wasn't used before?

pukkandan commented 2 years ago

So do we have to check for some error return and retry with params=8AEB if it wasn't used before?

yt-dlp always passes that param - it doesnt seem to current cause any issues with normal videos. @coletdjnz is the expert though

gbr000 commented 2 years ago

Why don't you use Python to remove /shorts/ automatically, and/or modify the whole URL string in case of youtube-dl giving an error?

Just an update here, I'm using Windows 10 now fully updated, and youtube-dl with not extra args.

ghost commented 1 year ago

YouTube stories are not the same as shorts. Stories have the same URL as normal videos, but expire after a week or so, and normally opens only in the app. We can fore it to open in browser (and make it work in ytdl) by adding a params=8AEB query to the URL.

I would not recommend using this. I noticed that YT-DLP is using it, but it has the side effect of breaking viewCount. so that value will no longer be in the output.

pukkandan commented 1 year ago

(you're back 👋)

YouTube stories are not the same as shorts. Stories have the same URL as normal videos, but expire after a week or so, and normally opens only in the app. We can fore it to open in browser (and make it work in ytdl) by adding a params=8AEB query to the URL.

@4cq2: I would not recommend using this. I noticed that YT-DLP is using it, but it has the side effect of breaking viewCount. so that value will no longer be in the output.

Nope

❯ yt-dlp test:youtube -O id,view_count=
id = "BaW_jenozKc"
view_count = 55266
dirkf commented 1 year ago

With/out this change:

         webpage = self._download_webpage(
-            webpage_url + '&bpctr=9999999999&has_verified=1', video_id, fatal=False)
+            webpage_url, video_id, fatal=False, query={
+                'bpctr': 9999999999,
+                'has_verified': 1,
+                'params': '8AEB',
+            })

yt-dl master gets the view_count. Possibly it disappears from some API result?

$ python -m youtube_dl --get-filename -o '%(view_count)s' test:youtube
55268
$
ghost commented 1 year ago

Nope

❯ yt-dlp test:youtube -O id,view_count=
id = "BaW_jenozKc"
view_count = 55266

sigh, I see you havent changed. pity. at any rate, with current YT-DLP master, a request like this is made:

POST /youtubei/v1/player?key=AIzaSyA8eiZmM1FaDVjRy-df2KTyQ_vz_yYM39w&prettyPrint=false HTTP/1.1
Accept-Encoding: identity
Accept-Language: en-us,en;q=0.5
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Connection: close
Content-Type: application/json
Cookie: PREF=hl=en&tz=UTC; CONSENT=YES+cb.20210328-17-p0.en+FX+317; GPS=1; YSC=zUITQDTUE18; VISITOR_INFO1_LIVE=bsbbNUhgT0E
Host: www.youtube.com
Origin: https://www.youtube.com
Sec-Fetch-Mode: navigate
User-Agent: com.google.android.youtube/17.31.35 (Linux; U; Android 11) gzip
X-Youtube-Client-Name: 3
X-Youtube-Client-Version: 17.31.35

{
  "context": {
    "client": {
      "clientName": "ANDROID",
      "clientVersion": "17.31.35",
      "androidSdkVersion": 30,
      "userAgent": "com.google.android.youtube/17.31.35 (Linux; U; Android 11) gzip",
      "hl": "en",
      "timeZone": "UTC",
      "utcOffsetMinutes": 0
    }
  },
  "videoId": "BaW_jenozKc",
  "params": "8AEB",
  "playbackContext": {
    "contentPlaybackContext": {
      "html5Preference": "HTML5_PREF_WANTS"
    }
  },
  "contentCheckOk": true,
  "racyCheckOk": true
}

viewCount is normally found under videoDetails. here is that part from the response JSON:

  "videoDetails": {
    "videoId": "BaW_jenozKc",
    "title": "youtube-dl test video \"'/\\ä↭𝕐",
    "lengthSeconds": "10",
    "keywords": [
      "youtube-dl"
    ],
    "channelId": "UCLqxVugv74EIW3VWh2NOa3Q",
    "isOwnerViewing": false,
    "shortDescription": "test chars:  \"'/\\ä↭𝕐\ntest URL: https://github.com/rg3/youtube-dl/issues/1892\n\nThis is a test video for youtube-dl.\n\nFor more information, contact phihag@phihag.de .",
    "isCrawlable": true,
    "thumbnail": {
      "thumbnails": [
        {
          "url": "https://i.ytimg.com/vi/BaW_jenozKc/default.jpg",
          "width": 120,
          "height": 90
        },
        {
          "url": "https://i.ytimg.com/vi/BaW_jenozKc/mqdefault.jpg",
          "width": 320,
          "height": 180
        },
        {
          "url": "https://i.ytimg.com/vi/BaW_jenozKc/hqdefault.jpg",
          "width": 480,
          "height": 360
        },
        {
          "url": "https://i.ytimg.com/vi/BaW_jenozKc/sddefault.jpg",
          "width": 640,
          "height": 480
        }
      ]
    },
    "allowRatings": true,
    "author": "Philipp Hagemeister",
    "isPrivate": false,
    "isUnpluggedCorpus": false,
    "isLiveContent": false
  },

youll notice that viewCount is missing. So IF YT-DLP is returning viewCount, its NOT from the JSON API, its from something else. My previous comment is in reference to the JSON API ONLY, I dont care about anything else.

yt-dl master gets the view_count. Possibly it disappears from some API result?

As far as I understand YouTube-DL doesnt currently use the JSON API, as least not using the default options. So my previous comment was more of a caution against adopting the params value currently used by YT-DLP, should YouTube-DL decide to start using the JSON API.

dirkf commented 1 year ago

... Possibly it disappears from some API result?

That's a yes, then.

In both cases view_count is extracted from the hydration JSON in the page, I think, and not from a separate API response, although IIRC an API call may be made if the expected hydration JSON wasn't found. So generally view_count survives params=8AEB.

If someone shows an actual URL where params=8AEB has an effect, I'm happy to test the change shown above and commit something like it if appropriate.

ghost commented 1 year ago

I dont know how to trigger YouTube-DL to even use the JSON API, if in fact thats even possible. If someone can explain some circumstance and/or option that makes YouTube-DL do that, I am happy to run some tests.

dirkf commented 1 year ago

I'm happy to believe that the API behaves as you say.

An API request is made (search the extractor code for _call_api( for details) if the YT page is not available, or if it doesn't contain one of the expected hydration JSON blocks (initial player response, yt initial data), or if the page is age-gated. view_count, if present, is found if the initial player response was valid or from an interactionCount in the page metadata.

Given an example age-gated YT story URL, it would be possible to check whether

ghost commented 1 year ago

if the page is age-gated

Ah this works. So using this trigger, I was able to capture this request from YouTube-DL master:

POST /youtubei/v1/player?key=AIzaSyAO_FJ2SlqU8Q4STEHLGCilw_Y9_11qcW8 HTTP/1.1
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Accept-Encoding: identity
Accept-Language: en-us,en;q=0.5
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Connection: close
Content-Type: application/json
Cookie: CONSENT=YES+cb.20210328-17-p0.en+FX+683; GPS=1; YSC=PIPGcVvqCd8; VISITOR_INFO1_LIVE=k6qGh2M82ps
Host: www.youtube.com
Origin: https://www.youtube.com
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.111 Safari/537.36
X-Youtube-Client-Name: 85
X-Youtube-Client-Version: 2.0

{
  "context": {
    "client": {
      "clientName": "TVHTML5_SIMPLY_EMBEDDED_PLAYER",
      "clientVersion": "2.0",
      "hl": "en",
      "clientScreen": "EMBED"
    },
    "thirdParty": {
      "embedUrl": "https://google.com"
    }
  },
  "playbackContext": {
    "contentPlaybackContext": {
      "html5Preference": "HTML5_PREF_WANTS",
      "signatureTimestamp": 19459
    }
  },
  "contentCheckOk": true,
  "racyCheckOk": true,
  "videoId": "HtVdAasjOgU"
}

which is missing any params entry, and returns viewCount as expected:

"videoDetails": {
    "videoId": "HtVdAasjOgU",
    "title": "The Witcher 3: Wild Hunt - The Sword Of Destiny Trailer",
    "lengthSeconds": "142",
    "keywords": [
      "RPG",
      "E3 2014",
      "e3",
      "Gameplay Trailer",
      "witcher",
      "the witcher",
      "the witcher 3",
      "e3 trailer",
      "e3 2014 trailer",
      "witcher 3 e3 trailer",
      "witcher 3 e3",
      "witcher e3",
      "sword of destiny",
      "wild hunt",
      "thewitcher",
      "trailer",
      "gameplay trailer"
    ],
    "channelId": "UCzybXLxv08IApdjdN0mJhEg",
    "isOwnerViewing": false,
    "shortDescription": "For more information visit:\nBuy now: http://buy.thewitcher.com\nFacebook: https://www.fb.com/thewitcher\nTwitter: https://twitter.com/witchergame\nHomepage: http://thewitcher.com/witcher3/\n\nThe game is scheduled to launch May 19, 2015 and will be available on Xbox One, PC and PlayStation®4.\n\nAbout the Game\n\nThe Witcher 3: Wild Hunt is a story-driven, next-generation open world role-playing game, set in a fantasy universe full of meaningful choices and impactful consequences. In The Witcher 3, you play as Geralt of Rivia, one of a dying caste of monster hunters, and embark on an epic journey in a war-ravaged world that will inevitably lead you to confront a foe darker than anything humanity has faced so far—the Wild Hunt.\n\n--------------------------------------------------------------------------------------\nMusic from Sword of Destiny trailer : http://bit.ly/1lbwCNX\nCollector's Edition UNBOXING [Official] : http://bit.ly/1nX9Scu",
    "isCrawlable": true,
    "thumbnail": {
      "thumbnails": [
        {
          "url": "https://i.ytimg.com/vi/HtVdAasjOgU/default.jpg",
          "width": 120,
          "height": 90
        },
        {
          "url": "https://i.ytimg.com/vi/HtVdAasjOgU/mqdefault.jpg",
          "width": 320,
          "height": 180
        },
        {
          "url": "https://i.ytimg.com/vi/HtVdAasjOgU/hqdefault.jpg",
          "width": 480,
          "height": 360
        },
        {
          "url": "https://i.ytimg.com/vi/HtVdAasjOgU/sddefault.jpg",
          "width": 640,
          "height": 480
        },
        {
          "url": "https://i.ytimg.com/vi/HtVdAasjOgU/maxresdefault.jpg",
          "width": 1920,
          "height": 1080
        }
      ]
    },
    "allowRatings": true,
    "viewCount": "5545589",
    "author": "The Witcher",
    "isPrivate": false,
    "isUnpluggedCorpus": false,
    "isLiveContent": false
  }

however if you add "params": "8AEB" as shown in the YT-DLP request above, you end up missing viewCount:

"videoDetails": {
    "videoId": "HtVdAasjOgU",
    "title": "The Witcher 3: Wild Hunt - The Sword Of Destiny Trailer",
    "lengthSeconds": "142",
    "keywords": [
      "RPG",
      "E3 2014",
      "e3",
      "Gameplay Trailer",
      "witcher",
      "the witcher",
      "the witcher 3",
      "e3 trailer",
      "e3 2014 trailer",
      "witcher 3 e3 trailer",
      "witcher 3 e3",
      "witcher e3",
      "sword of destiny",
      "wild hunt",
      "thewitcher",
      "trailer",
      "gameplay trailer"
    ],
    "channelId": "UCzybXLxv08IApdjdN0mJhEg",
    "isOwnerViewing": false,
    "shortDescription": "For more information visit:\nBuy now: http://buy.thewitcher.com\nFacebook: https://www.fb.com/thewitcher\nTwitter: https://twitter.com/witchergame\nHomepage: http://thewitcher.com/witcher3/\n\nThe game is scheduled to launch May 19, 2015 and will be available on Xbox One, PC and PlayStation®4.\n\nAbout the Game\n\nThe Witcher 3: Wild Hunt is a story-driven, next-generation open world role-playing game, set in a fantasy universe full of meaningful choices and impactful consequences. In The Witcher 3, you play as Geralt of Rivia, one of a dying caste of monster hunters, and embark on an epic journey in a war-ravaged world that will inevitably lead you to confront a foe darker than anything humanity has faced so far—the Wild Hunt.\n\n--------------------------------------------------------------------------------------\nMusic from Sword of Destiny trailer : http://bit.ly/1lbwCNX\nCollector's Edition UNBOXING [Official] : http://bit.ly/1nX9Scu",
    "isCrawlable": true,
    "thumbnail": {
      "thumbnails": [
        {
          "url": "https://i.ytimg.com/vi/HtVdAasjOgU/default.jpg",
          "width": 120,
          "height": 90
        },
        {
          "url": "https://i.ytimg.com/vi/HtVdAasjOgU/mqdefault.jpg",
          "width": 320,
          "height": 180
        },
        {
          "url": "https://i.ytimg.com/vi/HtVdAasjOgU/hqdefault.jpg",
          "width": 480,
          "height": 360
        },
        {
          "url": "https://i.ytimg.com/vi/HtVdAasjOgU/sddefault.jpg",
          "width": 640,
          "height": 480
        },
        {
          "url": "https://i.ytimg.com/vi/HtVdAasjOgU/maxresdefault.jpg",
          "width": 1920,
          "height": 1080
        }
      ]
    },
    "allowRatings": true,
    "author": "The Witcher",
    "isPrivate": false,
    "isUnpluggedCorpus": false,
    "isLiveContent": false
  }

Also note, I dont think I have mentioned this until now, but it seems the params value is important, but as demonstrated you want to avoid the value YT-DLP is using. From my own research YouTube seems to be A/B testing this key, and if it is missing or has an improper value, it can lead to missing values like this, or even omitting serious items like adaptiveFormats.

dirkf commented 1 year ago

As shown, the API request isn't currently using the params query parameter.

Again, as no-one has ever provided an actual YT Story URL for this (or any other) issue, let alone an age-gated one, we can't easily see what value YT is using or test how yt-dl works in either case.

dirkf commented 1 year ago

And also, YouTube withdrew the Stories feature in June.