search

ytdl-org / youtube-dl

Command-line program to download videos from YouTube.com and other video sites
http://ytdl-org.github.io/youtube-dl/
The Unlicense
129.81k stars 9.79k forks source link

Youtube channel listing does not show previous uploaded streams, only uploaded videos #31326

Open alien999999999 opened 1 year ago

alien999999999 commented 1 year ago

Checklist

Verbose log

./youtube-dl -v --get-filename -o "%(upload_date)s, %(title)s, https://www.youtube.com/watch?v=%(id)s" "https://www.youtube.com/c/NASASpaceflightVideos"
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'-v', u'--get-filename', u'-o', u'%(upload_date)s, %(title)s, https://www.youtube.com/watch?v=%(id)s', u'https://www.youtube.com/c/NASASpaceflightVideos']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2021.12.17
[debug] Python version 2.7.18 (CPython) - Linux-5.10.46-desktop-1.mga7-x86_64-with-mageia-7-Official
[debug] exe versions: ffmpeg 4.1.6, ffprobe 4.1.6
[debug] Proxy map: {}
[debug] Default format spec: bestvideo+bestaudio/best
20221102, Full Stack and Ship 25 Cryogenic Testing _ SpaceX Boca Chica, https://www.youtube.com/watch?v=rk7z-my16r4
[debug] Default format spec: bestvideo+bestaudio/best
20221027, SpaceX Falcon Heavy Static Fires Ahead of USSF-44 Mission, https://www.youtube.com/watch?v=glw-AuKvMbs
[debug] Default format spec: bestvideo+bestaudio/best
20221027, Full Stack Cryo Testing _ SpaceX Boca Chica, https://www.youtube.com/watch?v=Nx4ZD1MqrTo
[debug] Default format spec: bestvideo+bestaudio/best
20221024, SpaceX Tests Raptor Rapid Relight Capability, https://www.youtube.com/watch?v=AM78w3FWC0Q
[debug] Default format spec: bestvideo+bestaudio/best
20221023, Medevac Helicopter Lands at Starbase for Training _ SpaceX Boca Chica, https://www.youtube.com/watch?v=DOy2rQxjPcY
[debug] Default format spec: bestvideo+bestaudio/best
20221022, How SpaceX is Returning Falcon Heavy to Flight, https://www.youtube.com/watch?v=L6aLXtBqMxg
[debug] Default format spec: bestvideo+bestaudio/best
20221021, SpaceX Removes Starship Tanks at LC-39A, https://www.youtube.com/watch?v=K-TVnL4SDFM
[debug] Default format spec: bestvideo+bestaudio/best
20221020, Starship 25 Rolled out and Starship 24 Re-Stacked _ SpaceX Boca Chica, https://www.youtube.com/watch?v=c7UpQ7vO8OM
[debug] Default format spec: bestvideo+bestaudio/best
20221018, Starship 24 Destacked Amid Potential Problems _ SpaceX Boca Chica, https://www.youtube.com/watch?v=dpydSsTgnfA
[debug] Default format spec: bestvideo+bestaudio/best
20221016, SpaceX Tests Raptor's Thrust Vector Control for Starship, https://www.youtube.com/watch?v=Lo1x7zYbGOM
[debug] Default format spec: bestvideo+bestaudio/best
20221014, SpaceX Troubleshoots Problems with Full Starship Stack, https://www.youtube.com/watch?v=CovWogC6OSw
[debug] Default format spec: bestvideo+bestaudio/best
20221012, Ship 24 Stacked Onto Booster 7 _ SpaceX Boca Chica, https://www.youtube.com/watch?v=ubII_K-QvUA
^C
ERROR: Interrupted by user

Description

So, when i download from a channel name, with --get-filename, in order to get listing, i only have the uploaded videos, not the previous livestreams. This is likely from a very recent change that Youtube has decided to put the previous live streams under the "live" tab... while the actual live streams (24x7 streams) are not there, or if they are, then they are well hidden...

Please can you make a change to also include the previous livestreams, if I pass a channel link? previous livestreams are kind of videos too... I disagree with how youtube changed things, and I hope that youtube-dl could give a listing for both; it used to have a dropdown select on videos to show all videos, but that's gone now...

If possible also a way to get actual live stream info, or even planned live stream info...

Thanks in advance

october262 commented 1 year ago

try with youtube-dl --yes-playlist "https://www.youtube.com/c/NASASpaceflightVideos"

dirkf commented 1 year ago

Presumably related: https://github.com/yt-dlp/yt-dlp/issues/5419.

@october262, your suggestion seems to give the same result. There are 4 playlists and they get enumerated as in OP's log:

$ python -m youtube_dl --yes-playlist --flat-playlist "https://www.youtube.com/c/NASASpaceflightVideos"
[youtube:tab] NASASpaceflightVideos: Downloading webpage
[download] Downloading playlist: NASASpaceflight - Home
[youtube:tab] playlist NASASpaceflight - Home: Downloading 4 videos
[download] Downloading video 1 of 4
[download] Downloading video 2 of 4
[download] Downloading video 3 of 4
[download] Downloading video 4 of 4
[download] Finished downloading playlist: NASASpaceflight - Home
$ python -m youtube_dl --yes-playlist --get-filename "https://www.youtube.com/c/NASASpaceflightVideos"
Full Stack and Ship 25 Cryogenic Testing _ SpaceX Boca Chica-rk7z-my16r4.mp4
SpaceX Falcon Heavy Static Fires Ahead of USSF-44 Mission-glw-AuKvMbs.webm
Full Stack Cryo Testing _ SpaceX Boca Chica-Nx4ZD1MqrTo.mp4
SpaceX Tests Raptor Rapid Relight Capability-AM78w3FWC0Q.mp4
Medevac Helicopter Lands at Starbase for Training _ SpaceX Boca Chica-DOy2rQxjPcY.mp4
How SpaceX is Returning Falcon Heavy to Flight-L6aLXtBqMxg.webm
SpaceX Removes Starship Tanks at LC-39A-K-TVnL4SDFM.mp4
Starship 25 Rolled out and Starship 24 Re-Stacked _ SpaceX Boca Chica-c7UpQ7vO8OM.mp4
Starship 24 Destacked Amid Potential Problems _ SpaceX Boca Chica-dpydSsTgnfA.mp4
SpaceX Tests Raptor's Thrust Vector Control for Starship-Lo1x7zYbGOM.mp4
SpaceX Troubleshoots Problems with Full Starship Stack-CovWogC6OSw.mp4
Ship 24 Stacked Onto Booster 7 _ SpaceX Boca Chica-ubII_K-QvUA.mp4
Booster 7 Lifted Onto the Launch Mount _ SpaceX Boca Chica-cvwRJcU2i78.mp4
What's Going On With Our Daily Videos and More _ NSF Update-3GTzN02CoFg.webm
Potential Lunar Starship Thruster Fired _ This Week at SpaceX McGregor-46YEqL2KTk8.mp4
Booster 7 Rolled Out for Launch (we hope) _ SpaceX Boca Chica-N3TMebPf_Vw.mp4
Starship 24 Thermal Protection Tiles Repaired _ SpaceX Boca Chica-XNqvFy-dUsw.webm
Another Starship Launch Tower in Florida Soon-PdzgSLkfLJs.mp4
Will Starship be the First Methane-Powered Rocket to Fly-YsGdi5IadNY.mp4
Brand New Falcon 9 Test Fired-VrLaKuZ5Vm0.mp4
Ship 24 Moved From Suborbital Pad B _ SpaceX Boca Chica-sVqpXZeFvaU.mp4
Hurricane Ian Hits Cape Canaveral-82b5H9bI5_k.mp4
SpaceX Tests Raptor Engine Rapid Relight-BO5Uu6xA3N8.mp4
Booster 9 and the Orbital Launch Mount Upgrades _ SpaceX Boca Chica-2wZ0ooflgBI.mp4
SpaceX's Vertical Raptor Stand Enables Rapid Test Firings-kXJrIZQP0cc.mp4
SpaceX Falcon 9 Recovered After Record 14th Flight-RPykfbJbxyE.mp4
Orbital Launch Mount Water Deluge System Tested _ SpaceX Boca Chica-OsbnYp0FYKU.mp4
^C
ERROR: Interrupted by user
$
dirkf commented 1 year ago

This analysis of some example YT channels may be relevant.

pukkandan commented 1 year ago

Since the last youtube redesign (2019?), youtube-dl (unlike yt-dlp) only downloads videos from the exact URL that is passed. So https://www.youtube.com/c/NASASpaceflightVideos downloads videos shown in the "featured" tab, and https://www.youtube.com/c/NASASpaceflightVideos/videos downloads videos in "videos" tab. So far, the videos tab contained all uploads of a channel. But the new redesign has further split this into videos, streams, and shorts tabs. So, to get live streams, all you need is to pass the /streams URL

❯ youtube-dl https://www.youtube.com/c/NASASpaceflightVideos/streams
[youtube:tab] NASASpaceflightVideos: Downloading webpage
[download] Downloading playlist: NASASpaceflight - Live
[youtube:tab] Downloading page 1
[youtube:tab] Downloading page 2
[youtube:tab] Downloading page 3
[youtube:tab] Downloading page 4
[youtube:tab] Downloading page 5
[youtube:tab] Downloading page 6
[youtube:tab] Downloading page 7
[youtube:tab] Downloading page 8
[youtube:tab] Downloading page 9
[youtube:tab] Downloading page 10
[youtube:tab] Downloading page 11
[youtube:tab] Downloading page 12
[youtube:tab] Downloading page 13
[youtube:tab] Downloading page 14
[youtube:tab] Downloading page 15
[youtube:tab] Downloading page 16
[youtube:tab] Downloading page 17
[youtube:tab] playlist NASASpaceflight - Live: Downloading 531 videos
[download] Downloading video 1 of 531
[youtube] wEZo0qR75Zo: Downloading webpage
[youtube] wEZo0qR75Zo: Downloading player c4225c42
[download] Destination: Antares Launches Cygnus NG-18 to Space Station-wEZo0qR75Zo.f137.mp4
[download]   5.2% of 580.30MiB at  2.41MiB/s ETA 03:47
ERROR: Interrupted by user

Unless @dirkf wants to align the behavior of https://www.youtube.com/c/NASASpaceflightVideos with how it used to work pre-2019 (which yt-dlp emulates), this is working as expected

Edit: 2020, not 19

dirkf commented 1 year ago

In other words, YoutubeTabIE._extract_from_tabs() starts with this

        selected_tab = self._extract_selected_tab(tabs)

and then goes on to process that selected_tab.

As a result of another recent issue, the tab extractor is acquiring the ability to handle the Shorts tab (it has a different renderer).

I'm open to persuasion, but it seems to me that yt-dl should give you what would have been shown in the browser for the same URL. For YT, I expect that there are URL variations that would allow you to pull whatever yt-dlp gets, but if not you can always use several separate URLs.

alien999999999 commented 1 year ago

it would be very useful to have some mix of streams (they call it Live) and videos; there are lots of channels that make videos and just sometimes stream the similar content if editor is unavailable. but for some reason actual live and planned live is not in Live? and on home page you have uploads, but if you click on it, you go to videos (without the previous streams).

I think it's valuable to get both previous streams and videos somehow. but even better would be if you could use switches on a channel or something and we get actual live, planned live, previous streams, videos and we can just see in the json what it is?

The previous way that videos worked is what i'm looking for, but i can see value in having a "channel url" there and be like: --channel-types streams,videos,planned,live or whatever combination

Edit: Removing shorts would also be valuable

pukkandan commented 1 year ago

I think it's valuable to get both previous streams and videos somehow.

Use the uploads playlist - e.g. https://www.youtube.com/playlist?list=UUSUu1lih2RifWkKtDOJdsBA

yump commented 1 year ago

How are you getting from https://www.youtube.com/c/NASASpaceflightVideos to "UUSUu1lih2RifWkKtDOJdsBA"?

pukkandan commented 1 year ago

Get the uploader id and then replace the starting UC to UU

❯ youtube-dl https://www.youtube.com/c/NASASpaceflightVideos --playlist-end 1 --get-filename -o %(uploader_id)s
UCSUu1lih2RifWkKtDOJdsBA
dirkf commented 1 year ago

Is that also linked from the channel page in some way?

rautamiekka commented 1 year ago

Is that also linked from the channel page in some way?

Every channel has an RSS feed address which contains it ("https://www.youtube.com/feeds/videos.xml?channel_id=..."), easiest to acquire with something like https://greasyfork.org/en/scripts/32384-youtube-rss (which reads the HTML meta tag where the UC code is as a plain string) used via TamperMonkey, adding an RSS subscription address button.

dirkf commented 1 year ago

Or, more useful for yt-dl, in the ytInitialData,

.metadata.channelMetadataRenderer.rssUrl

with value https://www.youtube.com/feeds/videos.xml?channel_id=UCSUu1lih2RifWkKtDOJdsBA.

pukkandan commented 1 year ago

Is that also linked from the channel page in some way?

It used to be, but doesn't seem to be anymore