Support Twitter Spaces streams

[Igetin]

Checklist

• [x] I'm reporting a site feature request
• [x] I've verified that I'm running youtube-dl version 2021.05.16
• [x] I've searched the bugtracker for similar site feature requests including closed ones

Description

An example tweet with a Spaces link: https://twitter.com/akirosenthal/status/1397936847741800454

youtube-dl output:

$ youtube-dl https://twitter.com/akirosenthal/status/1397936847741800454
[twitter] 1397936847741800454: Downloading guest token
[twitter] 1397936847741800454: Downloading JSON metadata
[generic] 1yNGaWpbVQQxj: Requesting header
WARNING: Falling back on generic information extractor.
[generic] 1yNGaWpbVQQxj: Downloading webpage
[generic] 1yNGaWpbVQQxj: Extracting information
ERROR: Unsupported URL: https://twitter.com/i/spaces/1yNGaWpbVQQxj

Looking at the network tab in Chrome’s developer tools, it seems like the site’s making connections to pscp.tv, so possibly the already existing Periscope extractor could be used as a base to support Spaces URLs. An example playlist URL:

https://prod-fastly-ap-northeast-1.video.pscp.tv/Transcoding/v1/hls/E2iot0n3CMPrkdBrsf2mBZdvoZb6mvaVHlKLy8Kriww5Mi_ItgaIB4UdvCg0CtyapoSTlbhm9sfSfyKxQ1d16A/non_transcode/ap-northeast-1/periscope-replay-direct-prod-ap-northeast-1-public/audio-space/dynamic_playlist.m3u8?type=live

Example playlist content:

#EXTM3U
#EXT-X-VERSION:6
#EXT-X-TARGETDURATION:4
#EXT-X-INDEPENDENT-SEGMENTS
#EXT-X-MEDIA-SEQUENCE:501
#EXT-X-DISCONTINUITY-SEQUENCE:0
#EXT-X-START:TIME-OFFSET=0.01
#EXT-X-PROGRAM-DATE-TIME:2021-05-27T15:43:10.989Z
#EXTINF:2.986,
chunk_1622130185269794294_501_a.aac?type=live
#EXT-X-PROGRAM-DATE-TIME:2021-05-27T15:43:14.008Z
#EXTINF:3.030,
chunk_1622130188372878103_502_a.aac?type=live
#EXT-X-PROGRAM-DATE-TIME:2021-05-27T15:43:17.053Z
#EXTINF:3.093,
chunk_1622130191393163530_503_a.aac?type=live
#EXT-X-PROGRAM-DATE-TIME:2021-05-27T15:43:20.084Z
#EXTINF:3.008,
chunk_1622130194383671872_504_a.aac?type=live

The last chunk in that playlist would refer to this URL:

https://prod-fastly-ap-northeast-1.video.pscp.tv/Transcoding/v1/hls/E2iot0n3CMPrkdBrsf2mBZdvoZb6mvaVHlKLy8Kriww5Mi_ItgaIB4UdvCg0CtyapoSTlbhm9sfSfyKxQ1d16A/non_transcode/ap-northeast-1/periscope-replay-direct-prod-ap-northeast-1-public/audio-space/chunk_1622130194383671872_504_a.aac?type=live

[mortyobnoxious]

It would really be great if this is supported.

More Twitter Spaces links to inspect: https://twitter.com/search?q=lang%3Aen%20filter%3Aspaces&f=live

[christian-korneck]

this would be extra useful as a Twitter Spaces recording only lives for 30 days, afterwards it's lost.

[christian-korneck]

If someone can provide an example that actually works, I can try to fix this

link

[christian-korneck]

@89z

If some "Twitter Spaces" is available for a short time after the live period is over, then I can work on that.

I believe it's possible for the creator/moderator of a Twitter spaces session to optionally record that live session and share the recording afterwards via a Tweet. Users will see a player in the Twitter UI. I'm not entirely sure, but I believe "recording" is a beta feature that isn't rolled out yet to everyone (at least I couldn't find this recording option when I just tried to create a Twitter Spaces session). So I couldn't produce a recording myself. However, in this recording the creator/moderator of the session says (around 7:01):

"This is recorded. Twitter allows you to record. So you might be able to catch up later. My understand is the recording will last for 30 days. I have no idea how to download and save them anywhere else".

I'm not sure if that expiration for recordings is true or not, I could not find any docs confirming or not confirming this. However, it's definitely true that Twitter Spaces sessions can be recorded (though not yet by any user) and if you search on Twitter you will find many links to such recordings.

Hope this helps - this is all I know.

Again, here's an example of a recorded session: link

[christian-korneck]

@89z

If Twitter Spaces cant be recorded "by any user", then how was the recording created that you linked to [1]? It appears that "Andrew Brown" is a user [2], which contradicts your comment above.

This user ("Andrew Brown") is the creator/owner/moderator of that Twitter spaces live session. He started the live session. As owner of the session he had an option in the UI to record the session. When the session ended he was able to share the link to the recording via a Tweet.

Only the owner of the session can record it. Other participants (listeners) can't record a session that they don't own.

(With my comment that not all users can record yet, I meant that not everyone who starts/owns a live session can see that "record" button yet due to Beta).

[christian-korneck]

I believe I have implemented this now

awesome!

[Igetin]

Just dropping a note here: even Spaces that don’t have the recording option turned on can still be downloaded from the servers afterwards. Twitter will retain copies of all streams for up to 120 days for review purposes. I know some people have been downloading the full streams using the live_video_stream API after the stream has ended, even before Spaces supported the recording option (back in May). It requires one to obtain the playlist URL while the stream is still live, since it can't be obtained afterwards.

[swyxio]

I believe I have implemented this now

what version is this in please? i just tried it and it didnt work, and i dont see a PR that mentions twitter spaces

[swyxio]

thanks! the example i'm currently trying to do is

youtube-dl https://twitter.com/i/spaces/1RDGlgoYkDkKL

[christian-korneck]

@89z If you can find an example that works, that's three months or older, then I will restore and maintain the Twitter Spaces code

your code works just fine (I use v1.45.3 regularly, just tried it today), thanks again for it! I think you won't find a long living example as Twitter spaces recordings expire automatically after some time by design. Doesn't break your code though. Just makes it harder to write tests.

[Jojo8275]

thanks! the example i'm currently trying to do is

youtube-dl https://twitter.com/i/spaces/1RDGlgoYkDkKL

Hello, when i try it i have an error...

youtube-dl https://twitter.com/i/spaces/1lPKqmorjLdKb [generic] 1lPKqmorjLdKb: Requesting header WARNING: Falling back on generic information extractor. [generic] 1lPKqmorjLdKb: Downloading webpage [generic] 1lPKqmorjLdKb: Extracting information ERROR: Unsupported URL: https://twitter.com/i/spaces/1lPKqmorjLdKb

[tim-tx]

I have noticed that when ripping from a pscp.tv url obtained from a Twitter Spaces, yt-dlp is significantly slower than youtube-dl. Also, the file produced by yt-dlp is not readable by mpv, while vlc can read but has trouble seeking. If you pass -x to yt-dlp these problems go away. The youtube-dl file is fine without extra options.

[Infinitay]

If anyone develops this feature I would like to request also adding support for their built-in transcription of the Space as well.

Seems like they send the transcriptions via WebSockets to a /chatnow endpoint (wss://prod-chatman-ancillary-us-east-1.pscp.tv/chatapi/v1/chatnow)

[fulldecent]

If your spaces is NOT recorded and you end the event, it is still possible to get the recording media_key. Here is how to do that:

Just click on your past Tweet and check your network access for a URL like this:

https://api.twitter.com/graphql/.../AudioSpaceById

This will expose a rest_id and media_key in the format of:

• rest_id: "1BRJjZArbrBJw"
• media_key: 28_1621306926306050048

That key does NOT work in this URL format which works for actually recorded streams:

https://api.twitter.com/1.1/live_video_stream/status/:media_key_here:

---

Not sure if this helps anyone, just sharing notes.

Also, fun fact if you replay that graphql request but you change all the =false to =true you get extra stuff.

[creigh]

[generic] 1ZkKzXOXajXJv: Requesting header WARNING: Falling back on generic information extractor. [generic] 1ZkKzXOXajXJv: Downloading webpage [generic] 1ZkKzXOXajXJv: Extracting information ERROR: Unsupported URL: https://twitter.com/i/spaces/1ZkKzXOXajXJv

Same issue for me. Any helpful tips that don't include a reference to twitter api?

Perhaps I'm doing something wrong?

[dirkf]

Same issue for me. Any helpful tips that don't include a reference to twitter api?

You've just reproduced the problem shown in the original issue.

[dirkf]

I have noticed that when ripping from a pscp.tv url obtained from a Twitter Spaces, yt-dlp is significantly slower than youtube-dl. Also, the file produced by yt-dlp is not readable by mpv, while vlc can read but has trouble seeking. If you pass -x to yt-dlp these problems go away. The youtube-dl file is fine without extra options.

Sounds strange. Raise this with supporting logs at yt-dlp if reproducible.